From patchwork Tue Jan 23 17:30:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527911 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BD8680045 for ; Tue, 23 Jan 2024 17:33:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031198; cv=none; b=An0pvCmdkWYajriTwzz3tugRJvecV+l/tASXE7yl796k88jqp/TOUciZDnK8YMY296TJLpME0uqsrH25d94UbRV5NK1waBdCdUfTnV0Tl9FPFx5vvFJrN4/dOQHyOPKqZJ1cGDYpqEPgHC2KATmGNlm2rJVyjs+NJXXvCKbpi+U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031198; c=relaxed/simple; bh=+mtYadm3M+ilgJiBZY+vUq8IU5xjNYe+J28FqoPsnj8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N63iue3Y5D0K/PxNHpeowWvctjf1b/ghL3L8JwLcAbtC2mFiNgbi/qtTR7uC0QEFlxb+n821jZSotCjQNrK/jZZbuPRHkGUq4Pb/Z6ilVTWwa0gI5kke53fK0KRRdCZU25/StCVihztKWNvOu0bhsg/ZAsSb6uI8gI51lTbcyUg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=aT9eV3uk; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="aT9eV3uk" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-7bed82030faso59609039f.1 for ; Tue, 23 Jan 2024 09:33:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031196; x=1706635996; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Fl7xBNgNabyen0blKRPhiVPUOU+ukJNqSOFOnGFVt7U=; b=aT9eV3ukMh5DAxR6ZhVn/P/FnDOUAJeoGq3HR+WxjOAnPZlJVO0Biny+h/oxOWb+Pf a2gVHuM/oXDTe477XvTTugibWbQvwIF+DLrudxIdFy0jRmIa0i3QmEGH6sqJwqqB24BL uln5287LLbqOq9IM1Ofh60I1CkTjjOhXh8uzd1cuZIq6QktEHw1iWZPIfk++QC0zu2PH xQYNLMt3YJ7MEmuCYyD4PaXM1vMDYJpDjewWaIPkzjMLvRBAGGKAqNm2C45KPA0iuasB 8o1mBHviMX+LvAdeVW63GDenWzx5zyZMKL/n+LFk1TPTYsDErdTShrlrJWVkGQCrC6qv zd9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031196; x=1706635996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Fl7xBNgNabyen0blKRPhiVPUOU+ukJNqSOFOnGFVt7U=; b=duS+dCpIBYPTELkcgZafTInlTgKWyQuvkezio9TokfCPVC9u9+H3qblSImtctzSYwb UjBu6w9njPkEgyFrAWTgRADZzxuOFFPTcmir/cuA2m1dJ5W5/q4RUMP/9wYy5WEHFBkT qbq2F39aBOBcbE7FuVdY+C+lgS76dvSS/uu366kiqwdASY/Semy+2G1pE5txkYjDXv0+ IvZP/AcgE1FHpOfMIyYh3Gz23ZxsQ69jRmM8mBPl0RmF6U79y1HNujhj80+yW2kcByHc 2Qn98vgeD5Az1x14q9sa+gRi/IZZHIKjHgiA47+btzuRTtlI2JE/mHigpAkHr+ga6N+7 jsgA== X-Gm-Message-State: AOJu0YxGc3n1CDnYdegCLiQbF1WXXKPfAB+c+oOoMAiuLTVJXBnTpLPg VpVXs5RQOvc1koopcdGPOIgejNX63VdPR9xaAf9oO0Vb4p0fCEw3Gt54JPC2+N3v6ycQjTSuOTH 6ZSg= X-Google-Smtp-Source: AGHT+IErkVziVzfS6ew5ABzlNQMgAkQebrYZQOZ0J5XKlYRQmU4xtW+E+YhbZYSnceYH94OnLb0Vag== X-Received: by 2002:a5d:8e0b:0:b0:7bf:4758:2a12 with SMTP id e11-20020a5d8e0b000000b007bf47582a12mr10047616iod.0.1706031195647; Tue, 23 Jan 2024 09:33:15 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id gu12-20020a0566382e0c00b0046df4450843sm3640708jab.50.2024.01.23.09.33.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:33:13 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/6] block: move cgroup time handling code into blkdev.h Date: Tue, 23 Jan 2024 10:30:33 -0700 Message-ID: <20240123173310.1966157-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123173310.1966157-1-axboe@kernel.dk> References: <20240123173310.1966157-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for moving time keeping into blkdev.h, move the cgroup related code for timestamps in here too. This will help avoid a circular dependency. Leave struct bio_issue in blk_types.h as it's a proper time definition. Signed-off-by: Jens Axboe --- include/linux/blk_types.h | 42 -------------------------------------- include/linux/blkdev.h | 43 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 42 deletions(-) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index f288c94374b3..1c07848dea7e 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -206,52 +206,10 @@ static inline bool blk_path_error(blk_status_t error) return true; } -/* - * From most significant bit: - * 1 bit: reserved for other usage, see below - * 12 bits: original size of bio - * 51 bits: issue time of bio - */ -#define BIO_ISSUE_RES_BITS 1 -#define BIO_ISSUE_SIZE_BITS 12 -#define BIO_ISSUE_RES_SHIFT (64 - BIO_ISSUE_RES_BITS) -#define BIO_ISSUE_SIZE_SHIFT (BIO_ISSUE_RES_SHIFT - BIO_ISSUE_SIZE_BITS) -#define BIO_ISSUE_TIME_MASK ((1ULL << BIO_ISSUE_SIZE_SHIFT) - 1) -#define BIO_ISSUE_SIZE_MASK \ - (((1ULL << BIO_ISSUE_SIZE_BITS) - 1) << BIO_ISSUE_SIZE_SHIFT) -#define BIO_ISSUE_RES_MASK (~((1ULL << BIO_ISSUE_RES_SHIFT) - 1)) - -/* Reserved bit for blk-throtl */ -#define BIO_ISSUE_THROTL_SKIP_LATENCY (1ULL << 63) - struct bio_issue { u64 value; }; -static inline u64 __bio_issue_time(u64 time) -{ - return time & BIO_ISSUE_TIME_MASK; -} - -static inline u64 bio_issue_time(struct bio_issue *issue) -{ - return __bio_issue_time(issue->value); -} - -static inline sector_t bio_issue_size(struct bio_issue *issue) -{ - return ((issue->value & BIO_ISSUE_SIZE_MASK) >> BIO_ISSUE_SIZE_SHIFT); -} - -static inline void bio_issue_init(struct bio_issue *issue, - sector_t size) -{ - size &= (1ULL << BIO_ISSUE_SIZE_BITS) - 1; - issue->value = ((issue->value & BIO_ISSUE_RES_MASK) | - (ktime_get_ns() & BIO_ISSUE_TIME_MASK) | - ((u64)size << BIO_ISSUE_SIZE_SHIFT)); -} - typedef __u32 __bitwise blk_opf_t; typedef unsigned int blk_qc_t; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 99e4f5e72213..da0f7e1caa5a 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -974,6 +974,49 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) int blkdev_issue_flush(struct block_device *bdev); long nr_blockdev_pages(void); + +/* + * From most significant bit: + * 1 bit: reserved for other usage, see below + * 12 bits: original size of bio + * 51 bits: issue time of bio + */ +#define BIO_ISSUE_RES_BITS 1 +#define BIO_ISSUE_SIZE_BITS 12 +#define BIO_ISSUE_RES_SHIFT (64 - BIO_ISSUE_RES_BITS) +#define BIO_ISSUE_SIZE_SHIFT (BIO_ISSUE_RES_SHIFT - BIO_ISSUE_SIZE_BITS) +#define BIO_ISSUE_TIME_MASK ((1ULL << BIO_ISSUE_SIZE_SHIFT) - 1) +#define BIO_ISSUE_SIZE_MASK \ + (((1ULL << BIO_ISSUE_SIZE_BITS) - 1) << BIO_ISSUE_SIZE_SHIFT) +#define BIO_ISSUE_RES_MASK (~((1ULL << BIO_ISSUE_RES_SHIFT) - 1)) + +/* Reserved bit for blk-throtl */ +#define BIO_ISSUE_THROTL_SKIP_LATENCY (1ULL << 63) + +static inline u64 __bio_issue_time(u64 time) +{ + return time & BIO_ISSUE_TIME_MASK; +} + +static inline u64 bio_issue_time(struct bio_issue *issue) +{ + return __bio_issue_time(issue->value); +} + +static inline sector_t bio_issue_size(struct bio_issue *issue) +{ + return ((issue->value & BIO_ISSUE_SIZE_MASK) >> BIO_ISSUE_SIZE_SHIFT); +} + +static inline void bio_issue_init(struct bio_issue *issue, + sector_t size) +{ + size &= (1ULL << BIO_ISSUE_SIZE_BITS) - 1; + issue->value = ((issue->value & BIO_ISSUE_RES_MASK) | + (ktime_get_ns() & BIO_ISSUE_TIME_MASK) | + ((u64)size << BIO_ISSUE_SIZE_SHIFT)); +} + #else /* CONFIG_BLOCK */ struct blk_plug { }; From patchwork Tue Jan 23 17:30:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527912 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1BB36811E4 for ; Tue, 23 Jan 2024 17:33:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031201; cv=none; b=Rh2B13L4Ge0ZPuqb5WHY9YY0jcRNiK+i0Bc7mpIeGgkatbYH1xZrKGldn4fp0al7gyUauYXaTWCbojl0MpYgCP1H40pykIxC3E45/OTPZHC6NerQN5FoT7tfOKpd7EcrIZvf/hjWcY+NyD07yzJG+W/TuY+JX7nztKqLDDNP4cI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031201; c=relaxed/simple; bh=mt/csPSlwKvDneruuiEWWkV4pXV+fbUbPTLIXXRuAg8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PIxuEc/q4B6UJlZQ5BYhILbAS7LnRa04Me1Y2dcZKrCC1m+0BXKWlEPRdF9SS32r3t0vH0H+iQpfS/AjDxehmIesi/CpJBg8yic1w/N2tjU1FLcJD7yFVCiXk2dvc8Tj1fzCh4lVUg2uds3S6HrvVlM6F1M87eR0mLpzXHl5j8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=yFa7ajAd; arc=none smtp.client-ip=209.85.166.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="yFa7ajAd" Received: by mail-io1-f47.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso60897139f.0 for ; Tue, 23 Jan 2024 09:33:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031198; x=1706635998; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HPpFX+3PRbo71Vw707Mebi2sbecSvziHGvYdR5TY9cI=; b=yFa7ajAdGLjn/c3Y6gSXwlRyQfjr9drjrBnp9ZHApW2X6XQSeQPyr4lRe8IXWPcGZ6 D3HiirzhOyQeWpSAknXw9jmpJyGOQj727HK1zBS5QqLBX9BzfSn7JAgXAiJ0V3wVJtD8 xXCsETgs2Lku1rlk3wmlIuqDmmKeiWduFRbqk1Pi1nj34jNdsSV0TOC77CTE3T5wkiSk fnkOm+6OKtvxHWYkS0tr6AnGwpuBmghsbwu/zuQ9p7m5dqqkzSdJZv1f5OYVwxP/sikr AI8YXo2r7CTr4IVfSuCSEQOXXNUCYjAJsSAkJRHtPWwCeRCDcL8fqzitYBMmMBEvpvU6 XvAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031198; x=1706635998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HPpFX+3PRbo71Vw707Mebi2sbecSvziHGvYdR5TY9cI=; b=NLqfxzdUvN1kZ0pbpJjIDgXQnmYkvkvoByle+O+alq6hkvtZl3862VoWXrMJKL7IQy 1fe2+uC0iY+TyvAuViM/ePt5rZA0mkgoDnsf28X2VUGss9DmIAdaYcOKvE5yidPWWpDB /CYxXbkhLtrA03mlbTPj7FNQPpOXP/VDGq22M7vUH3Z7Y1rPcMg+lZ6WQ5OMrgGDrPam q7Zq5KPlXbVVP0sZ3foAXN35rxA8FdNV9jHBG1H4fik6h6DdQkre0DNkljIAbTrC61mm JeRdTyQnQPfmQXbMvRVzG9RSKUcB6DjhWcISlHltajwQPnNaWfki8nVL9ljY4RfKtNCO yh3g== X-Gm-Message-State: AOJu0Yz3GUibAR+NKTdFRD9SAKfTSE8sKq45yeIecBwJ1smU/v4F+uAu U4GLABS7ieNPY14FALimSKgbYzpQi1PG/stc04Xs/wEMW5b+Fd7NM5bsuRzbxMyekr+amCUudPy E/YM= X-Google-Smtp-Source: AGHT+IGZagkVOKMV+EmvD+aE6B6Vg2MA/HyEhaQG2yWOkViL+2pCMWpg4GKtU2YpaVcbdbSs8K7VZA== X-Received: by 2002:a5d:9304:0:b0:7be:e376:fc44 with SMTP id l4-20020a5d9304000000b007bee376fc44mr9545075ion.2.1706031197658; Tue, 23 Jan 2024 09:33:17 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id gu12-20020a0566382e0c00b0046df4450843sm3640708jab.50.2024.01.23.09.33.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:33:15 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe , Johannes Thumshirn Subject: [PATCH 2/6] block: add blk_time_get_ns() and blk_time_get() helpers Date: Tue, 23 Jan 2024 10:30:34 -0700 Message-ID: <20240123173310.1966157-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123173310.1966157-1-axboe@kernel.dk> References: <20240123173310.1966157-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Convert any user of ktime_get_ns() to use blk_time_get_ns(), and ktime_get() to blk_time_get(), so we have a unified API for querying the current time in nanoseconds or as ktime. No functional changes intended, this patch just wraps ktime_get_ns() and ktime_get() with a block helper. Reviewed-by: Johannes Thumshirn Signed-off-by: Jens Axboe --- block/bfq-cgroup.c | 14 +++++++------- block/bfq-iosched.c | 28 ++++++++++++++-------------- block/blk-cgroup.c | 2 +- block/blk-flush.c | 2 +- block/blk-iocost.c | 8 ++++---- block/blk-iolatency.c | 6 +++--- block/blk-mq.c | 16 ++++++++-------- block/blk-throttle.c | 6 +++--- block/blk-wbt.c | 5 ++--- include/linux/blkdev.h | 13 ++++++++++++- 10 files changed, 55 insertions(+), 45 deletions(-) diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index 2c90e5de0acd..d442ee358fc2 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -127,7 +127,7 @@ static void bfqg_stats_update_group_wait_time(struct bfqg_stats *stats) if (!bfqg_stats_waiting(stats)) return; - now = ktime_get_ns(); + now = blk_time_get_ns(); if (now > stats->start_group_wait_time) bfq_stat_add(&stats->group_wait_time, now - stats->start_group_wait_time); @@ -144,7 +144,7 @@ static void bfqg_stats_set_start_group_wait_time(struct bfq_group *bfqg, return; if (bfqg == curr_bfqg) return; - stats->start_group_wait_time = ktime_get_ns(); + stats->start_group_wait_time = blk_time_get_ns(); bfqg_stats_mark_waiting(stats); } @@ -156,7 +156,7 @@ static void bfqg_stats_end_empty_time(struct bfqg_stats *stats) if (!bfqg_stats_empty(stats)) return; - now = ktime_get_ns(); + now = blk_time_get_ns(); if (now > stats->start_empty_time) bfq_stat_add(&stats->empty_time, now - stats->start_empty_time); @@ -183,7 +183,7 @@ void bfqg_stats_set_start_empty_time(struct bfq_group *bfqg) if (bfqg_stats_empty(stats)) return; - stats->start_empty_time = ktime_get_ns(); + stats->start_empty_time = blk_time_get_ns(); bfqg_stats_mark_empty(stats); } @@ -192,7 +192,7 @@ void bfqg_stats_update_idle_time(struct bfq_group *bfqg) struct bfqg_stats *stats = &bfqg->stats; if (bfqg_stats_idling(stats)) { - u64 now = ktime_get_ns(); + u64 now = blk_time_get_ns(); if (now > stats->start_idle_time) bfq_stat_add(&stats->idle_time, @@ -205,7 +205,7 @@ void bfqg_stats_set_start_idle_time(struct bfq_group *bfqg) { struct bfqg_stats *stats = &bfqg->stats; - stats->start_idle_time = ktime_get_ns(); + stats->start_idle_time = blk_time_get_ns(); bfqg_stats_mark_idling(stats); } @@ -242,7 +242,7 @@ void bfqg_stats_update_completion(struct bfq_group *bfqg, u64 start_time_ns, u64 io_start_time_ns, blk_opf_t opf) { struct bfqg_stats *stats = &bfqg->stats; - u64 now = ktime_get_ns(); + u64 now = blk_time_get_ns(); if (now > io_start_time_ns) blkg_rwstat_add(&stats->service_time, opf, diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 3cce6de464a7..4b88a54a9b76 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -1005,7 +1005,7 @@ static struct request *bfq_check_fifo(struct bfq_queue *bfqq, rq = rq_entry_fifo(bfqq->fifo.next); - if (rq == last || ktime_get_ns() < rq->fifo_time) + if (rq == last || blk_time_get_ns() < rq->fifo_time) return NULL; bfq_log_bfqq(bfqq->bfqd, bfqq, "check_fifo: returned %p", rq); @@ -1829,7 +1829,7 @@ static void bfq_bfqq_handle_idle_busy_switch(struct bfq_data *bfqd, * bfq_bfqq_update_budg_for_activation for * details on the usage of the next variable. */ - arrived_in_time = ktime_get_ns() <= + arrived_in_time = blk_time_get_ns() <= bfqq->ttime.last_end_request + bfqd->bfq_slice_idle * 3; unsigned int act_idx = bfq_actuator_index(bfqd, rq->bio); @@ -2208,7 +2208,7 @@ static void bfq_add_request(struct request *rq) struct request *next_rq, *prev; unsigned int old_wr_coeff = bfqq->wr_coeff; bool interactive = false; - u64 now_ns = ktime_get_ns(); + u64 now_ns = blk_time_get_ns(); bfq_log_bfqq(bfqd, bfqq, "add_request %d", rq_is_sync(rq)); bfqq->queued[rq_is_sync(rq)]++; @@ -2262,7 +2262,7 @@ static void bfq_add_request(struct request *rq) bfqd->rqs_injected && bfqd->tot_rq_in_driver > 0)) && time_is_before_eq_jiffies(bfqq->decrease_time_jif + msecs_to_jiffies(10))) { - bfqd->last_empty_occupied_ns = ktime_get_ns(); + bfqd->last_empty_occupied_ns = blk_time_get_ns(); /* * Start the state machine for measuring the * total service time of rq: setting @@ -3294,7 +3294,7 @@ static void bfq_set_budget_timeout(struct bfq_data *bfqd, else timeout_coeff = bfqq->entity.weight / bfqq->entity.orig_weight; - bfqd->last_budget_start = ktime_get(); + bfqd->last_budget_start = blk_time_get(); bfqq->budget_timeout = jiffies + bfqd->bfq_timeout * timeout_coeff; @@ -3394,7 +3394,7 @@ static void bfq_arm_slice_timer(struct bfq_data *bfqd) else if (bfqq->wr_coeff > 1) sl = max_t(u32, sl, 20ULL * NSEC_PER_MSEC); - bfqd->last_idling_start = ktime_get(); + bfqd->last_idling_start = blk_time_get(); bfqd->last_idling_start_jiffies = jiffies; hrtimer_start(&bfqd->idle_slice_timer, ns_to_ktime(sl), @@ -3433,7 +3433,7 @@ static void bfq_reset_rate_computation(struct bfq_data *bfqd, struct request *rq) { if (rq != NULL) { /* new rq dispatch now, reset accordingly */ - bfqd->last_dispatch = bfqd->first_dispatch = ktime_get_ns(); + bfqd->last_dispatch = bfqd->first_dispatch = blk_time_get_ns(); bfqd->peak_rate_samples = 1; bfqd->sequential_samples = 0; bfqd->tot_sectors_dispatched = bfqd->last_rq_max_size = @@ -3590,7 +3590,7 @@ static void bfq_update_rate_reset(struct bfq_data *bfqd, struct request *rq) */ static void bfq_update_peak_rate(struct bfq_data *bfqd, struct request *rq) { - u64 now_ns = ktime_get_ns(); + u64 now_ns = blk_time_get_ns(); if (bfqd->peak_rate_samples == 0) { /* first dispatch */ bfq_log(bfqd, "update_peak_rate: goto reset, samples %d", @@ -4162,7 +4162,7 @@ static bool bfq_bfqq_is_slow(struct bfq_data *bfqd, struct bfq_queue *bfqq, if (compensate) delta_ktime = bfqd->last_idling_start; else - delta_ktime = ktime_get(); + delta_ktime = blk_time_get(); delta_ktime = ktime_sub(delta_ktime, bfqd->last_budget_start); delta_usecs = ktime_to_us(delta_ktime); @@ -5591,7 +5591,7 @@ static void bfq_init_bfqq(struct bfq_data *bfqd, struct bfq_queue *bfqq, struct bfq_io_cq *bic, pid_t pid, int is_sync, unsigned int act_idx) { - u64 now_ns = ktime_get_ns(); + u64 now_ns = blk_time_get_ns(); bfqq->actuator_idx = act_idx; RB_CLEAR_NODE(&bfqq->entity.rb_node); @@ -5903,7 +5903,7 @@ static void bfq_update_io_thinktime(struct bfq_data *bfqd, */ if (bfqq->dispatched || bfq_bfqq_busy(bfqq)) return; - elapsed = ktime_get_ns() - bfqq->ttime.last_end_request; + elapsed = blk_time_get_ns() - bfqq->ttime.last_end_request; elapsed = min_t(u64, elapsed, 2ULL * bfqd->bfq_slice_idle); ttime->ttime_samples = (7*ttime->ttime_samples + 256) / 8; @@ -6194,7 +6194,7 @@ static bool __bfq_insert_request(struct bfq_data *bfqd, struct request *rq) bfq_add_request(rq); idle_timer_disabled = waiting && !bfq_bfqq_wait_request(bfqq); - rq->fifo_time = ktime_get_ns() + bfqd->bfq_fifo_expire[rq_is_sync(rq)]; + rq->fifo_time = blk_time_get_ns() + bfqd->bfq_fifo_expire[rq_is_sync(rq)]; list_add_tail(&rq->queuelist, &bfqq->fifo); bfq_rq_enqueued(bfqd, bfqq, rq); @@ -6370,7 +6370,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) bfq_weights_tree_remove(bfqq); } - now_ns = ktime_get_ns(); + now_ns = blk_time_get_ns(); bfqq->ttime.last_end_request = now_ns; @@ -6585,7 +6585,7 @@ static void bfq_completed_request(struct bfq_queue *bfqq, struct bfq_data *bfqd) static void bfq_update_inject_limit(struct bfq_data *bfqd, struct bfq_queue *bfqq) { - u64 tot_time_ns = ktime_get_ns() - bfqd->last_empty_occupied_ns; + u64 tot_time_ns = blk_time_get_ns() - bfqd->last_empty_occupied_ns; unsigned int old_limit = bfqq->inject_limit; if (bfqq->last_serv_time_ns > 0 && bfqd->rqs_injected) { diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index ff93c385ba5a..bdbb557feb5a 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1846,7 +1846,7 @@ static void blkcg_maybe_throttle_blkg(struct blkcg_gq *blkg, bool use_memdelay) { unsigned long pflags; bool clamp; - u64 now = ktime_to_ns(ktime_get()); + u64 now = blk_time_get_ns(); u64 exp; u64 delay_nsec = 0; int tok; diff --git a/block/blk-flush.c b/block/blk-flush.c index 3f4d41952ef2..b0f314f4bc14 100644 --- a/block/blk-flush.c +++ b/block/blk-flush.c @@ -143,7 +143,7 @@ static void blk_account_io_flush(struct request *rq) part_stat_lock(); part_stat_inc(part, ios[STAT_FLUSH]); part_stat_add(part, nsecs[STAT_FLUSH], - ktime_get_ns() - rq->start_time_ns); + blk_time_get_ns() - rq->start_time_ns); part_stat_unlock(); } diff --git a/block/blk-iocost.c b/block/blk-iocost.c index c8beec6d7df0..4b0b483a9693 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -829,7 +829,7 @@ static int ioc_autop_idx(struct ioc *ioc, struct gendisk *disk) /* step up/down based on the vrate */ vrate_pct = div64_u64(ioc->vtime_base_rate * 100, VTIME_PER_USEC); - now_ns = ktime_get_ns(); + now_ns = blk_time_get_ns(); if (p->too_fast_vrate_pct && p->too_fast_vrate_pct <= vrate_pct) { if (!ioc->autop_too_fast_at) @@ -1044,7 +1044,7 @@ static void ioc_now(struct ioc *ioc, struct ioc_now *now) unsigned seq; u64 vrate; - now->now_ns = ktime_get(); + now->now_ns = blk_time_get_ns(); now->now = ktime_to_us(now->now_ns); vrate = atomic64_read(&ioc->vtime_rate); @@ -2810,7 +2810,7 @@ static void ioc_rqos_done(struct rq_qos *rqos, struct request *rq) return; } - on_q_ns = ktime_get_ns() - rq->alloc_time_ns; + on_q_ns = blk_time_get_ns() - rq->alloc_time_ns; rq_wait_ns = rq->start_time_ns - rq->alloc_time_ns; size_nsec = div64_u64(calc_size_vtime_cost(rq, ioc), VTIME_PER_NSEC); @@ -2893,7 +2893,7 @@ static int blk_iocost_init(struct gendisk *disk) ioc->vtime_base_rate = VTIME_PER_USEC; atomic64_set(&ioc->vtime_rate, VTIME_PER_USEC); seqcount_spinlock_init(&ioc->period_seqcount, &ioc->lock); - ioc->period_at = ktime_to_us(ktime_get()); + ioc->period_at = ktime_to_us(blk_time_get()); atomic64_set(&ioc->cur_period, 0); atomic_set(&ioc->hweight_gen, 0); diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index c1a6aba1d59e..ebb522788d97 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -609,7 +609,7 @@ static void blkcg_iolatency_done_bio(struct rq_qos *rqos, struct bio *bio) if (!iolat->blkiolat->enabled) return; - now = ktime_to_ns(ktime_get()); + now = blk_time_get_ns(); while (blkg && blkg->parent) { iolat = blkg_to_lat(blkg); if (!iolat) { @@ -661,7 +661,7 @@ static void blkiolatency_timer_fn(struct timer_list *t) struct blk_iolatency *blkiolat = from_timer(blkiolat, t, timer); struct blkcg_gq *blkg; struct cgroup_subsys_state *pos_css; - u64 now = ktime_to_ns(ktime_get()); + u64 now = blk_time_get_ns(); rcu_read_lock(); blkg_for_each_descendant_pre(blkg, pos_css, @@ -985,7 +985,7 @@ static void iolatency_pd_init(struct blkg_policy_data *pd) struct blkcg_gq *blkg = lat_to_blkg(iolat); struct rq_qos *rqos = iolat_rq_qos(blkg->q); struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos); - u64 now = ktime_to_ns(ktime_get()); + u64 now = blk_time_get_ns(); int cpu; if (blk_queue_nonrot(blkg->q)) diff --git a/block/blk-mq.c b/block/blk-mq.c index aa87fcfda1ec..aff9e9492f59 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -323,7 +323,7 @@ void blk_rq_init(struct request_queue *q, struct request *rq) RB_CLEAR_NODE(&rq->rb_node); rq->tag = BLK_MQ_NO_TAG; rq->internal_tag = BLK_MQ_NO_TAG; - rq->start_time_ns = ktime_get_ns(); + rq->start_time_ns = blk_time_get_ns(); rq->part = NULL; blk_crypto_rq_set_defaults(rq); } @@ -333,7 +333,7 @@ EXPORT_SYMBOL(blk_rq_init); static inline void blk_mq_rq_time_init(struct request *rq, u64 alloc_time_ns) { if (blk_mq_need_time_stamp(rq)) - rq->start_time_ns = ktime_get_ns(); + rq->start_time_ns = blk_time_get_ns(); else rq->start_time_ns = 0; @@ -444,7 +444,7 @@ static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data) /* alloc_time includes depth and tag waits */ if (blk_queue_rq_alloc_time(q)) - alloc_time_ns = ktime_get_ns(); + alloc_time_ns = blk_time_get_ns(); if (data->cmd_flags & REQ_NOWAIT) data->flags |= BLK_MQ_REQ_NOWAIT; @@ -629,7 +629,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, /* alloc_time includes depth and tag waits */ if (blk_queue_rq_alloc_time(q)) - alloc_time_ns = ktime_get_ns(); + alloc_time_ns = blk_time_get_ns(); /* * If the tag allocator sleeps we could get an allocation for a @@ -1042,7 +1042,7 @@ static inline void __blk_mq_end_request_acct(struct request *rq, u64 now) inline void __blk_mq_end_request(struct request *rq, blk_status_t error) { if (blk_mq_need_time_stamp(rq)) - __blk_mq_end_request_acct(rq, ktime_get_ns()); + __blk_mq_end_request_acct(rq, blk_time_get_ns()); blk_mq_finish_request(rq); @@ -1085,7 +1085,7 @@ void blk_mq_end_request_batch(struct io_comp_batch *iob) u64 now = 0; if (iob->need_ts) - now = ktime_get_ns(); + now = blk_time_get_ns(); while ((rq = rq_list_pop(&iob->req_list)) != NULL) { prefetch(rq->bio); @@ -1255,7 +1255,7 @@ void blk_mq_start_request(struct request *rq) if (test_bit(QUEUE_FLAG_STATS, &q->queue_flags) && !blk_rq_is_passthrough(rq)) { - rq->io_start_time_ns = ktime_get_ns(); + rq->io_start_time_ns = blk_time_get_ns(); rq->stats_sectors = blk_rq_sectors(rq); rq->rq_flags |= RQF_STATS; rq_qos_issue(q, rq); @@ -3107,7 +3107,7 @@ blk_status_t blk_insert_cloned_request(struct request *rq) blk_mq_run_dispatch_ops(q, ret = blk_mq_request_issue_directly(rq, true)); if (ret) - blk_account_io_done(rq, ktime_get_ns()); + blk_account_io_done(rq, blk_time_get_ns()); return ret; } EXPORT_SYMBOL_GPL(blk_insert_cloned_request); diff --git a/block/blk-throttle.c b/block/blk-throttle.c index 16f5766620a4..da9dc1f793c3 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1815,7 +1815,7 @@ static bool throtl_tg_is_idle(struct throtl_grp *tg) time = min_t(unsigned long, MAX_IDLE_TIME, 4 * tg->idletime_threshold); ret = tg->latency_target == DFL_LATENCY_TARGET || tg->idletime_threshold == DFL_IDLE_THRESHOLD || - (ktime_get_ns() >> 10) - tg->last_finish_time > time || + (blk_time_get_ns() >> 10) - tg->last_finish_time > time || tg->avg_idletime > tg->idletime_threshold || (tg->latency_target && tg->bio_cnt && tg->bad_bio_cnt * 5 < tg->bio_cnt); @@ -2060,7 +2060,7 @@ static void blk_throtl_update_idletime(struct throtl_grp *tg) if (last_finish_time == 0) return; - now = ktime_get_ns() >> 10; + now = blk_time_get_ns() >> 10; if (now <= last_finish_time || last_finish_time == tg->checked_last_finish_time) return; @@ -2327,7 +2327,7 @@ void blk_throtl_bio_endio(struct bio *bio) if (!tg->td->limit_valid[LIMIT_LOW]) return; - finish_time_ns = ktime_get_ns(); + finish_time_ns = blk_time_get_ns(); tg->last_finish_time = finish_time_ns >> 10; start_time = bio_issue_time(&bio->bi_issue) >> 10; diff --git a/block/blk-wbt.c b/block/blk-wbt.c index 5ba3cd574eac..4c1c04345040 100644 --- a/block/blk-wbt.c +++ b/block/blk-wbt.c @@ -274,13 +274,12 @@ static inline bool stat_sample_valid(struct blk_rq_stat *stat) static u64 rwb_sync_issue_lat(struct rq_wb *rwb) { - u64 now, issue = READ_ONCE(rwb->sync_issue); + u64 issue = READ_ONCE(rwb->sync_issue); if (!issue || !rwb->sync_cookie) return 0; - now = ktime_to_ns(ktime_get()); - return now - issue; + return blk_time_get_ns() - issue; } static inline unsigned int wbt_inflight(struct rq_wb *rwb) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index da0f7e1caa5a..40d3d7da39a4 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -24,6 +24,7 @@ #include #include #include +#include struct module; struct request_queue; @@ -975,6 +976,16 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) int blkdev_issue_flush(struct block_device *bdev); long nr_blockdev_pages(void); +static inline u64 blk_time_get_ns(void) +{ + return ktime_get_ns(); +} + +static inline ktime_t blk_time_get(void) +{ + return ns_to_ktime(blk_time_get_ns()); +} + /* * From most significant bit: * 1 bit: reserved for other usage, see below @@ -1013,7 +1024,7 @@ static inline void bio_issue_init(struct bio_issue *issue, { size &= (1ULL << BIO_ISSUE_SIZE_BITS) - 1; issue->value = ((issue->value & BIO_ISSUE_RES_MASK) | - (ktime_get_ns() & BIO_ISSUE_TIME_MASK) | + (blk_time_get_ns() & BIO_ISSUE_TIME_MASK) | ((u64)size << BIO_ISSUE_SIZE_SHIFT)); } From patchwork Tue Jan 23 17:30:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527913 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC3737FBA1 for ; Tue, 23 Jan 2024 17:33:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031202; cv=none; b=PjYjHN7vzSmmFi0SzRmHK2NFrUTdFxKuXUbOyBiul+jxUMUFc+x3CF73Vo2NhV4RvjISaLX/24kEsWcYnlGY2c6OwPVEtcueZG+C+WDHYoOJzu7cR3zIIjqfnBxj7gWyTWW679MWYVWXPUAbSDfr1YPxnk81V4qoX3jqgAzZen8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031202; c=relaxed/simple; bh=xTeZ1MtdqkFINZNWvPzMmND18TlLFbX8py4lgGaKItg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dDl5eg7a0omJh1u6Z87ZvI/kx3ilD36SwVCKR6lPlQOar/p0iR2adtfiFsJ1gvEm4r7B5bpRyO01cJIHCiEb8DsMQ4RAL6uktKU8AItHN0p80jpR8p5oYA+3B+EGx9okraEip0rT+fZ1zQih3pmw3jVPUhLLAwfa15kd5H6PuDc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=BLJZjmMg; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="BLJZjmMg" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7bbdd28a52aso56467339f.1 for ; Tue, 23 Jan 2024 09:33:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031199; x=1706635999; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x81FZEO1B1ujEbVMMlyBpMIKlpsHmPREgp0HyEDYxfk=; b=BLJZjmMgGeK44Hrn7Qj39PvnibOCiDMuR1GSkOsuOzJ/24I/JKiwClWH3lhgdOSY+i sPpxjG0DOokusYp6r5hSRz0eQ3Bt7sTLU6iBt1quuzLtgyMcPg4y3YPDukoxRDxbR+ul zK6dUMPki1k8bc71QXiFOaUBq5xu+nQCGgFDqkHGvCvT9Xa2tlv3rImyNE/P2cVr/cNq E6KlvCqIBNqtryGTvTVgv/mVGnH1VF061lvXlnpMoQ/pNoNqocCCL9Jem/eHlFu6bUHm 3b62SfN14n1Nb1tzhkiuzFQ39iGTZFkjLNx5XihFmkJo4aS1KsEaMCgqsWTXt6TWonV+ D+Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031199; x=1706635999; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x81FZEO1B1ujEbVMMlyBpMIKlpsHmPREgp0HyEDYxfk=; b=M7DYv7o4CgJTiqnmBn2b1etBlsHIcQk4bj7Wd1a7yeZR7TTARkaS/2Qkkp1OyHc6Vz VBWaf/bCPnhrQTT4UnX7ELC3NE2H+tfBsZQDJlXUHypRIAozKGTGj5kH7W9Bz3cli4D8 03vRoAAFkIigWQRCWnN1reTL0MNhoop6tFUA3PCgTwxiZulDUy3bftGgiNxs/ybF81eQ zh8eu1IDxycZ08WKMQpGZt/g4qozu4Olyci90ZkXeS+Xahhll1Y3cfM8MZks/jm/6kYm vAwsBAMNq1iRzgWNph6ylOdQ540SLSqG4mRxCUVM9Et8z/IINvVFKGiDokgLYItPOOkx w4oA== X-Gm-Message-State: AOJu0YzbBghzDEyH/kWZRjzln2nmHoZIIHgstx3bABY4oRe0oCXkF8C+ OlycwtfEoPkTA//5hQfJJLeRVMj2G5ekjg9PvZfeVFBBuCpQcRnLu75d1m6KSgFR8WCLIxzM0Ke E1ms= X-Google-Smtp-Source: AGHT+IFmaRaZCkFqboHS5XbolfwX29kx1gKmj7iYQJkVTWhdorjmZvcbsYeRSijv7C3poMrtKNhl9g== X-Received: by 2002:a05:6602:e04:b0:7bf:f20:2c78 with SMTP id gp4-20020a0566020e0400b007bf0f202c78mr9470495iob.1.1706031199556; Tue, 23 Jan 2024 09:33:19 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id gu12-20020a0566382e0c00b0046df4450843sm3640708jab.50.2024.01.23.09.33.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:33:17 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe , Johannes Thumshirn Subject: [PATCH 3/6] block: cache current nsec time in struct blk_plug Date: Tue, 23 Jan 2024 10:30:35 -0700 Message-ID: <20240123173310.1966157-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123173310.1966157-1-axboe@kernel.dk> References: <20240123173310.1966157-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Querying the current time is the most costly thing we do in the block layer per IO, and depending on kernel config settings, we may do it many times per IO. None of the callers actually need nsec granularity. Take advantage of that by caching the current time in the plug, with the assumption here being that any time checking will be temporally close enough that the slight loss of precision doesn't matter. If the block plug gets flushed, eg on preempt or schedule out, then we invalidate the cached clock. On a basic peak IOPS test case with iostats enabled, this changes the performance from: IOPS=108.41M, BW=52.93GiB/s, IOS/call=31/31 IOPS=108.43M, BW=52.94GiB/s, IOS/call=32/32 IOPS=108.29M, BW=52.88GiB/s, IOS/call=31/32 IOPS=108.35M, BW=52.91GiB/s, IOS/call=32/32 IOPS=108.42M, BW=52.94GiB/s, IOS/call=31/31 IOPS=108.40M, BW=52.93GiB/s, IOS/call=32/32 IOPS=108.31M, BW=52.89GiB/s, IOS/call=32/31 to IOPS=118.79M, BW=58.00GiB/s, IOS/call=31/32 IOPS=118.62M, BW=57.92GiB/s, IOS/call=31/31 IOPS=118.80M, BW=58.01GiB/s, IOS/call=32/31 IOPS=118.78M, BW=58.00GiB/s, IOS/call=32/32 IOPS=118.69M, BW=57.95GiB/s, IOS/call=32/31 IOPS=118.62M, BW=57.92GiB/s, IOS/call=32/31 IOPS=118.63M, BW=57.92GiB/s, IOS/call=31/32 which is more than a 9% improvement in performance. Looking at perf diff, we can see a huge reduction in time overhead: 10.55% -9.88% [kernel.vmlinux] [k] read_tsc 1.31% -1.22% [kernel.vmlinux] [k] ktime_get Note that since this relies on blk_plug for the caching, it's only applicable to the issue side. But this is where most of the time calls happen anyway. On the completion side, cached time stamping is done with struct io_comp patch, as long as the driver supports it. It's also worth noting that the above testing doesn't enable any of the higher cost CPU items on the block layer side, like wbt, cgroups, iocost, etc, which all would add additional time querying and hence overhead. IOW, results would likely look even better in comparison with those enabled, as distros would do. Reviewed-by: Johannes Thumshirn Signed-off-by: Jens Axboe --- block/blk-core.c | 1 + include/linux/blkdev.h | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/block/blk-core.c b/block/blk-core.c index 11342af420d0..cc4db4d92c75 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1073,6 +1073,7 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) if (tsk->plug) return; + plug->cur_ktime = 0; plug->mq_list = NULL; plug->cached_rq = NULL; plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 40d3d7da39a4..d59610483619 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -943,6 +943,7 @@ struct blk_plug { /* if ios_left is > 1, we can batch tag/rq allocations */ struct request *cached_rq; + u64 cur_ktime; unsigned short nr_ios; unsigned short rq_count; @@ -978,7 +979,19 @@ long nr_blockdev_pages(void); static inline u64 blk_time_get_ns(void) { - return ktime_get_ns(); + struct blk_plug *plug = current->plug; + + if (!plug) + return ktime_get_ns(); + + /* + * 0 could very well be a valid time, but rather than flag "this is + * a valid timestamp" separately, just accept that we'll do an extra + * ktime_get_ns() if we just happen to get 0 as the current time. + */ + if (!plug->cur_ktime) + plug->cur_ktime = ktime_get_ns(); + return plug->cur_ktime; } static inline ktime_t blk_time_get(void) From patchwork Tue Jan 23 17:30:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527914 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC44D7FBD7 for ; Tue, 23 Jan 2024 17:33:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031204; cv=none; b=fjuzC/LUrpINrXL65hMjurHJF/BfeXTcw7ZtWBHH8cAXFA3rxHnQtxiBnd/XiDR73cTQSyUXsBpRcVkn1Xj0tygXnk4Jpng2Q/6Pu746u6kotPuzkLcTsU9Ia+lG+ISAfckwHq7INH0oP3V7esCm1cAetM8gCpwTctDbnL7scaY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031204; c=relaxed/simple; bh=cHEVYaZrC0cKZQbM1XCitnRvrM7dSUljuu8ypkDVvuU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pW/DK0jQdzVsqEij3OCtfVld7qxT5MlUIfnNcw5R8zQ2qQYEuDssQs0GN7y7EU8betZ0lql71W0ELJHAMjr/2hXximXdgIyDFVIBiaroeF47AICQY+/Gw05+MkIfMkvng5lvmTIbKPTdAOkwz8HLtLdzlbfZepJWPCxVmfZBTdM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=2m74Jqyj; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="2m74Jqyj" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7beeeb1ba87so61633539f.0 for ; Tue, 23 Jan 2024 09:33:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031201; x=1706636001; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BxbR1PijRVCdMvYYvc9G6hASa6voFvqAh71Hnsi23v8=; b=2m74JqyjFQGd4OvTYn0RRWyO0+y3byXZYSvyXfak2Q4U9K+BZSDyawbA16YcZtvScv pe+ejBD/KYFjmPaf4Warx0nRI+5aofjfuNm7YxyajbIROjS0vCGRKEUA533xuGphcj/T UKi+G7g/SoAD2BWEOYjIZjrI6yDLKa0Qkb5gVSqHbgvE7UZZ7wDrHgqzNCoT6c91vl6N jzNTueBji7TuF0ZpFz0koAfS5C5t0LZFGZ2V4i27jkrMkWLlHDwnmVjTjDrv3R/5Fnl6 5BGempXHxrc/jftT7YW7D3UHFija9U9vRM3OIbZkSRqSUSY5clBWdoGDeV/8NudGRC3H iznA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031201; x=1706636001; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BxbR1PijRVCdMvYYvc9G6hASa6voFvqAh71Hnsi23v8=; b=Nnnec2ONoJZQbF3oQjS9wRSM3mTxnCBe4p5CeF0JGU9Q3kb7biCpWmR0vqfmiPJsWv vvB1G0263uTWqoAx1dfuP6XJ3RaVytOiZmO4727HdcP88xiI3NwYMlzNc4Ao36982510 +bJQ0fHoJPDwrp4AQ8P1YHxEwTzc1YGGlyHb+ZAx2CQqvTYtvW0VbBXootNddvdPfr+v swFpPizXqpYa6+X1yzSAJQf970uReRKlwD6QZdAz3C9N4T2gKqahMSpBLkVEH3MSjbNY YC7xonKjzFsI+jnh2Ndh1Qs9nPUElY2koUyUHXkUUhj7FkEYN+IoX8odc9iURYoDVwZL VgGg== X-Gm-Message-State: AOJu0Yw7bJryIB28uDpj5iYHnEMlSeRjOokfi30eAidBznykOJkwJm0/ XNhMI1AyjgxBWeiJowfGnvzg+Fa+dQ3R4y3q0zrntaAQS5Eunule73VB8U+6vApjx9HFx2NBWO4 oAJI= X-Google-Smtp-Source: AGHT+IGwhS+Gfjyer4EGMf4Ee+ZRUoyF9JnKZfz9ky6ZsWBW1yW1Egb+jzpjnmZ8pRLV0G4aKd5SLw== X-Received: by 2002:a5d:8e0b:0:b0:7bf:4758:2a12 with SMTP id e11-20020a5d8e0b000000b007bf47582a12mr10047787iod.0.1706031201396; Tue, 23 Jan 2024 09:33:21 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id gu12-20020a0566382e0c00b0046df4450843sm3640708jab.50.2024.01.23.09.33.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:33:19 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe , Johannes Thumshirn Subject: [PATCH 4/6] block: update cached timestamp post schedule/preemption Date: Tue, 23 Jan 2024 10:30:36 -0700 Message-ID: <20240123173310.1966157-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123173310.1966157-1-axboe@kernel.dk> References: <20240123173310.1966157-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Mark the task as having a cached timestamp when set assign it, so we can efficiently check if it needs updating post being scheduled back in. This covers both the actual schedule out case, which would've flushed the plug, and the preemption case which doesn't touch the plugged requests (for many reasons, one of them being then we'd need to have preemption disabled around plug state manipulation). Reviewed-by: Johannes Thumshirn Signed-off-by: Jens Axboe --- block/blk-core.c | 2 ++ include/linux/blkdev.h | 20 +++++++++++++++++++- include/linux/sched.h | 2 +- kernel/sched/core.c | 4 +++- 4 files changed, 25 insertions(+), 3 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index cc4db4d92c75..71c6614a97fe 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1173,6 +1173,8 @@ void __blk_flush_plug(struct blk_plug *plug, bool from_schedule) */ if (unlikely(!rq_list_empty(plug->cached_rq))) blk_mq_free_plug_rqs(plug); + + current->flags &= ~PF_BLOCK_TS; } /** diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index d59610483619..9660a65bb927 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -977,6 +977,18 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) int blkdev_issue_flush(struct block_device *bdev); long nr_blockdev_pages(void); +/* + * tsk == current here + */ +static inline void blk_plug_invalidate_ts(struct task_struct *tsk) +{ + struct blk_plug *plug = tsk->plug; + + if (plug) + plug->cur_ktime = 0; + current->flags &= ~PF_BLOCK_TS; +} + static inline u64 blk_time_get_ns(void) { struct blk_plug *plug = current->plug; @@ -989,8 +1001,10 @@ static inline u64 blk_time_get_ns(void) * a valid timestamp" separately, just accept that we'll do an extra * ktime_get_ns() if we just happen to get 0 as the current time. */ - if (!plug->cur_ktime) + if (!plug->cur_ktime) { plug->cur_ktime = ktime_get_ns(); + current->flags |= PF_BLOCK_TS; + } return plug->cur_ktime; } @@ -1062,6 +1076,10 @@ static inline void blk_flush_plug(struct blk_plug *plug, bool async) { } +static inline void blk_plug_invalidate_ts(struct task_struct *tsk) +{ +} + static inline int blkdev_issue_flush(struct block_device *bdev) { return 0; diff --git a/include/linux/sched.h b/include/linux/sched.h index cdb8ea53c365..801233cef2fc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1642,7 +1642,7 @@ extern struct pid *cad_pid; #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ #define PF_MEMALLOC_PIN 0x10000000 /* Allocation context constrained to zones which allow long term pinning. */ -#define PF__HOLE__20000000 0x20000000 +#define PF_BLOCK_TS 0x20000000 /* plug has ts that needs updating */ #define PF__HOLE__40000000 0x40000000 #define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9116bcc90346..4675d59313ba 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6787,7 +6787,9 @@ static inline void sched_submit_work(struct task_struct *tsk) static void sched_update_worker(struct task_struct *tsk) { - if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER)) { + if (tsk->flags & (PF_WQ_WORKER | PF_IO_WORKER | PF_BLOCK_TS)) { + if (tsk->flags & PF_BLOCK_TS) + blk_plug_invalidate_ts(tsk); if (tsk->flags & PF_WQ_WORKER) wq_worker_running(tsk); else From patchwork Tue Jan 23 17:30:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527915 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DEDD7FBA1 for ; Tue, 23 Jan 2024 17:33:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031206; cv=none; b=qYdfYV6fvdjf7S7ZBbwfihEPM4R1Feg5+g6iO5CbrTc24Pe5h/RWbt61r63FVotbVA9DM8qUZCDwG7wXzDdHackTeHrGet98qmmFMsdAFAiLvsn8UnJzEy54fEzxihFbPyQMCbVO/AGDibIRx9fBrC2PApX7ZSjm85x6qzsI0m4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031206; c=relaxed/simple; bh=p7vwFJau7qEQ4ufBglXQjs3pqU2/i4O3ue7lco2L1hc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dJdvZJ7UsjxDvjvrXkpR2gg8HeHUjKDiq8doPxqiJR8sQmYIBS/2sak4sW02smN6mv+Fa3ieAuhd4woW2xq8PHFJKGYlgXG68nIDQmRboiB6T6WgrYkoVI7ZCInQeF8MjWLLT7LXYvo9TdQlA1SXpG/neAxIyptCeTMuYREODys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=sSZQmbmB; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="sSZQmbmB" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-7bee9f626caso62081239f.0 for ; Tue, 23 Jan 2024 09:33:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031203; x=1706636003; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gH4lZhxqtdD3FbEMI55dx5z4UKpqUWwGArOpZ0i0C1Q=; b=sSZQmbmBNdylw4QsOXos2AKOHpNy9FXdq/nQZkQbpigksozGN9P/miKNSptjP6hYKX chRbBlth6Yo9dTTXtZI4Qc6hzGzO/8Clvp9X8NDKMMr8AYmTwK6SFbNWRlXqoBZInbL+ DFHPqTBcBQneFziHXDOKaSryQrvVK+boVMfTUKhYYfQ7fY3M0P+sZvZ5PJEnec8f6IvC DNOTm4EU4JSAXBrCAZRR8VMjPFuvCr0zxUmY6TSffRokF6HrORjsuwHxaPZTHZBR3Kw/ 7XIEOU0P9fVRzDeiuK/eZNZ80ttbVb0fKSXyiW9i9V+5S19GGUM1kVdTWR71lvebpTPj iTbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031203; x=1706636003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gH4lZhxqtdD3FbEMI55dx5z4UKpqUWwGArOpZ0i0C1Q=; b=ULmwCmwdsJ9rfEdWTMgRz1rrRhxVtYjZbjO0kdBSmKM5IkIGc3V7a6mWA8PdNu8xyN bcid59ky56XP59PtCfu4txRe9vXWlO0CA3iVXmytJO7W/W9YbsItI0sstacrRfcHb4o2 36XjRcpOmtrUGrOUZwiDRxIsyqPn0d6qW7I5P+DZnHZ8e0ZIFvb83kf5TevRDi4gVqJo Tx+sRPPoLuTB5CzlQFu+Sj+5SVPAqzWHxmLsLYGvLN+edeu9ucFuVhs98POFG6kq3KL+ 5gy/QWYTw2zEjLraW77Mu9x3/16mxiyUmkhYb2cdeSpDx9y5nC7ef9GAtIKLTRwfRM7N I+WA== X-Gm-Message-State: AOJu0Yze17viNpg4XoxcZCsNOhIjvR8Lk9voIHlZp+QZbSVTZ7ANOFdN O/QXbOhhRTJTz8vVsHSWWy+KVnuyv0NetH9zPRNYucFq3JPE+BtLL3+vZtzw+aJrpon1ofwWcr9 xpK4= X-Google-Smtp-Source: AGHT+IHgLQJxIFtXneGZwVz9S47OXeWhgLLk2cedeglfcOKnm0WCh1YP3OXrzkQ4vjXjGkKh2d8z1A== X-Received: by 2002:a05:6602:e04:b0:7bf:f20:2c78 with SMTP id gp4-20020a0566020e0400b007bf0f202c78mr9470693iob.1.1706031203236; Tue, 23 Jan 2024 09:33:23 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id gu12-20020a0566382e0c00b0046df4450843sm3640708jab.50.2024.01.23.09.33.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:33:21 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/6] block: shrink plug->{nr_ios, rq_count} to unsigned char Date: Tue, 23 Jan 2024 10:30:37 -0700 Message-ID: <20240123173310.1966157-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123173310.1966157-1-axboe@kernel.dk> References: <20240123173310.1966157-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We never use more than 64 max in here, we can change them from unsigned short to just a byte. Add a BUILD_BUG_ON() check, in case the max plug count changes in the future. Signed-off-by: Jens Axboe --- block/blk-core.c | 4 ++-- block/blk-mq.c | 2 ++ include/linux/blkdev.h | 8 ++++---- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 71c6614a97fe..dd593008511c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1063,7 +1063,7 @@ int kblockd_mod_delayed_work_on(int cpu, struct delayed_work *dwork, } EXPORT_SYMBOL(kblockd_mod_delayed_work_on); -void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) +void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned char nr_ios) { struct task_struct *tsk = current; @@ -1076,7 +1076,7 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned short nr_ios) plug->cur_ktime = 0; plug->mq_list = NULL; plug->cached_rq = NULL; - plug->nr_ios = min_t(unsigned short, nr_ios, BLK_MAX_REQUEST_COUNT); + plug->nr_ios = min_t(unsigned char, nr_ios, BLK_MAX_REQUEST_COUNT); plug->rq_count = 0; plug->multiple_queues = false; plug->has_elevator = false; diff --git a/block/blk-mq.c b/block/blk-mq.c index aff9e9492f59..a9b4a66e1e13 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1283,6 +1283,8 @@ EXPORT_SYMBOL(blk_mq_start_request); */ static inline unsigned short blk_plug_max_rq_count(struct blk_plug *plug) { + BUILD_BUG_ON(2 * BLK_MAX_REQUEST_COUNT > U8_MAX); + if (plug->multiple_queues) return BLK_MAX_REQUEST_COUNT * 2; return BLK_MAX_REQUEST_COUNT; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 9660a65bb927..ce6d057de2f0 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -944,9 +944,9 @@ struct blk_plug { /* if ios_left is > 1, we can batch tag/rq allocations */ struct request *cached_rq; u64 cur_ktime; - unsigned short nr_ios; + unsigned char nr_ios; - unsigned short rq_count; + unsigned char rq_count; bool multiple_queues; bool has_elevator; @@ -964,7 +964,7 @@ struct blk_plug_cb { extern struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data, int size); extern void blk_start_plug(struct blk_plug *); -extern void blk_start_plug_nr_ios(struct blk_plug *, unsigned short); +extern void blk_start_plug_nr_ios(struct blk_plug *, unsigned char); extern void blk_finish_plug(struct blk_plug *); void __blk_flush_plug(struct blk_plug *plug, bool from_schedule); @@ -1060,7 +1060,7 @@ struct blk_plug { }; static inline void blk_start_plug_nr_ios(struct blk_plug *plug, - unsigned short nr_ios) + unsigned char nr_ios) { } From patchwork Tue Jan 23 17:30:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13527916 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D90580028 for ; Tue, 23 Jan 2024 17:33:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031207; cv=none; b=YKI5X3HfEsAXcCMVf08jhiRTC2x5St8AD5W+LYZe5PiDEPVf+j6qifZfer2DmefC3ke0KaktiCZJFYdKoW3vLiipqLSEHHd5MHJZphQ36I7wTYXBhbKzlQE5d1a+nzECDNuP23ibXhsbc0iUUinoxxn+o2gvSi3C5io0oRuGS34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706031207; c=relaxed/simple; bh=0dAmO6x2styloC+APT5Im4vJgC6epFT1Zp74CyP8Mww=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jWm4KjnrvLy92DtzA5li9V+MqK7wyzCvTe90qIDLl0x8A0bN5LTitgdYoA7ecnApZDwF1Kxcm+QIVqep9jqp4wrTQ0/JscaH//B1B9o1d+qyyoYBC6LS0G651AdbINIhyrJnFdG5eBo9B3UWav1AkS/X727aQgcyjcKDE8Ihz14= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=U1nfBwfU; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="U1nfBwfU" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso60900239f.0 for ; Tue, 23 Jan 2024 09:33:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706031205; x=1706636005; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+/Ald4gHrRglycNy6wXvcN5PFyAD6wLCcn4WkvqJFC0=; b=U1nfBwfUVvg4gjho/xp8VxuWK3QoxpChY/ACWDu9+B/v70n4oTQx7ubSDKG57Bflo8 Xwxy6Wggb8ZUf6U/cbCOhEWssGQTE1PnQtxKA4JevUbcVJqA4mu4JJHG6dDBtqS7Z4cf iL6f8PDYD0FdfdoTLUBUVlsQyziN2Blp7L0HLAAsiQVw6Vy62dIy4pPueIJh2qPAuHBY BA/TV/d15zjsaz3M/bmKbQraQq7IT6J7kE6Uux9DcJnVZcHnsPiZy5TRmQW4ZGFIa+rx p/qxlWzH9eCk2nYMfpfbHEbpppmJ65KstRtRoyO79XsIVvMuORejj+4ZwpBJO8Ohzn42 y9KQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706031205; x=1706636005; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+/Ald4gHrRglycNy6wXvcN5PFyAD6wLCcn4WkvqJFC0=; b=svn3zoq/DVkPOdGstggYmUNHunWiFlUTSvHzycNJBCVZuUFYUJnn3oHqorZ2tHveDx gld/yJe+V26tLgE02lgNsh6n7iiZivS2lovoouAbUY0YYGte4BakmmIn+8gROtS1XZW2 QJ7QhQxvz36IhLpp5kihFRvwNIc8fGBRcElHJA5o0NDHZfq7WzOv4aYdLGWAVlalU1y8 vaTlOXcGPqNxg5fuN/mnzt4eovBf9qZ6cYQdqAave3/mrQPpXpKlJc7hZ+hcHBgkDPpj L3lVsQYe2GtGdE1HG8KCDPS3IdrSeWGeG4cClnsk+nCT133BpTfT7haiDDc/mYSkcnl7 Otxg== X-Gm-Message-State: AOJu0YzKh+yzP0aun5xXs3AdHGnxqwtzjZO5y/CLW8OTDnihX4UNZuuY SSL8BQx3IiRgOZ/DiIihI9/MU+ohS3mnlh0HpRn0jz01GxGo0TrvIS/CmijMnfDOCcg/twtUy7k quTU= X-Google-Smtp-Source: AGHT+IElRCm4FAwDKHywvZV0YwCO0C/6R+Ki3OzPgiP7nMOonsmT50PyINzaXuntWvrzICAf02VBow== X-Received: by 2002:a5d:9304:0:b0:7be:e376:fc44 with SMTP id l4-20020a5d9304000000b007bee376fc44mr9545381ion.2.1706031205127; Tue, 23 Jan 2024 09:33:25 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id gu12-20020a0566382e0c00b0046df4450843sm3640708jab.50.2024.01.23.09.33.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 09:33:23 -0800 (PST) From: Jens Axboe To: linux-block@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/6] block: convert struct blk_plug callback list to hlists Date: Tue, 23 Jan 2024 10:30:38 -0700 Message-ID: <20240123173310.1966157-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240123173310.1966157-1-axboe@kernel.dk> References: <20240123173310.1966157-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We currently use a doubly linked list, which means the head takes up 16 bytes. As any iteration goes over the full list by first splicing it to an on-stack copy, we never need to remove members from the middle of the list. Convert it to an hlist instead, saving 8 bytes in the blk_plug structure. This also helps save 40 bytes of text in the core block code, tested on arm64. This does mean that flush callbacks will be run in reverse. While this should not pose a problem, we can always change the list splicing to just iteration-and-add instead, preservering ordering. These lists are generally just a single entry (or a few entries), either way this should be fine. Signed-off-by: Jens Axboe --- block/blk-core.c | 26 ++++++++++++++------------ drivers/md/raid1-10.c | 2 +- include/linux/blkdev.h | 4 ++-- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index dd593008511c..f28859b4a3ef 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1080,7 +1080,7 @@ void blk_start_plug_nr_ios(struct blk_plug *plug, unsigned char nr_ios) plug->rq_count = 0; plug->multiple_queues = false; plug->has_elevator = false; - INIT_LIST_HEAD(&plug->cb_list); + INIT_HLIST_HEAD(&plug->cb_list); /* * Store ordering should not be needed here, since a potential @@ -1120,16 +1120,18 @@ EXPORT_SYMBOL(blk_start_plug); static void flush_plug_callbacks(struct blk_plug *plug, bool from_schedule) { - LIST_HEAD(callbacks); + HLIST_HEAD(callbacks); - while (!list_empty(&plug->cb_list)) { - list_splice_init(&plug->cb_list, &callbacks); + while (!hlist_empty(&plug->cb_list)) { + struct hlist_node *entry, *tmp; - while (!list_empty(&callbacks)) { - struct blk_plug_cb *cb = list_first_entry(&callbacks, - struct blk_plug_cb, - list); - list_del(&cb->list); + hlist_move_list(&plug->cb_list, &callbacks); + + hlist_for_each_safe(entry, tmp, &callbacks) { + struct blk_plug_cb *cb; + + cb = hlist_entry(entry, struct blk_plug_cb, list); + hlist_del(&cb->list); cb->callback(cb, from_schedule); } } @@ -1144,7 +1146,7 @@ struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data, if (!plug) return NULL; - list_for_each_entry(cb, &plug->cb_list, list) + hlist_for_each_entry(cb, &plug->cb_list, list) if (cb->callback == unplug && cb->data == data) return cb; @@ -1154,7 +1156,7 @@ struct blk_plug_cb *blk_check_plugged(blk_plug_cb_fn unplug, void *data, if (cb) { cb->data = data; cb->callback = unplug; - list_add(&cb->list, &plug->cb_list); + hlist_add_head(&cb->list, &plug->cb_list); } return cb; } @@ -1162,7 +1164,7 @@ EXPORT_SYMBOL(blk_check_plugged); void __blk_flush_plug(struct blk_plug *plug, bool from_schedule) { - if (!list_empty(&plug->cb_list)) + if (!hlist_empty(&plug->cb_list)) flush_plug_callbacks(plug, from_schedule); blk_mq_flush_plug_list(plug, from_schedule); /* diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c index 512746551f36..4a1b6f17067f 100644 --- a/drivers/md/raid1-10.c +++ b/drivers/md/raid1-10.c @@ -152,7 +152,7 @@ static inline bool raid1_add_bio_to_plug(struct mddev *mddev, struct bio *bio, plug = container_of(cb, struct raid1_plug_cb, cb); bio_list_add(&plug->pending, bio); if (++plug->count / MAX_PLUG_BIO >= copies) { - list_del(&cb->list); + hlist_del(&cb->list); cb->callback(cb, false); } diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ce6d057de2f0..f3105a519ef4 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -951,13 +951,13 @@ struct blk_plug { bool multiple_queues; bool has_elevator; - struct list_head cb_list; /* md requires an unplug callback */ + struct hlist_head cb_list; /* md requires an unplug callback */ }; struct blk_plug_cb; typedef void (*blk_plug_cb_fn)(struct blk_plug_cb *, bool); struct blk_plug_cb { - struct list_head list; + struct hlist_node list; blk_plug_cb_fn callback; void *data; };