From patchwork Sat Sep 14 17:07:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 13804441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B67BFC021A8 for ; Sat, 14 Sep 2024 17:07:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D6BF6B0082; Sat, 14 Sep 2024 13:07:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 185C66B0083; Sat, 14 Sep 2024 13:07:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0261E6B0088; Sat, 14 Sep 2024 13:07:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DEA536B0082 for ; Sat, 14 Sep 2024 13:07:27 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 576DCC213C for ; Sat, 14 Sep 2024 17:07:27 +0000 (UTC) X-FDA: 82563974934.19.5E40E51 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf27.hostedemail.com (Postfix) with ESMTP id 8D04F4000A for ; Sat, 14 Sep 2024 17:07:25 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="qgrESo/2"; spf=pass (imf27.hostedemail.com: domain of jlayton@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=jlayton@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726333615; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=odZkZJo7zbTgSpD61JoTnbkExErPRRGlLKPRz32iL48=; b=k4DECgLF9WrJumbyo2MUyL2TTUuboUv+zvX+6RIqc81V7+Vn8gn+z/bDsyTQuVqi8AOU/m dnIMs9dUz3ZdH0pK5PuXKXboDLWONmfi2etXPVZXw+aB9EidUlUSOwKFXHGkdzr9sTPEwj DbI84K9/UQaAabOKLvzHb/LB7KJuiUM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="qgrESo/2"; spf=pass (imf27.hostedemail.com: domain of jlayton@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=jlayton@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726333615; a=rsa-sha256; cv=none; b=wVlG1auQDjbHHDd6tIjX99J49xCvznF2EtAGPtzD//K2U7/fKegNyrhGsiw8Q6kMPeMSoy 4uzkD8JLaIIP3OIrEkbUz7l0SE7A4LfAFH1OJwi5Nwjd41jtfBPZhBsVxK5MuF46/2lXdM VVC5A1BZXD5jXIn7sGehXalsqjd1ohM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 22637A401E9; Sat, 14 Sep 2024 17:07:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08B09C4CECE; Sat, 14 Sep 2024 17:07:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1726333644; bh=I9/47O7qo7bvUG1tGenasxTgf1bKbVz1h75bJIuETII=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=qgrESo/2dKlnGWdwdJt4WBacgChSPGID4vFllkaU0ksLvfh8W0/erR8TcsEd20YuZ qy7LG7yQaw1+e/hwVRstO2c5SRLjUDVSP/ai1Epg+DZnh8JrRgfa5TP41U31y2xz/7 +080vYTYI1+3AEo46MOZstgKVgoVQmbrOFSFzlkobnc00Vjxiieldhagjt4wBVp973 XWYxtSPcLauiLWBgbApL12W40KgS+mjYM9Rf0aPgs+ds4DZXA4+rRSTEnRWVCNll2X J3CCfCt44JNNKu6dnZep+GyBvY2msx2KWB6EgyIY2/rZe6t72Qcb7qRfabSY4kKAsr o+N7Q0cpT2x6w== From: Jeff Layton Date: Sat, 14 Sep 2024 13:07:14 -0400 Subject: [PATCH v8 01/11] timekeeping: move multigrain timestamp floor handling into timekeeper MIME-Version: 1.0 Message-Id: <20240914-mgtime-v8-1-5bd872330bed@kernel.org> References: <20240914-mgtime-v8-0-5bd872330bed@kernel.org> In-Reply-To: <20240914-mgtime-v8-0-5bd872330bed@kernel.org> To: John Stultz , Thomas Gleixner , Stephen Boyd , Alexander Viro , Christian Brauner , Jan Kara , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Jonathan Corbet , Chandan Babu R , "Darrick J. Wong" , Theodore Ts'o , Andreas Dilger , Chris Mason , Josef Bacik , David Sterba , Hugh Dickins , Andrew Morton , Chuck Lever , Vadim Fedorenko Cc: Randy Dunlap , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-xfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-nfs@vger.kernel.org, linux-mm@kvack.org, Jeff Layton X-Mailer: b4 0.14.1 X-Developer-Signature: v=1; a=openpgp-sha256; l=5475; i=jlayton@kernel.org; h=from:subject:message-id; bh=I9/47O7qo7bvUG1tGenasxTgf1bKbVz1h75bJIuETII=; b=owEBbQKS/ZANAwAIAQAOaEEZVoIVAcsmYgBm5cLFY6lXZvZm8RxN6kOFAGtO8JT2riURAMDHv 52tncP7VVSJAjMEAAEIAB0WIQRLwNeyRHGyoYTq9dMADmhBGVaCFQUCZuXCxQAKCRAADmhBGVaC FVmqD/42vnYgXZg5mzwCTj8STzoscinyiV3K/zfYJZXBKlhsahthN4ivhC4Pb/lyArui3/IxAQq EWZuccCCsliVOR5fGIXxBnGCyGUE/NFyJcgQVEdAjlz/lglEXC0qzAgFyqnBFXLjHIO/vEBN9A+ 3YgW/1rz3+jDt7H0Kfaszw9349+fGDmnzIlYg0fLqOG1l9LIdTFY7d1tCWtOxFJOhGeP+m5EUna +jv01w18PUKFWAnYS5w3rpbXPSkmQt/sw6EeN19nRR3QwlkumZck7VVgKrkWPzk1k1fY1omGGeN uu8YXrAhrI48XJ65fOyuUw4jBALv+ImDZwQGejRRPsguzETloqOXZI7HFDS+gpo52sMT22HwmgI XrpgwBa1uVZVVqf02ZnjHftQRwVpHryUcccG4KCDzFW1ZkTKyIjUxpqhXcEAp6jR8/Roa58umQ1 qXMl1cHM2aIpIjH7vLxInGCDw/RpXZMST+q5+QwCvBnKs4WeJBuaI0XLBsBMmbfQhhqiKf3L+HF 7xIK1uDDqzl9gZXVjuD8xpxGxLyhcf6BoT+dbv1E7KV5V8C8OJtRfPiuB/r6fbvaeqXDQLdlNxv vHj65OVIB+mRWvY0NEVU9WSQgxeWZ24UWiflRgY4sxq6+zKOkdbJNn9cAkfgQhI/GS1tqCKU3Mq e6rjZuBfD33jY/w== X-Developer-Key: i=jlayton@kernel.org; a=openpgp; fpr=4BC0D7B24471B2A184EAF5D3000E684119568215 X-Rspam-User: X-Stat-Signature: aujqfbcn7hkbjhrh5a594rbd3uda5rwt X-Rspamd-Queue-Id: 8D04F4000A X-Rspamd-Server: rspam11 X-HE-Tag: 1726333645-532210 X-HE-Meta: U2FsdGVkX1/HQHdd6opPqcyV3qxyfvTKh2EN6+XQjVkMUvKHno7zy0yZGcgGed75iPGlKSkhUddNklQ/vxVOQIh6pdhqlDUO/aJhYl4eUpOH/jjQROjaP+AkctlkPp3gH/ceZNtTh6Hcm8YDNf1jyzHVZhUfO/jgAem14kI5TH9OkIINEEKHgGliGyWlTzZ/n2rFWRA3GL4R2qyNptNMMNlyGrqSwyK9EKfpwcYvKFesGgDy21gp4KJ2042G4P4AOuE61RPhNGd+HcXuVt4fY54MAiNhQmoX/7utH7lo2zWDM6m9PV1ACdFw8XV8TmbutFYoS/vZP8k4RnC89JqalkAUrIFNsIJzEmEvwcXJi6ax+KdHzB1McuMwL6TCN1p15sIfvkiEqxz1f12Ws+LWm0QIVJgl2itP3c3jnt42fEJ5A3GhYIiyBZuYzY4ZA03BjRVBOiVpEXhgubL5ju8tjNjS3kuOvjGvmiBGVA9WlXLWZ0nhfXvaX6w587CD/PAoldUlyAoxtocC1C2W0TAhwR4MtjdOC8BlVTvQyQqnVAoG11Ua+/IaNkPex78pSUF9n3dqfZnNWScOWHMCZGkTOizOeQGpctArGDUBLX7+Vk9CONEAL5YcX468wibWxVqAcD949LxM8SiAyg5xifvQEDy6B0IzNNCRiqXKNh/eaOJ4Hnf4VSZG6W80sgDkPNnejJQF/x3ejZ8Ot0lsDDp3m+lJMiFEHDdQ5FVmgCJQ0IoeYhUPVwJCfxR/IwUoC0XWX/el9mIqAZ5oMUcbsYCC4Dygi6yAf6ulfNaJRalAMyE3CuicNVU/1ICTR1r4zfOmkp473TCTLDHSSaDORIR+PQNNOmcUG4tfOedoFe/WD0Ul4JPTloauXJDtHCo/F3dro2dXsq2CzcPglhecNMCsHQVPvvhnSf03XQ/Xx9bBjDQNMUM9pnkU+oGQ8zreGnz46rmVmNt4t+m8dnvMQEt 90zLxei6 p+icBekep44hJZHqcNkDhfd4wwhO8JSCL22yh4fzyDIgzTz52vqBtMLFfECkpln2zcch/fpXdsB9RBKQbHaIm7CHRdYr65qFRy5exm5IBhDo2p2GMmXUU2nR4nybNJ3QaB7ieDa7yQESuXNfXQlGnoX/hgAY4icRDMew0vC4FgkJGgDL3HCqYw05XclNYSqU1LspFJf+uRex/mAQrYM0MsCDNU2usjeDsJDpfxG/L4ghEr1vbQ8ubQi6b0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For multigrain timestamps, we must keep track of the latest timestamp that has ever been handed out, and never hand out a coarse time below that value. Add a static singleton atomic64_t into timekeeper.c that we can use to keep track of the latest fine-grained time ever handed out. This is tracked as a monotonic ktime_t value to ensure that it isn't affected by clock jumps. Add two new public interfaces: - ktime_get_coarse_real_ts64_mg() fills a timespec64 with the later of the coarse-grained clock and the floor time - ktime_get_real_ts64_mg() gets the fine-grained clock value, and tries to swap it into the floor. A timespec64 is filled with the result. Since the floor is global, we take great pains to avoid updating it unless it's absolutely necessary. If we do the cmpxchg and find that the value has been updated since we fetched it, then we discard the fine-grained time that was fetched in favor of the recent update. To maximize the window of this occurring when multiple tasks are racing to update the floor, ktime_get_coarse_real_ts64_mg returns a cookie value that represents the state of the floor tracking word, and ktime_get_real_ts64_mg accepts a cookie value that it uses as the "old" value when calling cmpxchg(). Signed-off-by: Jeff Layton --- include/linux/timekeeping.h | 4 +++ kernel/time/timekeeping.c | 82 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/include/linux/timekeeping.h b/include/linux/timekeeping.h index fc12a9ba2c88..7aa85246c183 100644 --- a/include/linux/timekeeping.h +++ b/include/linux/timekeeping.h @@ -45,6 +45,10 @@ extern void ktime_get_real_ts64(struct timespec64 *tv); extern void ktime_get_coarse_ts64(struct timespec64 *ts); extern void ktime_get_coarse_real_ts64(struct timespec64 *ts); +/* Multigrain timestamp interfaces */ +extern void ktime_get_coarse_real_ts64_mg(struct timespec64 *ts); +extern void ktime_get_real_ts64_mg(struct timespec64 *ts); + void getboottime64(struct timespec64 *ts); /* diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 5391e4167d60..16937242b904 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -114,6 +114,13 @@ static struct tk_fast tk_fast_raw ____cacheline_aligned = { .base[1] = FAST_TK_INIT, }; +/* + * This represents the latest fine-grained time that we have handed out as a + * timestamp on the system. Tracked as a monotonic ktime_t, and converted to the + * realtime clock on an as-needed basis. + */ +static __cacheline_aligned_in_smp atomic64_t mg_floor; + static inline void tk_normalize_xtime(struct timekeeper *tk) { while (tk->tkr_mono.xtime_nsec >= ((u64)NSEC_PER_SEC << tk->tkr_mono.shift)) { @@ -2394,6 +2401,81 @@ void ktime_get_coarse_real_ts64(struct timespec64 *ts) } EXPORT_SYMBOL(ktime_get_coarse_real_ts64); +/** + * ktime_get_coarse_real_ts64_mg - get later of coarse grained time or floor + * @ts: timespec64 to be filled + * + * Adjust floor to realtime and compare it to the coarse time. Fill + * @ts with the latest one. Note that this is a filesystem-specific + * interface and should be avoided outside of that context. + */ +void ktime_get_coarse_real_ts64_mg(struct timespec64 *ts) +{ + struct timekeeper *tk = &tk_core.timekeeper; + u64 floor = atomic64_read(&mg_floor); + ktime_t f_real, offset, coarse; + unsigned int seq; + + WARN_ON(timekeeping_suspended); + + do { + seq = read_seqcount_begin(&tk_core.seq); + *ts = tk_xtime(tk); + offset = *offsets[TK_OFFS_REAL]; + } while (read_seqcount_retry(&tk_core.seq, seq)); + + coarse = timespec64_to_ktime(*ts); + f_real = ktime_add(floor, offset); + if (ktime_after(f_real, coarse)) + *ts = ktime_to_timespec64(f_real); +} +EXPORT_SYMBOL_GPL(ktime_get_coarse_real_ts64_mg); + +/** + * ktime_get_real_ts64_mg - attempt to update floor value and return result + * @ts: pointer to the timespec to be set + * + * Get a current monotonic fine-grained time value and attempt to swap + * it into the floor. @ts will be filled with the resulting floor value, + * regardless of the outcome of the swap. Note that this is a filesystem + * specific interface and should be avoided outside of that context. + */ +void ktime_get_real_ts64_mg(struct timespec64 *ts, u64 cookie) +{ + struct timekeeper *tk = &tk_core.timekeeper; + ktime_t old = atomic64_read(&mg_floor); + ktime_t offset, mono; + unsigned int seq; + u64 nsecs; + + WARN_ON(timekeeping_suspended); + + do { + seq = read_seqcount_begin(&tk_core.seq); + + ts->tv_sec = tk->xtime_sec; + mono = tk->tkr_mono.base; + nsecs = timekeeping_get_ns(&tk->tkr_mono); + offset = *offsets[TK_OFFS_REAL]; + } while (read_seqcount_retry(&tk_core.seq, seq)); + + mono = ktime_add_ns(mono, nsecs); + + if (atomic64_try_cmpxchg(&mg_floor, &old, mono)) { + ts->tv_nsec = 0; + timespec64_add_ns(ts, nsecs); + } else { + /* + * Something has changed mg_floor since "old" was + * fetched. "old" has now been updated with the + * current value of mg_floor, so use that to return + * the current coarse floor value. + */ + *ts = ktime_to_timespec64(ktime_add(old, offset)); + } +} +EXPORT_SYMBOL_GPL(ktime_get_real_ts64_mg); + void ktime_get_coarse_ts64(struct timespec64 *ts) { struct timekeeper *tk = &tk_core.timekeeper;