From patchwork Fri Sep 13 13:54:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 13803487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 524D1FA3758 for ; Fri, 13 Sep 2024 13:54:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0F416B00CC; Fri, 13 Sep 2024 09:54:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C95E66B00CD; Fri, 13 Sep 2024 09:54:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0F4A6B00D0; Fri, 13 Sep 2024 09:54:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9447B6B00CC for ; Fri, 13 Sep 2024 09:54:27 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C91E61C6D75 for ; Fri, 13 Sep 2024 13:54:26 +0000 (UTC) X-FDA: 82559859732.24.8DB48A0 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf07.hostedemail.com (Postfix) with ESMTP id 26AC140017 for ; Fri, 13 Sep 2024 13:54:25 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=MSnStM37; spf=pass (imf07.hostedemail.com: domain of jlayton@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=jlayton@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726235524; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nyu6mD19JtxubxbMpdD0/wdnWtCgS+fUrtUr8fO0q60=; b=7zlALp8gUN8PWg9uupTS9YULYhq7VErCpOAXb6pLmuWdB0R4N7Sg4lqZqiIt7CsKIq9sQM /VJNWg0l5v3M8bxQCW5PTLjlyteBin8goTxykTtGjKsQWA9mG1bMycdbH7bx9MwIJ70BuJ mix6ulPnPJOoie8pSdsTKPrX+M6xHL4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=MSnStM37; spf=pass (imf07.hostedemail.com: domain of jlayton@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=jlayton@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726235524; a=rsa-sha256; cv=none; b=fqc2s6WoOKwGET4J8Qn571PHAc94UwMazfjR9/IMsTSDZ/0b79sT9sezkzyOuQgr+g9ly/ zN9d/pN25KUXk+L9dtCPjMISpmnNywC+nu8wccd6iZwjGS0ppC6r6buVH68RjTQdbzAiDT SI1gT0P5UF7QNln50KbrR/w4S87NhfQ= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 6B390A44784; Fri, 13 Sep 2024 13:54:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1FF1C4CECE; Fri, 13 Sep 2024 13:54:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1726235663; bh=1wQqjayOZU9OU7OIyp79413cwp1jIKED+/ddvabpxRE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=MSnStM37sPnp9DXk4F6EIH6b72VjVzSahHG8G3GrAM200gv2K3Q39nAW3S5z3EOtE YdNIC+JimHrlIa7gtqOhHM521Fior7jUBndNkAbd529aesrqar5Nwxs7WMIQ1ynSLt qyp+qe8ZzfaDvLGTiVAMQR3llWrqqGicXXd510d8kigpkfYCkdYbPmnFL3hn6uh/Ky B+htVPC9d6ON+9bsUM8ujYAuur0rLfdN14R7QQrFOKUU/8LHZCEe6GPNCR+eU0Tunv caCGQZZBsW3OvNBhbBOihZkDcWUqE0qreZ8YpbezQZVWmOMSC/JMhMgDnV9wpxSoYW NZ50YB4avb+8Q== From: Jeff Layton Date: Fri, 13 Sep 2024 09:54:10 -0400 Subject: [PATCH v7 01/11] timekeeping: move multigrain timestamp floor handling into timekeeper MIME-Version: 1.0 Message-Id: <20240913-mgtime-v7-1-92d4020e3b00@kernel.org> References: <20240913-mgtime-v7-0-92d4020e3b00@kernel.org> In-Reply-To: <20240913-mgtime-v7-0-92d4020e3b00@kernel.org> To: John Stultz , Thomas Gleixner , Stephen Boyd , Alexander Viro , Christian Brauner , Jan Kara , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Jonathan Corbet , Chandan Babu R , "Darrick J. Wong" , Theodore Ts'o , Andreas Dilger , Chris Mason , Josef Bacik , David Sterba , Hugh Dickins , Andrew Morton , Chuck Lever , Vadim Fedorenko Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-xfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-nfs@vger.kernel.org, linux-mm@kvack.org, Jeff Layton X-Mailer: b4 0.14.1 X-Developer-Signature: v=1; a=openpgp-sha256; l=5382; i=jlayton@kernel.org; h=from:subject:message-id; bh=1wQqjayOZU9OU7OIyp79413cwp1jIKED+/ddvabpxRE=; b=owEBbQKS/ZANAwAIAQAOaEEZVoIVAcsmYgBm5EQHdrxoA4Zv70eWUKOVfFaSb4FDc6+mmDM7U djCsOXWJIqJAjMEAAEIAB0WIQRLwNeyRHGyoYTq9dMADmhBGVaCFQUCZuREBwAKCRAADmhBGVaC FahsD/kBt0wSc8RQuJxojIJblnZIg2g0aJ85YqPnmoTQvcCecxKf6H0dOsXyVro9MKCuVISYgOy RfqTSz7o4LRR/lbBCwc7T2w3n0WRmW4SM2LpjyJBP7pxAvnzVaoXcBkkxE0OOEl6QnBnGqBvkah n3pXUc0xiOQarhuwPSzTNc8oRsz0UD6nk5p0pFdyFwnsi9CeVXkWat6VjuOW/iKnvjdz0mW3lAs uqb8yLh139vG5amFjLrdGvlUc1zuSoFqy+psf+DNCdq/thiUJdueGvjN/pNooci0W4j09oXs5bV dCyMx7CanVzHwG8k8HP/TBR/HFXopkhtTMZavqvTmNx/LqK1IV1m5g+4mMNZn1zEiv7pjMzgzVT Op7Rk8bqkRddOdSlWbUPnE1bSaSRE7cd2RDX9BThB6UT0a3eiePLw1J9me1BdjWpHwdWek0OuRI ar8r0yN4lrmoUl3qWKFCyqLOAXsxqzihaAbJexTcpSBF+8t+WwD+laHMrtLMWKNVbbJAdhiSQxv w9IXUdEcB/NTRfacm1UeULZsgoMMXqWNZ2WFORMx1wgI3Df6PpBnwaoQ+Ot5P248reTCTnkKBdO 1BPzWFcTt+qzByEH0lJMjIT6kNjOL8nU5Z52XyHttqAYCIyPpWxjWtaSAsRQK+TWrUCtfVw3nbp ko5CuOLpgy0RNxA== X-Developer-Key: i=jlayton@kernel.org; a=openpgp; fpr=4BC0D7B24471B2A184EAF5D3000E684119568215 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 26AC140017 X-Stat-Signature: 1xuhjiwn6ewcjoop9uaimb1rohdkfofp X-Rspam-User: X-HE-Tag: 1726235665-730774 X-HE-Meta: U2FsdGVkX18UXZ9tmK10uxSM/uKG8mOh5NAbiWYmby2BGsZJatUcvaRqr4hATTcjDydnYdyo+/hKHRDWN9sz8tnGkPZKz0Pe7o/oI8KnCzY0sEW8T+OyLxFx+zsPeTDyxNoPKt5w4laBrPEZ8QYhcgmTvC8Ft+PzIsnl79wQrfIsxqzHiXeZ5yAF5h5NiI+lbmZIPeuedqu8C/97sK2fA0MaCtG6Wc9IywmpQGI1oIBAAd8uenWzR3N+/qGVMQFen2eU33HZuL3JfNyenkE4iahMI2Wf2/0MlMbya2ZmOWqXa2veGrqrMNfz0oEcmMvSwDsb/WnRQnPq3Id8tRFme1Z3tYi9Xb498E3JdIWHdcU00dJh/9g6H7+jW6Hm3DCYd7u9EUnaZOK3X1bcVX5j64Ctc+LVUkhP7eKApxkFiu6JpuVjBaaveib023/8F1s6/gAtZI9ayfOMYdS0UcoysfX9cqM5SgCN9rBKQ/MYFNvFuOmx4aXqBlfFM8bon+nkPCI743+t/Khqph0D70pFAXqz/0BjqTQ+LcvwhdTstEy02KjxWnrW6jmwKUiluLJSLxjgmWmNuT/nFx9hUkPu8Df2RMjXc6iLGn6mZPKHizfFX+nf2xCBYjXVq9a4DbyKxWZr7iw9OZKJ5SCy8aEXjBxe43IvJylxIBLiZ8k/0Cf5XDX0FAMHAe/Xup8Kq5RBYDiCEhHNsev+RVWT5n3P3krqhp6JlglHmfiuXMUayV5POlEg2lwgqQ50CW8iOxRu76YRW1i0UzzaRg50dX2oJWIepYUo2XPVyOZE8j589rX9WdSYbgkAkHtOXsO7ubcW0oGrmlKHAjr4TQapwpILu8u4Wgxza0jGEISlQd6wCbX6kXOLV4zSSnZvPqU6xE12MXEfqQ54vRWItWUoC21vnaW3j8aTgd+dCUiNJjT08mI3dslbqnS1FBSLKgnILqOwoQo1Q5rolxXTQq69yob sO2hbc4T 5V46KcZbq0BDVsGB6Xp1tfQ7gDj68sapH0HUxRsjJNS4e0vpG9ixrDdieQ6lSF4Q+HQFrP33Mv1qbhsnCMB8q1fRPkNJvo/tsoZxP4FWIB8Ve//CCtcj7QdlzCrxoGVrGYwNMHg4S/RzE7X/SFxIUwiXak7FXjixmyQREx9KCEWnviKJF6HNbjT6Oj9hf6Vqb/hS3XPFY57JWBQHUF/8AKBH1cqzhFSIZLHlRwaaSiyrxv8qoL+mIgxz3HQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For multigrain timestamps, we must keep track of the latest timestamp that has ever been handed out, and never hand out a coarse time below that value. Add a static singleton atomic64_t into timekeeper.c that we can use to keep track of the latest fine-grained time ever handed out. This is tracked as a monotonic ktime_t value to ensure that it isn't affected by clock jumps. Add two new public interfaces: - ktime_get_coarse_real_ts64_mg() fills a timespec64 with the later of the coarse-grained clock and the floor time - ktime_get_real_ts64_mg() gets the fine-grained clock value, and tries to swap it into the floor. A timespec64 is filled with the result. Since the floor is global, we take great pains to avoid updating it unless it's absolutely necessary. If we do the cmpxchg and find that the value has been updated since we fetched it, then we discard the fine-grained time that was fetched in favor of the recent update. To maximize the window of this occurring when multiple tasks are racing to update the floor, ktime_get_coarse_real_ts64_mg returns a cookie value that represents the state of the floor tracking word, and ktime_get_real_ts64_mg accepts a cookie value that it uses as the "old" value when calling cmpxchg(). Signed-off-by: Jeff Layton --- include/linux/timekeeping.h | 4 +++ kernel/time/timekeeping.c | 81 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+) diff --git a/include/linux/timekeeping.h b/include/linux/timekeeping.h index fc12a9ba2c88..cf2293158c65 100644 --- a/include/linux/timekeeping.h +++ b/include/linux/timekeeping.h @@ -45,6 +45,10 @@ extern void ktime_get_real_ts64(struct timespec64 *tv); extern void ktime_get_coarse_ts64(struct timespec64 *ts); extern void ktime_get_coarse_real_ts64(struct timespec64 *ts); +/* Multigrain timestamp interfaces */ +extern u64 ktime_get_coarse_real_ts64_mg(struct timespec64 *ts); +extern void ktime_get_real_ts64_mg(struct timespec64 *ts, u64 cookie); + void getboottime64(struct timespec64 *ts); /* diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c index 5391e4167d60..ee11006a224f 100644 --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -114,6 +114,13 @@ static struct tk_fast tk_fast_raw ____cacheline_aligned = { .base[1] = FAST_TK_INIT, }; +/* + * This represents the latest fine-grained time that we have handed out as a + * timestamp on the system. Tracked as a monotonic ktime_t, and converted to the + * realtime clock on an as-needed basis. + */ +static __cacheline_aligned_in_smp atomic64_t mg_floor; + static inline void tk_normalize_xtime(struct timekeeper *tk) { while (tk->tkr_mono.xtime_nsec >= ((u64)NSEC_PER_SEC << tk->tkr_mono.shift)) { @@ -2394,6 +2401,80 @@ void ktime_get_coarse_real_ts64(struct timespec64 *ts) } EXPORT_SYMBOL(ktime_get_coarse_real_ts64); +/** + * ktime_get_coarse_real_ts64_mg - get later of coarse grained time or floor + * @ts: timespec64 to be filled + * + * Adjust floor to realtime and compare it to the coarse time. Fill + * @ts with the latest one. Returns opaque cookie suitable for passing + * to ktime_get_real_ts64_mg(). + */ +u64 ktime_get_coarse_real_ts64_mg(struct timespec64 *ts) +{ + struct timekeeper *tk = &tk_core.timekeeper; + u64 floor = atomic64_read(&mg_floor); + ktime_t f_real, offset, coarse; + unsigned int seq; + + WARN_ON(timekeeping_suspended); + + do { + seq = read_seqcount_begin(&tk_core.seq); + *ts = tk_xtime(tk); + offset = *offsets[TK_OFFS_REAL]; + } while (read_seqcount_retry(&tk_core.seq, seq)); + + coarse = timespec64_to_ktime(*ts); + f_real = ktime_add(floor, offset); + if (ktime_after(f_real, coarse)) + *ts = ktime_to_timespec64(f_real); + return floor; +} +EXPORT_SYMBOL_GPL(ktime_get_coarse_real_ts64_mg); + +/** + * ktime_get_real_ts64_mg - attempt to update floor value and return result + * @ts: pointer to the timespec to be set + * @cookie: opaque cookie from earlier call to ktime_get_coarse_real_ts64_mg() + * + * Get a current monotonic fine-grained time value and attempt to swap + * it into the floor using @cookie as the "old" value. @ts will be + * filled with the resulting floor value, regardless of the outcome of + * the swap. + */ +void ktime_get_real_ts64_mg(struct timespec64 *ts, u64 cookie) +{ + struct timekeeper *tk = &tk_core.timekeeper; + ktime_t offset, mono, old = (ktime_t)cookie; + unsigned int seq; + u64 nsecs; + + WARN_ON(timekeeping_suspended); + + do { + seq = read_seqcount_begin(&tk_core.seq); + + ts->tv_sec = tk->xtime_sec; + mono = tk->tkr_mono.base; + nsecs = timekeeping_get_ns(&tk->tkr_mono); + offset = *offsets[TK_OFFS_REAL]; + } while (read_seqcount_retry(&tk_core.seq, seq)); + + mono = ktime_add_ns(mono, nsecs); + + if (atomic64_try_cmpxchg(&mg_floor, &old, mono)) { + ts->tv_nsec = 0; + timespec64_add_ns(ts, nsecs); + } else { + /* + * Something has changed mg_floor since "old" was + * fetched. That value is just as valid, so accept it. + */ + *ts = ktime_to_timespec64(ktime_add(old, offset)); + } +} +EXPORT_SYMBOL(ktime_get_real_ts64_mg); + void ktime_get_coarse_ts64(struct timespec64 *ts) { struct timekeeper *tk = &tk_core.timekeeper;