From patchwork Fri Oct 27 22:25:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Salyzyn X-Patchwork-Id: 10030729 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7EC146032C for ; Fri, 27 Oct 2017 22:27:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58F4F28FE7 for ; Fri, 27 Oct 2017 22:27:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D9DA28FEB; Fri, 27 Oct 2017 22:27:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D919428FE7 for ; Fri, 27 Oct 2017 22:27:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=U0dPFcRDto758DzaWuzxCaB9z545gKWhp8Kr9YuRi9Y=; b=YZG 9k0jJx5WpQV1pabTCUIMNIISkz2nejV1KVjS2zb0A18wuZfAqrQMfojd9jnoz8HC8nvS++vVyMu5E lSIyqBz1tKvafp9w848gz7fc04w/0XIujRTicTDKVQcNoOsTHsgQJN2jCX2Lrw/Sj24e7E0zP251E L4FDe/yST0ug+37O9bYaGD2Rohi1FeMpHDA6DYJDBPCFKvzFAQnWSX3z3Py0RlNGLP75nNCE++np2 SjUyDy7G2vBx7xbtfgBuoK6EBNVZ6Edl6cuf+SG0ypsqYnk6A6mILGPqdhOxcihc6Wa2Rr1gE3kxN 6Sa+37NdZ62QMwaJ2vr8pN9/6om/qPA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e8D5l-0003TE-1o; Fri, 27 Oct 2017 22:27:05 +0000 Received: from mail-io0-x241.google.com ([2607:f8b0:4001:c06::241]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e8D4r-0002t2-2y for linux-arm-kernel@lists.infradead.org; Fri, 27 Oct 2017 22:26:32 +0000 Received: by mail-io0-x241.google.com with SMTP id e89so15709179ioi.11 for ; Fri, 27 Oct 2017 15:25:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=iwygxAPWTinaD4eCjtDP7ZH0qRxcEmN+v9nPrWtfZ88=; b=lEvV8NkuwsJd57uXKriD0gBGK0e1tKWfNVOQ65ew5QMXXY3uTu/48SEeVS+RvT5ofZ cxgDCTJGrkclSI8U408DirDtmry1svb9CRnSLZ/DNvEinrJRgJIZTav7qSB6/Sst2w+D lxErWrv/h+fofzYSCO3NueQ1EsCMRhre1bRQRZbxsPkbUC1/8/huWjz7sz4yxxyv3aXg szWhDIH5WwmY92knEX8LA1WByZsH1J3cSDjqMTDukPY5/pfUlneRKP4Y2WARYJSXZbqh A1KH5Ga8vcmYaGRUtS9RdxLQlShKOZSST/ShMthSn756RJDL/e8LwRTpy/YKim3W4N27 /K3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=iwygxAPWTinaD4eCjtDP7ZH0qRxcEmN+v9nPrWtfZ88=; b=NeGWdbm8X+t1JOC8du6acsojrYSbZiFiqLPq+LxKJ9193VQBwmu0TLBEqzwG7CZybh XqERtipta22BXmXuSToEecw7PAgP2YalZm0B3sGfao5vqYU5xF7QMuBHmC8EMegkjDat NMrN/DNDpR9EWms9KUecHwCcbqWKc7KBbDZUySAW2VJ4GSa6HMRkpqEC7q2oBy9x81NM p8K2MNNPh5+nnAfNBPR18WYklPUau73g0gv8vQsNjQf+5y7s+Bt6sgJqWVYbiBOcC20f lirAQdX3E5ENwqWACYIUtfrZNbE23Vsz8Qs894JOL5iD/ToSNJ2bpanYaEBSf/M/rKiD VF4w== X-Gm-Message-State: AMCzsaUXFV7sGgv7wHBchnNm1jK1BoymRt+juDv7GaBx911z1O3Lxv62 naBC2xNQpUTZNvhA618NuJZjeA== X-Google-Smtp-Source: ABhQp+S0Q6TgBktwRskKR0nYehICJfDIAzJBLP47jur2QOk81CDaSsL97JzJTJi4KfGJ4cX4L11BjQ== X-Received: by 10.36.250.139 with SMTP id v133mr2635035ith.13.1509143148377; Fri, 27 Oct 2017 15:25:48 -0700 (PDT) Received: from nebulus.mtv.corp.google.com ([100.98.120.17]) by smtp.gmail.com with ESMTPSA id i201sm1379899ita.32.2017.10.27.15.25.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 27 Oct 2017 15:25:47 -0700 (PDT) From: Mark Salyzyn To: linux-kernel@vger.kernel.org Subject: [PATCH v3 05/12] arm: vdso: do calculations outside reader loops Date: Fri, 27 Oct 2017 15:25:40 -0700 Message-Id: <20171027222543.57279-1-salyzyn@android.com> X-Mailer: git-send-email 2.15.0.rc2.357.g7e34df9404-goog X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171027_152609_658396_66599178 X-CRM114-Status: GOOD ( 14.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Laura Abbott , Kees Cook , Ard Biesheuvel , Catalin Marinas , Kevin Brodsky , Will Deacon , Russell King , Andy Lutomirski , Mark Salyzyn , James Morse , Andrew Pinski , Dmitry Safonov , Andy Gross , John Stultz , Mark Salyzyn , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP In variable timer reading loops, pick up just the values until all are synchronized, then outside of loop pick up cntvct and perform calculations to determine final offset, shifted and multiplied output value. This replaces get_ns with get_clock_shifted_nsec as cntvct reader. Signed-off-by: Mark Salyzyn Cc: James Morse Cc: Russell King Cc: Catalin Marinas Cc: Will Deacon Cc: Andy Lutomirski Cc: Dmitry Safonov Cc: John Stultz Cc: Mark Rutland Cc: Laura Abbott Cc: Kees Cook Cc: Ard Biesheuvel Cc: Andy Gross Cc: Kevin Brodsky Cc: Andrew Pinski Cc: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org v2: - split first CL into 5 of 7 pieces v3: - rebase (unchanged) --- arch/arm/vdso/vgettimeofday.c | 95 ++++++++++++++++++++++++++++++------------- 1 file changed, 67 insertions(+), 28 deletions(-) diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 71003a1997c4..7fcc8cfcb7df 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -99,28 +99,38 @@ static notrace int do_monotonic_coarse(const struct vdso_data *vd, #ifdef CONFIG_ARM_ARCH_TIMER -static __always_inline notrace u64 get_ns(const struct vdso_data *vd) +/* + * Returns the clock delta, in nanoseconds left-shifted by the clock + * shift. + */ +static __always_inline notrace u64 get_clock_shifted_nsec(const u64 cycle_last, + const u32 mult, + const u64 mask) { - u64 cycle_delta; - u64 cycle_now; - u64 nsec; - - cycle_now = __arch_counter_get_cntvct(); + u64 res; - cycle_delta = (cycle_now - vd->cs_cycle_last) & vd->cs_mask; + /* Read the virtual counter. */ + res = __arch_counter_get_cntvct(); - nsec = (cycle_delta * vd->cs_mono_mult) + vd->xtime_clock_snsec; - nsec >>= vd->cs_shift; + res = res - cycle_last; - return nsec; + res &= mask; + return res * mult; } /* Code size doesn't matter (vdso is 4k/16k/64k anyway) and this is faster. */ static __always_inline notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) { - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->xtime_clock_sec) sec; do { seq = vdso_read_begin(vd); @@ -128,13 +138,24 @@ static __always_inline notrace int do_realtime(const struct vdso_data *vd, if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - } while (vdso_read_retry(vd, seq)); + mult = vd->cs_mono_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif + + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs); + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } @@ -142,9 +163,16 @@ static __always_inline notrace int do_realtime(const struct vdso_data *vd, static __always_inline notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) { - struct timespec tomono; - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->wtm_clock_nsec) wtm_nsec; + typeof(ts->tv_sec) sec; do { seq = vdso_read_begin(vd); @@ -152,17 +180,28 @@ static __always_inline notrace int do_monotonic(const struct vdso_data *vd, if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - tomono.tv_sec = vd->wtm_clock_sec; - tomono.tv_nsec = vd->wtm_clock_nsec; + mult = vd->cs_mono_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif - } while (vdso_read_retry(vd, seq)); + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_sec += tomono.tv_sec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs + tomono.tv_nsec); + sec += vd->wtm_clock_sec; + wtm_nsec = vd->wtm_clock_nsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + nsec += wtm_nsec; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; }