From patchwork Tue Mar 15 17:35:10 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 8590871 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 20358C0553 for ; Tue, 15 Mar 2016 17:38:51 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 0D88C202A1 for ; Tue, 15 Mar 2016 17:38:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA1102021B for ; Tue, 15 Mar 2016 17:38:48 +0000 (UTC) Received: from localhost ([::1]:50300 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1afsvg-0003sa-AR for patchwork-qemu-devel@patchwork.kernel.org; Tue, 15 Mar 2016 13:38:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57790) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1afssR-0006VZ-9H for qemu-devel@nongnu.org; Tue, 15 Mar 2016 13:35:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1afssN-0003nw-VS for qemu-devel@nongnu.org; Tue, 15 Mar 2016 13:35:27 -0400 Received: from mail-wm0-x231.google.com ([2a00:1450:400c:c09::231]:34874) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1afssN-0003nn-M0 for qemu-devel@nongnu.org; Tue, 15 Mar 2016 13:35:23 -0400 Received: by mail-wm0-x231.google.com with SMTP id l68so159976902wml.0 for ; Tue, 15 Mar 2016 10:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=rTAgKdycpaDHvTX0JlCRNxjeNnYXT2gPwYRj0GqJYrU=; b=U3YiZAcxvYTSXEkHjKSXI8sNrO7gzZtdCk1EitKytFe/84IbyhP1m++oQ1BHKryh11 aNCZncqihZUTp2b8iWt9p3ExQzrdT7NFH7R0zwvy8iT0JetkHNmV5Kr9sMfFzagiR50q IQdN7yaFKdjXH1SYO/HEfzwDOV1F2W4zqGYam/Ba326seaxV1hwNbfsJtbuKmY2+oFqm 7/LNBF45OSDEDp/JdgYKCig6/Hw2DLOLKuRVSVHb1cu+qyT0Vro0sZinpqfAgzdEsNid R4Sly7xL9tOHBTYgbxaSrTNmRMpJ8knd3YpThi22y8wVtg7/BYsg9idj/mJtKSQEc471 Bzjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=rTAgKdycpaDHvTX0JlCRNxjeNnYXT2gPwYRj0GqJYrU=; b=C0CietEg99duHJiesEmGC2PM/1sTL/IWiPVmld6/kZAGTREQwzJHNW8kiVXo6AS54e xB4y6Fjh3sQXVllRBujOPvP+xyOqNFnetOA4dY0d4yyazpUQUfDyrStPwguelRBQzqOZ b7sGyBZSV3vSwXAIPsO8hPiLdL1gOdBiw6Lx/HWKQv4FEdM6+XTBJaYG00rimKzPW5lq UiLFL5tbjWzAh9mURXsfzA6Pl8yvtCba4guVOI4/oP1XS2kTmALR72ucBukGDNYmt+kK ZQ3BYU7DetjpLFjPUbvfgSayaxBBG4jM74KXLykZhH/9pQVV5rgemGkUCNbDlmB3zwD3 zA3g== X-Gm-Message-State: AD7BkJKKqU7bDfaTJGL/53NtSnrBn6FX1jI2a+W2T+44c2yb3pHrVwfprmKPUO9e8ZhnrA== X-Received: by 10.194.24.39 with SMTP id r7mr29867020wjf.86.1458063322991; Tue, 15 Mar 2016 10:35:22 -0700 (PDT) Received: from 640k.lan (94-39-161-17.adsl-ull.clienti.tiscali.it. [94.39.161.17]) by smtp.gmail.com with ESMTPSA id z127sm21827680wme.5.2016.03.15.10.35.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Mar 2016 10:35:22 -0700 (PDT) From: Paolo Bonzini To: qemu-devel@nongnu.org Date: Tue, 15 Mar 2016 18:35:10 +0100 Message-Id: <1458063310-128525-9-git-send-email-pbonzini@redhat.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1458063310-128525-1-git-send-email-pbonzini@redhat.com> References: <1458063310-128525-1-git-send-email-pbonzini@redhat.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::231 Cc: Pavel Dovgalyuk Subject: [Qemu-devel] [PULL 8/8] icount: decouple warp calls X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Pavel Dovgalyuk qemu_clock_warp function is called to update virtual clock when CPU is sleeping. This function includes replay checkpoint to make execution deterministic in icount mode. Record/replay module flushes async event queue at checkpoints. Some of the events (e.g., block devices operations) include interaction with hardware. E.g., APIC polled by block devices sets one of IRQ flags. Flag to be set depends on currently executed thread (CPU or iothread). Therefore in replay mode we have to process the checkpoints in the same thread as they were recorded. qemu_clock_warp function (and its checkpoint) may be called from different thread. This patch decouples two different execution cases of this function: call when CPU is sleeping from iothread and call from cpu thread to update virtual clock. First task is performed by qemu_start_warp_timer function. It sets warp timer event to the moment of nearest pending virtual timer. Second function (qemu_account_warp_timer) is called from cpu thread before execution of the code. It advances virtual clock by adding the length of period while CPU was sleeping. Signed-off-by: Pavel Dovgalyuk Message-Id: <20160310115609.4812.44986.stgit@PASHA-ISP> [Update docs. - Paolo] Signed-off-by: Paolo Bonzini --- cpus.c | 58 ++++++++++++++++++++++++++++--------------------- docs/replay.txt | 21 +++++++++++++----- include/qemu/timer.h | 7 +++--- include/sysemu/replay.h | 3 ++- main-loop.c | 2 +- qemu-timer.c | 4 +++- stubs/clock-warp.c | 2 +- 7 files changed, 58 insertions(+), 39 deletions(-) diff --git a/cpus.c b/cpus.c index 85d0f87..4052be5 100644 --- a/cpus.c +++ b/cpus.c @@ -370,9 +370,12 @@ static void icount_warp_rt(void) } } -static void icount_dummy_timer(void *opaque) +static void icount_timer_cb(void *opaque) { - (void)opaque; + /* No need for a checkpoint because the timer already synchronizes + * with CHECKPOINT_CLOCK_VIRTUAL_RT. + */ + icount_warp_rt(); } void qtest_clock_warp(int64_t dest) @@ -396,17 +399,12 @@ void qtest_clock_warp(int64_t dest) qemu_clock_notify(QEMU_CLOCK_VIRTUAL); } -void qemu_clock_warp(QEMUClockType type) +void qemu_start_warp_timer(void) { int64_t clock; int64_t deadline; - /* - * There are too many global variables to make the "warp" behavior - * applicable to other clocks. But a clock argument removes the - * need for if statements all over the place. - */ - if (type != QEMU_CLOCK_VIRTUAL || !use_icount) { + if (!use_icount) { return; } @@ -418,29 +416,17 @@ void qemu_clock_warp(QEMUClockType type) } /* warp clock deterministically in record/replay mode */ - if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP)) { + if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_START)) { return; } - if (icount_sleep) { - /* - * If the CPUs have been sleeping, advance QEMU_CLOCK_VIRTUAL timer now. - * This ensures that the deadline for the timer is computed correctly - * below. - * This also makes sure that the insn counter is synchronized before - * the CPU starts running, in case the CPU is woken by an event other - * than the earliest QEMU_CLOCK_VIRTUAL timer. - */ - icount_warp_rt(); - timer_del(icount_warp_timer); - } if (!all_cpu_threads_idle()) { return; } if (qtest_enabled()) { /* When testing, qtest commands advance icount. */ - return; + return; } /* We want to use the earliest deadline from ALL vm_clocks */ @@ -496,6 +482,28 @@ void qemu_clock_warp(QEMUClockType type) } } +static void qemu_account_warp_timer(void) +{ + if (!use_icount || !icount_sleep) { + return; + } + + /* Nothing to do if the VM is stopped: QEMU_CLOCK_VIRTUAL timers + * do not fire, so computing the deadline does not make sense. + */ + if (!runstate_is_running()) { + return; + } + + /* warp clock deterministically in record/replay mode */ + if (!replay_checkpoint(CHECKPOINT_CLOCK_WARP_ACCOUNT)) { + return; + } + + timer_del(icount_warp_timer); + icount_warp_rt(); +} + static bool icount_state_needed(void *opaque) { return use_icount; @@ -624,7 +632,7 @@ void configure_icount(QemuOpts *opts, Error **errp) icount_sleep = qemu_opt_get_bool(opts, "sleep", true); if (icount_sleep) { icount_warp_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL_RT, - icount_dummy_timer, NULL); + icount_timer_cb, NULL); } icount_align_option = qemu_opt_get_bool(opts, "align", false); @@ -1496,7 +1504,7 @@ static void tcg_exec_all(void) int r; /* Account partial waits to QEMU_CLOCK_VIRTUAL. */ - qemu_clock_warp(QEMU_CLOCK_VIRTUAL); + qemu_account_warp_timer(); if (next_cpu == NULL) { next_cpu = first_cpu; diff --git a/docs/replay.txt b/docs/replay.txt index 149727e..3cedc25 100644 --- a/docs/replay.txt +++ b/docs/replay.txt @@ -107,7 +107,7 @@ at the specified moments of time. There are several kinds of timers: sources (e.g. real time clock chip). Host clock is the one of the sources of non-determinism. Host clock read operations should be logged to make the execution deterministic. - * Real time clock for icount. This clock is similar to real time clock but + * Virtual real time clock. This clock is similar to real time clock but it is used only for increasing virtual clock while virtual machine is sleeping. Due to its nature it is also non-deterministic as the host clock and has to be logged too. @@ -134,11 +134,20 @@ of time. That's why we do not process a group of timers until the checkpoint event will be read from the log. Such an event allows synchronizing CPU execution and timer events. -Another checkpoints application in record/replay is instruction counting -while the virtual machine is idle. This function (qemu_clock_warp) is called -from the wait loop. It changes virtual machine state and must be deterministic -then. That is why we added checkpoint to this function to prevent its -operation in replay mode when it does not correspond to record mode. +Two other checkpoints govern the "warping" of the virtual clock. +While the virtual machine is idle, the virtual clock increments at +1 ns per *real time* nanosecond. This is done by setting up a timer +(called the warp timer) on the virtual real time clock, so that the +timer fires at the next deadline of the virtual clock; the virtual clock +is then incremented (which is called "warping" the virtual clock) as +soon as the timer fires or the CPUs need to go out of the idle state. +Two functions are used for this purpose; because these actions change +virtual machine state and must be deterministic, each of them creates a +checkpoint. qemu_start_warp_timer checks if the CPUs are idle and if so +starts accounting real time to virtual clock. qemu_account_warp_timer +is called when the CPUs get an interrupt or when the warp timer fires, +and it warps the virtual clock by the amount of real time that has passed +since qemu_start_warp_timer. Bottom halves ------------- diff --git a/include/qemu/timer.h b/include/qemu/timer.h index d0946cb..7197d08 100644 --- a/include/qemu/timer.h +++ b/include/qemu/timer.h @@ -210,12 +210,11 @@ void qemu_clock_notify(QEMUClockType type); void qemu_clock_enable(QEMUClockType type, bool enabled); /** - * qemu_clock_warp: - * @type: the clock type + * qemu_start_warp_timer: * - * Warp a clock to a new value + * Starts a timer for virtual clock update */ -void qemu_clock_warp(QEMUClockType type); +void qemu_start_warp_timer(void); /** * qemu_clock_register_reset_notifier: diff --git a/include/sysemu/replay.h b/include/sysemu/replay.h index d24d502..e798919 100644 --- a/include/sysemu/replay.h +++ b/include/sysemu/replay.h @@ -27,7 +27,8 @@ typedef enum ReplayClockKind ReplayClockKind; /* IDs of the checkpoints */ enum ReplayCheckpoint { - CHECKPOINT_CLOCK_WARP, + CHECKPOINT_CLOCK_WARP_START, + CHECKPOINT_CLOCK_WARP_ACCOUNT, CHECKPOINT_RESET_REQUESTED, CHECKPOINT_SUSPEND_REQUESTED, CHECKPOINT_CLOCK_VIRTUAL, diff --git a/main-loop.c b/main-loop.c index 19beae7..3a7f4cd 100644 --- a/main-loop.c +++ b/main-loop.c @@ -509,7 +509,7 @@ int main_loop_wait(int nonblocking) /* CPU thread can infinitely wait for event after missing the warp */ - qemu_clock_warp(QEMU_CLOCK_VIRTUAL); + qemu_start_warp_timer(); qemu_clock_run_all_timers(); return ret; diff --git a/qemu-timer.c b/qemu-timer.c index e98ecc9..4441fe6 100644 --- a/qemu-timer.c +++ b/qemu-timer.c @@ -394,7 +394,9 @@ static bool timer_mod_ns_locked(QEMUTimerList *timer_list, static void timerlist_rearm(QEMUTimerList *timer_list) { /* Interrupt execution to force deadline recalculation. */ - qemu_clock_warp(timer_list->clock->type); + if (timer_list->clock->type == QEMU_CLOCK_VIRTUAL) { + qemu_start_warp_timer(); + } timerlist_notify(timer_list); } diff --git a/stubs/clock-warp.c b/stubs/clock-warp.c index 5ae32b9..8acb58a 100644 --- a/stubs/clock-warp.c +++ b/stubs/clock-warp.c @@ -2,7 +2,7 @@ #include "qemu-common.h" #include "qemu/timer.h" -void qemu_clock_warp(QEMUClockType type) +void qemu_start_warp_timer(void) { }