From patchwork Mon Sep 13 23:09:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 12491003 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E21D3C433F5 for ; Mon, 13 Sep 2021 23:20:41 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB2F060F3A for ; Mon, 13 Sep 2021 23:20:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AB2F060F3A Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=StRabGklrkdPFgJtBjibWqLVRNW9lwlNzI7PO4hOKqA=; b=mvCso5hLzxJK9dk6jgBi3/+lhx AN7mVhr8E6v5aV8dfcALOCLI25F6OLP71a5Jwo7mCFtYfNS5ZfgTaUamlR4TfThLT8xvuHYtilNr0 gjXl8mxmvqCUYISMlyKrCdOfup7W+cM7UG8TPeDJCF6klyqtbGX4gEXT6OQDU7mPAYioyBM3jV6O+ ulhOYywRn0doNkXHa7OTKf0OUZkcDHghFy6lxYX8pCdbrPhWSzH2R/GRAggsDOYuseZrlJNBHk1XP ba0dytIUoKDhJiOlMwmclTgoEHsmIil+fOrBe+6ocjjgOtxPzA/Aw5qUDtovPj8cGMqQj30sW8Csm Tc5V4sOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mPvDB-003aoq-HY; Mon, 13 Sep 2021 23:18:07 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mPv5u-003XfA-Gv for linux-arm-kernel@lists.infradead.org; Mon, 13 Sep 2021 23:10:36 +0000 Received: by mail-yb1-xb49.google.com with SMTP id a23-20020a25ae17000000b005ad73346312so7523024ybj.18 for ; Mon, 13 Sep 2021 16:10:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=mkrEw/6odZA643/v871JIfxaL0FOz/JzQ68NluiA4Io=; b=omSztA5jx1ZDK4bGekWatikS8TbQnmzTqKwcBBBDzYL2BqrEO7cS4B1CwTFlfJH+zl la3SCcyApQoAR2rBE8BzpKgjc8TOA5SZl3P5/Q9a9G5bW3znc3Vi7pdbi6V5JhdHpgTQ j+4Um5G1sI3S/PLcBq2MdaWMKG4OCo3AqeTJT6zSbLq585/gywL6olfHQJuW8watr0ou MrsHy62s3NRaT1fSPWjCxJ7JCqbIXf11ikshRUXP9hlpVkG0hweJMo5S1c1UcjSFdbam QTZZ2w7DBDPWl35weml+QVk4VLL1jkfO3IG7MblakMqQO3OnbMwXuyaH0vP9QI558Vio x6+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=mkrEw/6odZA643/v871JIfxaL0FOz/JzQ68NluiA4Io=; b=iZ89b8ixV3Urv1Ec/vCIJp6leH4cP5B0NHpfyFBtBaqvpALQtLDplLZezH2yi1kIG4 mFZmrUVscuZmv3ikBnEscmziU3W1nwllsskSDorOwAWuQKMaCImbQ6G0IADq5EBgp9kt MkdrfvO6J0OvGbpJikJPSc4NvV/hBMsnppFkZhDM0tN7kvAD8EfkJDR2aB++GVo+8mmn wXwTJszFHtUC05Lyea4Q5r1hy/3yhrNwTowlGzYsiNbufq9qbLz2xsj8it22zk50bXur FeXbOjpA11qyu0Cg40SruDJzghQL4KwWETW9XqvMclh6CCyuXgeZ+dnh+XHpfDbWfQT4 Ck2A== X-Gm-Message-State: AOAM531So1EulV0iv+wcQV5vStX6V5702khkcUw3H/GwE2MjOOI9t9jo f4gtAhIRHw5Wo15CiIiiDNJuB1DxOHV2 X-Google-Smtp-Source: ABdhPJwr/zwzZfn9H8PexHzVbehoGCRGw3mByNih/740GTSh9rk5TRjsqEPUSLw1qhKgiEOY0w090Wpoao0P X-Received: from rananta-virt.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1bcc]) (user=rananta job=sendgmr) by 2002:a25:b98b:: with SMTP id r11mr18356489ybg.189.1631574633078; Mon, 13 Sep 2021 16:10:33 -0700 (PDT) Date: Mon, 13 Sep 2021 23:09:55 +0000 In-Reply-To: <20210913230955.156323-1-rananta@google.com> Message-Id: <20210913230955.156323-15-rananta@google.com> Mime-Version: 1.0 References: <20210913230955.156323-1-rananta@google.com> X-Mailer: git-send-email 2.33.0.309.g3052b89438-goog Subject: [PATCH v6 14/14] KVM: arm64: selftests: arch_timer: Support vCPU migration From: Raghavendra Rao Ananta To: Paolo Bonzini , Marc Zyngier , Andrew Jones , James Morse , Alexandru Elisei , Suzuki K Poulose Cc: Catalin Marinas , Will Deacon , Peter Shier , Ricardo Koller , Oliver Upton , Reiji Watanabe , Jing Zhang , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210913_161034_632464_F0EF18DD X-CRM114-Status: GOOD ( 23.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Since the timer stack (hardware and KVM) is per-CPU, there are potential chances for races to occur when the scheduler decides to migrate a vCPU thread to a different physical CPU. Hence, include an option to stress-test this part as well by forcing the vCPUs to migrate across physical CPUs in the system at a particular rate. Originally, the bug for the fix with commit 3134cc8beb69d0d ("KVM: arm64: vgic: Resample HW pending state on deactivation") was discovered using arch_timer test with vCPU migrations and can be easily reproduced. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Andrew Jones --- .../selftests/kvm/aarch64/arch_timer.c | 115 +++++++++++++++++- 1 file changed, 114 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/aarch64/arch_timer.c b/tools/testing/selftests/kvm/aarch64/arch_timer.c index 3b6ea6a462f4..228e7ed5531c 100644 --- a/tools/testing/selftests/kvm/aarch64/arch_timer.c +++ b/tools/testing/selftests/kvm/aarch64/arch_timer.c @@ -14,6 +14,8 @@ * * The test provides command-line options to configure the timer's * period (-p), number of vCPUs (-n), and iterations per stage (-i). + * To stress-test the timer stack even more, an option to migrate the + * vCPUs across pCPUs (-m), at a particular rate, is also provided. * * Copyright (c) 2021, Google LLC. */ @@ -24,6 +26,8 @@ #include #include #include +#include +#include #include "kvm_util.h" #include "processor.h" @@ -36,17 +40,20 @@ #define NR_TEST_ITERS_DEF 5 #define TIMER_TEST_PERIOD_MS_DEF 10 #define TIMER_TEST_ERR_MARGIN_US 100 +#define TIMER_TEST_MIGRATION_FREQ_MS 2 struct test_args { int nr_vcpus; int nr_iter; int timer_period_ms; + int migration_freq_ms; }; static struct test_args test_args = { .nr_vcpus = NR_VCPUS_DEF, .nr_iter = NR_TEST_ITERS_DEF, .timer_period_ms = TIMER_TEST_PERIOD_MS_DEF, + .migration_freq_ms = TIMER_TEST_MIGRATION_FREQ_MS, }; #define msecs_to_usecs(msec) ((msec) * 1000LL) @@ -80,6 +87,9 @@ static struct test_vcpu_shared_data vcpu_shared_data[KVM_MAX_VCPUS]; static int vtimer_irq, ptimer_irq; +static unsigned long *vcpu_done_map; +static pthread_mutex_t vcpu_done_map_lock; + static void guest_configure_timer_action(struct test_vcpu_shared_data *shared_data) { @@ -215,6 +225,11 @@ static void *test_vcpu_run(void *arg) vcpu_run(vm, vcpuid); + /* Currently, any exit from guest is an indication of completion */ + pthread_mutex_lock(&vcpu_done_map_lock); + set_bit(vcpuid, vcpu_done_map); + pthread_mutex_unlock(&vcpu_done_map_lock); + switch (get_ucall(vm, vcpuid, &uc)) { case UCALL_SYNC: case UCALL_DONE: @@ -233,9 +248,78 @@ static void *test_vcpu_run(void *arg) return NULL; } +static uint32_t test_get_pcpu(void) +{ + uint32_t pcpu; + unsigned int nproc_conf; + cpu_set_t online_cpuset; + + nproc_conf = get_nprocs_conf(); + sched_getaffinity(0, sizeof(cpu_set_t), &online_cpuset); + + /* Randomly find an available pCPU to place a vCPU on */ + do { + pcpu = rand() % nproc_conf; + } while (!CPU_ISSET(pcpu, &online_cpuset)); + + return pcpu; +} + +static int test_migrate_vcpu(struct test_vcpu *vcpu) +{ + int ret; + cpu_set_t cpuset; + uint32_t new_pcpu = test_get_pcpu(); + + CPU_ZERO(&cpuset); + CPU_SET(new_pcpu, &cpuset); + + pr_debug("Migrating vCPU: %u to pCPU: %u\n", vcpu->vcpuid, new_pcpu); + + ret = pthread_setaffinity_np(vcpu->pt_vcpu_run, + sizeof(cpuset), &cpuset); + + /* Allow the error where the vCPU thread is already finished */ + TEST_ASSERT(ret == 0 || ret == ESRCH, + "Failed to migrate the vCPU:%u to pCPU: %u; ret: %d\n", + vcpu->vcpuid, new_pcpu, ret); + + return ret; +} + +static void *test_vcpu_migration(void *arg) +{ + unsigned int i, n_done; + bool vcpu_done; + + do { + usleep(msecs_to_usecs(test_args.migration_freq_ms)); + + for (n_done = 0, i = 0; i < test_args.nr_vcpus; i++) { + pthread_mutex_lock(&vcpu_done_map_lock); + vcpu_done = test_bit(i, vcpu_done_map); + pthread_mutex_unlock(&vcpu_done_map_lock); + + if (vcpu_done) { + n_done++; + continue; + } + + test_migrate_vcpu(&test_vcpu[i]); + } + } while (test_args.nr_vcpus != n_done); + + return NULL; +} + static void test_run(struct kvm_vm *vm) { int i, ret; + pthread_t pt_vcpu_migration; + + pthread_mutex_init(&vcpu_done_map_lock, NULL); + vcpu_done_map = bitmap_alloc(test_args.nr_vcpus); + TEST_ASSERT(vcpu_done_map, "Failed to allocate vcpu done bitmap\n"); for (i = 0; i < test_args.nr_vcpus; i++) { ret = pthread_create(&test_vcpu[i].pt_vcpu_run, NULL, @@ -243,8 +327,23 @@ static void test_run(struct kvm_vm *vm) TEST_ASSERT(!ret, "Failed to create vCPU-%d pthread\n", i); } + /* Spawn a thread to control the vCPU migrations */ + if (test_args.migration_freq_ms) { + srand(time(NULL)); + + ret = pthread_create(&pt_vcpu_migration, NULL, + test_vcpu_migration, NULL); + TEST_ASSERT(!ret, "Failed to create the migration pthread\n"); + } + + for (i = 0; i < test_args.nr_vcpus; i++) pthread_join(test_vcpu[i].pt_vcpu_run, NULL); + + if (test_args.migration_freq_ms) + pthread_join(pt_vcpu_migration, NULL); + + bitmap_free(vcpu_done_map); } static void test_init_timer_irq(struct kvm_vm *vm) @@ -301,6 +400,8 @@ static void test_print_help(char *name) NR_TEST_ITERS_DEF); pr_info("\t-p: Periodicity (in ms) of the guest timer (default: %u)\n", TIMER_TEST_PERIOD_MS_DEF); + pr_info("\t-m: Frequency (in ms) of vCPUs to migrate to different pCPU. 0 to turn off (default: %u)\n", + TIMER_TEST_MIGRATION_FREQ_MS); pr_info("\t-h: print this help screen\n"); } @@ -308,7 +409,7 @@ static bool parse_args(int argc, char *argv[]) { int opt; - while ((opt = getopt(argc, argv, "hn:i:p:")) != -1) { + while ((opt = getopt(argc, argv, "hn:i:p:m:")) != -1) { switch (opt) { case 'n': test_args.nr_vcpus = atoi(optarg); @@ -335,6 +436,13 @@ static bool parse_args(int argc, char *argv[]) goto err; } break; + case 'm': + test_args.migration_freq_ms = atoi(optarg); + if (test_args.migration_freq_ms < 0) { + pr_info("0 or positive value needed for -m\n"); + goto err; + } + break; case 'h': default: goto err; @@ -358,6 +466,11 @@ int main(int argc, char *argv[]) if (!parse_args(argc, argv)) exit(KSFT_SKIP); + if (test_args.migration_freq_ms && get_nprocs() < 2) { + print_skip("At least two physical CPUs needed for vCPU migration"); + exit(KSFT_SKIP); + } + vm = test_vm_create(); test_run(vm); kvm_vm_free(vm);