From patchwork Thu May 6 16:05:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12242735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93840C433B4 for ; Thu, 6 May 2021 16:28:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F2C9C60249 for ; Thu, 6 May 2021 16:28:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2C9C60249 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59818 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1legrc-0001x7-U0 for qemu-devel@archiver.kernel.org; Thu, 06 May 2021 12:28:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59838) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1legW3-0004Uc-UR for qemu-devel@nongnu.org; Thu, 06 May 2021 12:06:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:35605) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1legVx-0003e6-GQ for qemu-devel@nongnu.org; Thu, 06 May 2021 12:06:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1620317172; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XD6i3dbssdJ2dHzVQAQl0vHdXYxv6CZKwIRpTh9uJlg=; b=B8FmuQEV9DdkP0K9BlPnW18EFQmAkdMh3/WM8ZTodPpIXaLv/F5F0soP5dPyZEHhn/wA9f W/s+SnNK/y1UqxWUXu/5NMJDUHa4VGh7W3fQP5tdj0jxnr37pVYugrPyPDRpZBH/TFrfjG 6KpkwUKHaDmY4JXBGBeGABx7iO+1HKE= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-376-izZIXL02PzauZDdBzq5trg-1; Thu, 06 May 2021 12:06:11 -0400 X-MC-Unique: izZIXL02PzauZDdBzq5trg-1 Received: by mail-qv1-f69.google.com with SMTP id b10-20020a0cf04a0000b02901bda1df3afbso4487303qvl.13 for ; Thu, 06 May 2021 09:06:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XD6i3dbssdJ2dHzVQAQl0vHdXYxv6CZKwIRpTh9uJlg=; b=saXfuDwJBvacT8rWte/9/XJSwhKhcw4xzTAAfau5Bp5i9mrSlrprIhTI8MGkcKCrYP SMQFOvXFAogMvgb8dfSyKtBMZcX0W19FZZx6K6RAgXmf0zl8eKKDNFoOhux1zFDCKjG4 rVnnZ74aQ6u2VSLVrdOfZSToWfYNSghpml3N5vvG+rCrZ4cG6qoOj5Gdbb2XIvD0V3D4 AYD2ENzTzLlVYm81BuCOdP6oTWIgB6FnFNdNxB077T811k8YRGCnDwz4qqejfNwlrBxm ZHtk8JtnrP7omM5A2+nTvGPj09BDg0sgvr8qoTVb0BYgSZPwn3EC56UN5m4I8o86fD4I Htsw== X-Gm-Message-State: AOAM530A/HTsY0CaYHN7AjhJlK8JBbwMNYnVcZ7AgVLL+/JpzQ92crd5 oyhf66NzSquVrayF6LlxD9L2mK18x8To2GKkTCMEVePEGRruKpKLef3vjuNkJUY22WWmPN++jk4 3heLYgzkcDbhwgQhuGd+ti1oOpC9uSEqMClpZ43kkLVRO0D92fNc2syPzcG0I7Z9d X-Received: by 2002:a05:620a:12b1:: with SMTP id x17mr4735728qki.362.1620317170136; Thu, 06 May 2021 09:06:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwt1RurjpJEpHLksuB2DV1aBR8tTr+FPU7QmIJPgfecdgtgnNqFI6bAMIxCEmtrYCW3Vqne6g== X-Received: by 2002:a05:620a:12b1:: with SMTP id x17mr4735683qki.362.1620317169764; Thu, 06 May 2021 09:06:09 -0700 (PDT) Received: from t490s.redhat.com (bras-base-toroon474qw-grc-72-184-145-4-219.dsl.bell.ca. [184.145.4.219]) by smtp.gmail.com with ESMTPSA id y84sm2194131qkb.134.2021.05.06.09.06.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 May 2021 09:06:09 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v7 08/10] KVM: Add dirty-gfn-count property Date: Thu, 6 May 2021 12:05:47 -0400 Message-Id: <20210506160549.130416-9-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210506160549.130416-1-peterx@redhat.com> References: <20210506160549.130416-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -34 X-Spam_score: -3.5 X-Spam_bar: --- X-Spam_report: (-3.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.69, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Keqian Zhu , Hyman , "Dr . David Alan Gilbert" , peterx@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add a parameter for dirty gfn count for dirty rings. If zero, dirty ring is disabled. Otherwise dirty ring will be enabled with the per-vcpu gfn count as specified. If dirty ring cannot be enabled due to unsupported kernel or illegal parameter, it'll fallback to dirty logging. By default, dirty ring is not enabled (dirty-gfn-count default to 0). Signed-off-by: Peter Xu --- accel/kvm/kvm-all.c | 72 +++++++++++++++++++++++++++++++++++++++++++++ qemu-options.hx | 12 ++++++++ 2 files changed, 84 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 4af61734e37..e438e5d26ad 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -126,6 +126,9 @@ struct KVMState KVMMemoryListener *ml; AddressSpace *as; } *as; + bool kvm_dirty_ring_enabled; /* Whether KVM dirty ring is enabled */ + uint64_t kvm_dirty_ring_size; /* Size of the per-vcpu dirty ring */ + uint32_t kvm_dirty_gfn_count; /* Number of dirty GFNs per ring */ }; KVMState *kvm_state; @@ -2130,6 +2133,40 @@ static int kvm_init(MachineState *ms) s->coalesced_pio = s->coalesced_mmio && kvm_check_extension(s, KVM_CAP_COALESCED_PIO); + /* + * Enable KVM dirty ring if supported, otherwise fall back to + * dirty logging mode + */ + if (s->kvm_dirty_gfn_count > 0) { + uint64_t ring_size; + + ring_size = s->kvm_dirty_gfn_count * sizeof(struct kvm_dirty_gfn); + + /* Read the max supported pages */ + ret = kvm_vm_check_extension(s, KVM_CAP_DIRTY_LOG_RING); + if (ret > 0) { + if (ring_size > ret) { + error_report("KVM dirty GFN count %" PRIu32 " too big " + "(maximum is %ld). Please use a smaller value.", + s->kvm_dirty_gfn_count, + ret / sizeof(struct kvm_dirty_gfn)); + ret = -EINVAL; + goto err; + } + + ret = kvm_vm_enable_cap(s, KVM_CAP_DIRTY_LOG_RING, 0, ring_size); + if (ret) { + error_report("Enabling of KVM dirty ring failed: %d. " + "Suggested mininum value is 1024. " + "Please also make sure it's a power of two.", ret); + goto err; + } + + s->kvm_dirty_ring_size = ring_size; + s->kvm_dirty_ring_enabled = true; + } + } + dirty_log_manual_caps = kvm_check_extension(s, KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2); dirty_log_manual_caps &= (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | @@ -3173,6 +3210,33 @@ bool kvm_kernel_irqchip_split(void) return kvm_state->kernel_irqchip_split == ON_OFF_AUTO_ON; } +static void kvm_get_dirty_gfn_count(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + KVMState *s = KVM_STATE(obj); + uint32_t value = s->kvm_dirty_gfn_count; + + visit_type_uint32(v, name, &value, errp); +} + +static void kvm_set_dirty_gfn_count(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + KVMState *s = KVM_STATE(obj); + Error *error = NULL; + uint32_t value; + + visit_type_uint32(v, name, &value, &error); + if (error) { + error_propagate(errp, error); + return; + } + + s->kvm_dirty_gfn_count = value; +} + static void kvm_accel_instance_init(Object *obj) { KVMState *s = KVM_STATE(obj); @@ -3180,6 +3244,8 @@ static void kvm_accel_instance_init(Object *obj) s->kvm_shadow_mem = -1; s->kernel_irqchip_allowed = true; s->kernel_irqchip_split = ON_OFF_AUTO_AUTO; + /* KVM dirty ring is by default off */ + s->kvm_dirty_gfn_count = 0; } static void kvm_accel_class_init(ObjectClass *oc, void *data) @@ -3201,6 +3267,12 @@ static void kvm_accel_class_init(ObjectClass *oc, void *data) NULL, NULL); object_class_property_set_description(oc, "kvm-shadow-mem", "KVM shadow MMU size"); + + object_class_property_add(oc, "dirty-gfn-count", "uint32", + kvm_get_dirty_gfn_count, kvm_set_dirty_gfn_count, + NULL, NULL); + object_class_property_set_description(oc, "dirty-gfn-count", + "KVM dirty GFN count (=0 to disable dirty ring)"); } static const TypeInfo kvm_accel_type = { diff --git a/qemu-options.hx b/qemu-options.hx index 635dc8a6243..e40c17fb761 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -141,6 +141,7 @@ DEF("accel", HAS_ARG, QEMU_OPTION_accel, " kvm-shadow-mem=size of KVM shadow MMU in bytes\n" " split-wx=on|off (enable TCG split w^x mapping)\n" " tb-size=n (TCG translation block cache size)\n" + " dirty-gfn-count=n (KVM dirty ring GFN count, default 0)\n" " thread=single|multi (enable multi-threaded TCG)\n", QEMU_ARCH_ALL) SRST ``-accel name[,prop=value[,...]]`` @@ -181,6 +182,17 @@ SRST where both the back-end and front-ends support it and no incompatible TCG features have been enabled (e.g. icount/replay). + + ``dirty-gfn-count=n`` + When KVM accelerator is used, it controls the per-vcpu KVM dirty ring + size (number of entries one dirty ring contains, per-vcpu). It should + be a value that is power of two, and it should be 1024 or bigger (but + still less than the maximum value that the kernel supports). 4096 + could be a good initial value if you have no idea which is the best. + Set this value to 0 to disable the feature. By default, this feature + is disabled (dirty-gfn-count=0). When enabled, it'll automatically + replace the kvm get dirty log feature. + ERST DEF("smp", HAS_ARG, QEMU_OPTION_smp,