From patchwork Wed Jun 10 11:53:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597601 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50CDB138C for ; Wed, 10 Jun 2020 11:54:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 35FF02074B for ; Wed, 10 Jun 2020 11:54:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="N6kPDq9G" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728728AbgFJLyt (ORCPT ); Wed, 10 Jun 2020 07:54:49 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:60176 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728497AbgFJLys (ORCPT ); Wed, 10 Jun 2020 07:54:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790086; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U68jIOHuvNFPHE8HVjN9BLo12zDx2a4qnBp0EnzJr9o=; b=N6kPDq9GUHKnVFn89FGMwRHiUeNvT4ptvngs2Moi7z52dkElst5pMgOI/0d3m1y5HaA4UE +p5XK8Jk0iDL0yQONpbZ7pd9LUQeXYu1d6vwRzZn9NBUddtOs6TuIDszC/pGQ0IeL2I+mF 9LQVdDjkIAGF/kcXNz9UgrBOHrk5kr4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-283-LYF1y1dlMwuABbJGim0Zvg-1; Wed, 10 Jun 2020 07:54:40 -0400 X-MC-Unique: LYF1y1dlMwuABbJGim0Zvg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9C04E193F560; Wed, 10 Jun 2020 11:54:39 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id BC3F25D9D3; Wed, 10 Jun 2020 11:54:37 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 01/21] exec: Introduce ram_block_discard_(disable|require)() Date: Wed, 10 Jun 2020 13:53:59 +0200 Message-Id: <20200610115419.51688-2-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We want to replace qemu_balloon_inhibit() by something more generic. Especially, we want to make sure that technologies that really rely on RAM block discards to work reliably to run mutual exclusive with technologies that effectively break it. E.g., vfio will usually pin all guest memory, turning the virtio-balloon basically useless and make the VM consume more memory than reported via the balloon. While the balloon is special already (=> no guarantees, same behavior possible afer reboots and with huge pages), this will be different, especially, with virtio-mem. Let's implement a way such that we can make both types of technology run mutually exclusive. We'll convert existing balloon inhibitors in successive patches and add some new ones. Add the check to qemu_balloon_is_inhibited() for now. We might want to make virtio-balloon an acutal inhibitor in the future - however, that requires more thought to not break existing setups. Reviewed-by: Dr. David Alan Gilbert Cc: "Michael S. Tsirkin" Cc: Richard Henderson Cc: Paolo Bonzini Signed-off-by: David Hildenbrand --- balloon.c | 3 ++- exec.c | 52 +++++++++++++++++++++++++++++++++++++++++++ include/exec/memory.h | 41 ++++++++++++++++++++++++++++++++++ 3 files changed, 95 insertions(+), 1 deletion(-) diff --git a/balloon.c b/balloon.c index f104b42961..5fff79523a 100644 --- a/balloon.c +++ b/balloon.c @@ -40,7 +40,8 @@ static int balloon_inhibit_count; bool qemu_balloon_is_inhibited(void) { - return atomic_read(&balloon_inhibit_count) > 0; + return atomic_read(&balloon_inhibit_count) > 0 || + ram_block_discard_is_disabled(); } void qemu_balloon_inhibit(bool state) diff --git a/exec.c b/exec.c index be4be2df3a..c4c1d9df84 100644 --- a/exec.c +++ b/exec.c @@ -4051,4 +4051,56 @@ void mtree_print_dispatch(AddressSpaceDispatch *d, MemoryRegion *root) } } +/* + * If positive, discarding RAM is disabled. If negative, discarding RAM is + * required to work and cannot be disabled. + */ +static int ram_block_discard_disabled; + +int ram_block_discard_disable(bool state) +{ + int old; + + if (!state) { + atomic_dec(&ram_block_discard_disabled); + return 0; + } + + do { + old = atomic_read(&ram_block_discard_disabled); + if (old < 0) { + return -EBUSY; + } + } while (atomic_cmpxchg(&ram_block_discard_disabled, old, old + 1) != old); + return 0; +} + +int ram_block_discard_require(bool state) +{ + int old; + + if (!state) { + atomic_inc(&ram_block_discard_disabled); + return 0; + } + + do { + old = atomic_read(&ram_block_discard_disabled); + if (old > 0) { + return -EBUSY; + } + } while (atomic_cmpxchg(&ram_block_discard_disabled, old, old - 1) != old); + return 0; +} + +bool ram_block_discard_is_disabled(void) +{ + return atomic_read(&ram_block_discard_disabled) > 0; +} + +bool ram_block_discard_is_required(void) +{ + return atomic_read(&ram_block_discard_disabled) < 0; +} + #endif diff --git a/include/exec/memory.h b/include/exec/memory.h index 3e00cdbbfa..eea7d284b9 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -2474,6 +2474,47 @@ static inline MemOp devend_memop(enum device_endian end) } #endif +/* + * Inhibit technologies that require discarding of pages in RAM blocks, e.g., + * to manage the actual amount of memory consumed by the VM (then, the memory + * provided by RAM blocks might be bigger than the desired memory consumption). + * This *must* be set if: + * - Discarding parts of a RAM blocks does not result in the change being + * reflected in the VM and the pages getting freed. + * - All memory in RAM blocks is pinned or duplicated, invaldiating any previous + * discards blindly. + * - Discarding parts of a RAM blocks will result in integrity issues (e.g., + * encrypted VMs). + * Technologies that only temporarily pin the current working set of a + * driver are fine, because we don't expect such pages to be discarded + * (esp. based on guest action like balloon inflation). + * + * This is *not* to be used to protect from concurrent discards (esp., + * postcopy). + * + * Returns 0 if successful. Returns -EBUSY if a technology that relies on + * discards to work reliably is active. + */ +int ram_block_discard_disable(bool state); + +/* + * Inhibit technologies that disable discarding of pages in RAM blocks. + * + * Returns 0 if successful. Returns -EBUSY if discards are already set to + * broken. + */ +int ram_block_discard_require(bool state); + +/* + * Test if discarding of memory in ram blocks is disabled. + */ +bool ram_block_discard_is_disabled(void); + +/* + * Test if discarding of memory in ram blocks is required to work reliably. + */ +bool ram_block_discard_is_required(void); + #endif #endif From patchwork Wed Jun 10 11:54:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597605 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA48714B7 for ; Wed, 10 Jun 2020 11:55:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A4E5C2072E for ; Wed, 10 Jun 2020 11:55:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Y5DJo/Kv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728755AbgFJLzB (ORCPT ); Wed, 10 Jun 2020 07:55:01 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:56266 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728730AbgFJLzA (ORCPT ); Wed, 10 Jun 2020 07:55:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790097; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u3TnvMHIMVg+ynIb51APci4iCUQNAE73bAVqqwa7/LQ=; b=Y5DJo/KvQb9Bq3Li7DSELsUYe4HWkDYPSzXoUgdDC0hcB+yW6UF6vXTkjrVojw3o77A6Bg 1bAwFmi/FgKOMen0nDuGNSs7WBWds5cRLdpEfuJIasYmkNDJbc5S5BQwCXBxkVABbtYZft hcoa41/Iih6PcfGBtsfdiRqoCz/5bF8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-1-kUj70tKQP2ug4p5YQ87IUg-1; Wed, 10 Jun 2020 07:54:52 -0400 X-MC-Unique: kUj70tKQP2ug4p5YQ87IUg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 453A1107ACCA; Wed, 10 Jun 2020 11:54:51 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id EE79C5D9D3; Wed, 10 Jun 2020 11:54:39 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Cornelia Huck , Alex Williamson , Christian Borntraeger , Tony Krowiak , Halil Pasic , Pierre Morel , Eric Farman Subject: [PATCH v4 02/21] vfio: Convert to ram_block_discard_disable() Date: Wed, 10 Jun 2020 13:54:00 +0200 Message-Id: <20200610115419.51688-3-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org VFIO is (except devices without a physical IOMMU or some mediated devices) incompatible with discarding of RAM. The kernel will pin basically all VM memory. Let's convert to ram_block_discard_disable(), which can now fail, in contrast to qemu_balloon_inhibit(). Leave "x-balloon-allowed" named as it is for now. Cc: Cornelia Huck Cc: Alex Williamson Cc: Christian Borntraeger Cc: Tony Krowiak Cc: Halil Pasic Cc: Pierre Morel Cc: Eric Farman Signed-off-by: David Hildenbrand Reviewed-by: Tony Krowiak Acked-by: Cornelia Huck --- hw/vfio/ap.c | 10 +++---- hw/vfio/ccw.c | 11 ++++---- hw/vfio/common.c | 53 +++++++++++++++++++---------------- hw/vfio/pci.c | 6 ++-- include/hw/vfio/vfio-common.h | 4 +-- 5 files changed, 45 insertions(+), 39 deletions(-) diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index 95564c17ed..d0b1bc7581 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -105,12 +105,12 @@ static void vfio_ap_realize(DeviceState *dev, Error **errp) vapdev->vdev.dev = dev; /* - * vfio-ap devices operate in a way compatible with - * memory ballooning, as no pages are pinned in the host. - * This needs to be set before vfio_get_device() for vfio common to - * handle the balloon inhibitor. + * vfio-ap devices operate in a way compatible discarding of memory in + * RAM blocks, as no pages are pinned in the host. This needs to be + * set before vfio_get_device() for vfio common to handle + * ram_block_discard_disable(). */ - vapdev->vdev.balloon_allowed = true; + vapdev->vdev.ram_block_discard_allowed = true; ret = vfio_get_device(vfio_group, mdevid, &vapdev->vdev, errp); if (ret) { diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 63406184d2..82857f1615 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -418,12 +418,13 @@ static void vfio_ccw_get_device(VFIOGroup *group, VFIOCCWDevice *vcdev, /* * All vfio-ccw devices are believed to operate in a way compatible with - * memory ballooning, ie. pages pinned in the host are in the current - * working set of the guest driver and therefore never overlap with pages - * available to the guest balloon driver. This needs to be set before - * vfio_get_device() for vfio common to handle the balloon inhibitor. + * discarding of memory in RAM blocks, ie. pages pinned in the host are + * in the current working set of the guest driver and therefore never + * overlap e.g., with pages available to the guest balloon driver. This + * needs to be set before vfio_get_device() for vfio common to handle + * ram_block_discard_disable(). */ - vcdev->vdev.balloon_allowed = true; + vcdev->vdev.ram_block_discard_allowed = true; if (vfio_get_device(group, vcdev->cdev.mdevid, &vcdev->vdev, errp)) { goto out_err; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 0b3593b3c0..33357140b8 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -33,7 +33,6 @@ #include "qemu/error-report.h" #include "qemu/main-loop.h" #include "qemu/range.h" -#include "sysemu/balloon.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" #include "trace.h" @@ -1215,31 +1214,36 @@ static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, space = vfio_get_address_space(as); /* - * VFIO is currently incompatible with memory ballooning insofar as the + * VFIO is currently incompatible with discarding of RAM insofar as the * madvise to purge (zap) the page from QEMU's address space does not * interact with the memory API and therefore leaves stale virtual to * physical mappings in the IOMMU if the page was previously pinned. We - * therefore add a balloon inhibit for each group added to a container, + * therefore set discarding broken for each group added to a container, * whether the container is used individually or shared. This provides * us with options to allow devices within a group to opt-in and allow - * ballooning, so long as it is done consistently for a group (for instance + * discarding, so long as it is done consistently for a group (for instance * if the device is an mdev device where it is known that the host vendor * driver will never pin pages outside of the working set of the guest - * driver, which would thus not be ballooning candidates). + * driver, which would thus not be discarding candidates). * * The first opportunity to induce pinning occurs here where we attempt to * attach the group to existing containers within the AddressSpace. If any - * pages are already zapped from the virtual address space, such as from a - * previous ballooning opt-in, new pinning will cause valid mappings to be + * pages are already zapped from the virtual address space, such as from + * previous discards, new pinning will cause valid mappings to be * re-established. Likewise, when the overall MemoryListener for a new * container is registered, a replay of mappings within the AddressSpace * will occur, re-establishing any previously zapped pages as well. * - * NB. Balloon inhibiting does not currently block operation of the - * balloon driver or revoke previously pinned pages, it only prevents - * calling madvise to modify the virtual mapping of ballooned pages. + * Especially virtio-balloon is currently only prevented from discarding + * new memory, it will not yet set ram_block_discard_set_required() and + * therefore, neither stops us here or deals with the sudden memory + * consumption of inflated memory. */ - qemu_balloon_inhibit(true); + ret = ram_block_discard_disable(true); + if (ret) { + error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"); + return ret; + } QLIST_FOREACH(container, &space->containers, next) { if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { @@ -1405,7 +1409,7 @@ close_fd_exit: close(fd); put_space_exit: - qemu_balloon_inhibit(false); + ram_block_discard_disable(false); vfio_put_address_space(space); return ret; @@ -1526,8 +1530,8 @@ void vfio_put_group(VFIOGroup *group) return; } - if (!group->balloon_allowed) { - qemu_balloon_inhibit(false); + if (!group->ram_block_discard_allowed) { + ram_block_discard_disable(false); } vfio_kvm_device_del_group(group); vfio_disconnect_container(group); @@ -1565,22 +1569,23 @@ int vfio_get_device(VFIOGroup *group, const char *name, } /* - * Clear the balloon inhibitor for this group if the driver knows the - * device operates compatibly with ballooning. Setting must be consistent - * per group, but since compatibility is really only possible with mdev - * currently, we expect singleton groups. + * Set discarding of RAM as not broken for this group if the driver knows + * the device operates compatibly with discarding. Setting must be + * consistent per group, but since compatibility is really only possible + * with mdev currently, we expect singleton groups. */ - if (vbasedev->balloon_allowed != group->balloon_allowed) { + if (vbasedev->ram_block_discard_allowed != + group->ram_block_discard_allowed) { if (!QLIST_EMPTY(&group->device_list)) { - error_setg(errp, - "Inconsistent device balloon setting within group"); + error_setg(errp, "Inconsistent setting of support for discarding " + "RAM (e.g., balloon) within group"); close(fd); return -1; } - if (!group->balloon_allowed) { - group->balloon_allowed = true; - qemu_balloon_inhibit(false); + if (!group->ram_block_discard_allowed) { + group->ram_block_discard_allowed = true; + ram_block_discard_disable(false); } } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 342dd6b912..c33c11b7e4 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2796,7 +2796,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } /* - * Mediated devices *might* operate compatibly with memory ballooning, but + * Mediated devices *might* operate compatibly with discarding of RAM, but * we cannot know for certain, it depends on whether the mdev vendor driver * stays in sync with the active working set of the guest driver. Prevent * the x-balloon-allowed option unless this is minimally an mdev device. @@ -2809,7 +2809,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) trace_vfio_mdev(vdev->vbasedev.name, is_mdev); - if (vdev->vbasedev.balloon_allowed && !is_mdev) { + if (vdev->vbasedev.ram_block_discard_allowed && !is_mdev) { error_setg(errp, "x-balloon-allowed only potentially compatible " "with mdev devices"); vfio_put_group(group); @@ -3163,7 +3163,7 @@ static Property vfio_pci_dev_properties[] = { VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice, - vbasedev.balloon_allowed, false), + vbasedev.ram_block_discard_allowed, false), DEFINE_PROP_BOOL("x-no-kvm-intx", VFIOPCIDevice, no_kvm_intx, false), DEFINE_PROP_BOOL("x-no-kvm-msi", VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false), diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index fd564209ac..c78f3ff559 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -108,7 +108,7 @@ typedef struct VFIODevice { bool reset_works; bool needs_reset; bool no_mmap; - bool balloon_allowed; + bool ram_block_discard_allowed; VFIODeviceOps *ops; unsigned int num_irqs; unsigned int num_regions; @@ -128,7 +128,7 @@ typedef struct VFIOGroup { QLIST_HEAD(, VFIODevice) device_list; QLIST_ENTRY(VFIOGroup) next; QLIST_ENTRY(VFIOGroup) container_next; - bool balloon_allowed; + bool ram_block_discard_allowed; } VFIOGroup; typedef struct VFIODMABuf { From patchwork Wed Jun 10 11:54:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597603 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8967138C for ; Wed, 10 Jun 2020 11:54:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A625220760 for ; Wed, 10 Jun 2020 11:54:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VV0rwxwP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728740AbgFJLy5 (ORCPT ); Wed, 10 Jun 2020 07:54:57 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:27389 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728730AbgFJLy5 (ORCPT ); Wed, 10 Jun 2020 07:54:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790096; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q5ARNUWTcyRA1N5fC9aAjQ1xopL6Yzibnqxs9roC3AQ=; b=VV0rwxwPqHrFpMQVlD+f0iKMYDswKoKKd1fru+otoG5Nq/j4Voywe/S1WP54huw6NmeJCc vbhocEd3Pn0jON0Ap7eX+CVey/dl0GWaDNwlIW1gQzYKnAknTpcTyQsyjIkzcQFEHzrS3r crtIvUf48+DrT38+eski41U39oIVl04= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-286-paF5wbtsPCaRfoUBu8jcBA-1; Wed, 10 Jun 2020 07:54:54 -0400 X-MC-Unique: paF5wbtsPCaRfoUBu8jcBA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 80570A0BD7; Wed, 10 Jun 2020 11:54:53 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 963A15D9D3; Wed, 10 Jun 2020 11:54:51 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 03/21] accel/kvm: Convert to ram_block_discard_disable() Date: Wed, 10 Jun 2020 13:54:01 +0200 Message-Id: <20200610115419.51688-4-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Discarding memory does not work as expected. At the time this is called, we cannot have anyone active that relies on discards to work properly. Reviewed-by: Dr. David Alan Gilbert Cc: Paolo Bonzini Signed-off-by: David Hildenbrand --- accel/kvm/kvm-all.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index d06cc04079..fa18b2caae 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -40,7 +40,6 @@ #include "trace.h" #include "hw/irq.h" #include "sysemu/sev.h" -#include "sysemu/balloon.h" #include "qapi/visitor.h" #include "qapi/qapi-types-common.h" #include "qapi/qapi-visit-common.h" @@ -2143,7 +2142,8 @@ static int kvm_init(MachineState *ms) s->sync_mmu = !!kvm_vm_check_extension(kvm_state, KVM_CAP_SYNC_MMU); if (!s->sync_mmu) { - qemu_balloon_inhibit(true); + ret = ram_block_discard_disable(true); + assert(!ret); } return 0; From patchwork Wed Jun 10 11:54:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597607 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00B3D138C for ; Wed, 10 Jun 2020 11:55:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DD79C20760 for ; Wed, 10 Jun 2020 11:55:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Ea6ADcck" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728730AbgFJLzD (ORCPT ); Wed, 10 Jun 2020 07:55:03 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:58108 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728751AbgFJLzC (ORCPT ); Wed, 10 Jun 2020 07:55:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUu0CYVglzX/Ll9iBfJF7XJmcLsniLRcSuurJhAdw+U=; b=Ea6ADcck4+L0T/aY0Dc5tihNuThdNN1WZHcLDNUcMnTm7NfhBlAnuVVKNkl1B9/ZYWveQK ngsBgzM1508m5EL/MkIZVeP75obvDG3LVvdy5099Bsw8BGFVGrwZ6z7n96taZahGmkzLp+ y3ivy4F3cpuFUuS6Gg/1hL2n4fzMxIg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-198-nG8aSuP-M3m0pAJ0p-BRKg-1; Wed, 10 Jun 2020 07:54:57 -0400 X-MC-Unique: nG8aSuP-M3m0pAJ0p-BRKg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 57E4C8014D4; Wed, 10 Jun 2020 11:54:56 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id CEBC65D9D3; Wed, 10 Jun 2020 11:54:53 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Cornelia Huck , Halil Pasic , Christian Borntraeger , Janosch Frank Subject: [PATCH v4 04/21] s390x/pv: Convert to ram_block_discard_disable() Date: Wed, 10 Jun 2020 13:54:02 +0200 Message-Id: <20200610115419.51688-5-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Discarding RAM does not work as expected with protected VMs. Let's switch to ram_block_discard_disable() for now, as we want to get rid of qemu_balloon_inhibit(). Note that it will currently never fail, but might fail in the future with new technologies (e.g., virtio-mem). Cc: Richard Henderson Cc: Cornelia Huck Cc: Halil Pasic Cc: Christian Borntraeger Cc: Janosch Frank Signed-off-by: David Hildenbrand Acked-by: Cornelia Huck --- hw/s390x/s390-virtio-ccw.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 60b16fef77..c985bb56eb 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -43,7 +43,6 @@ #include "hw/qdev-properties.h" #include "hw/s390x/tod.h" #include "sysemu/sysemu.h" -#include "sysemu/balloon.h" #include "hw/s390x/pv.h" #include "migration/blocker.h" @@ -329,7 +328,7 @@ static void s390_machine_unprotect(S390CcwMachineState *ms) ms->pv = false; migrate_del_blocker(pv_mig_blocker); error_free_or_abort(&pv_mig_blocker); - qemu_balloon_inhibit(false); + ram_block_discard_disable(false); } static int s390_machine_protect(S390CcwMachineState *ms) @@ -338,17 +337,22 @@ static int s390_machine_protect(S390CcwMachineState *ms) int rc; /* - * Ballooning on protected VMs needs support in the guest for - * sharing and unsharing balloon pages. Block ballooning for - * now, until we have a solution to make at least Linux guests - * either support it or fail gracefully. + * Discarding of memory in RAM blocks does not work as expected with + * protected VMs. Sharing and unsharing pages would be required. Disable + * it for now, until until we have a solution to make at least Linux + * guests either support it (e.g., virtio-balloon) or fail gracefully. */ - qemu_balloon_inhibit(true); + rc = ram_block_discard_disable(true); + if (rc) { + error_report("protected VMs: cannot disable RAM discard"); + return rc; + } + error_setg(&pv_mig_blocker, "protected VMs are currently not migrateable."); rc = migrate_add_blocker(pv_mig_blocker, &local_err); if (rc) { - qemu_balloon_inhibit(false); + ram_block_discard_disable(false); error_report_err(local_err); error_free_or_abort(&pv_mig_blocker); return rc; @@ -357,7 +361,7 @@ static int s390_machine_protect(S390CcwMachineState *ms) /* Create SE VM */ rc = s390_pv_vm_enable(); if (rc) { - qemu_balloon_inhibit(false); + ram_block_discard_disable(false); migrate_del_blocker(pv_mig_blocker); error_free_or_abort(&pv_mig_blocker); return rc; From patchwork Wed Jun 10 11:54:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597609 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 88A70138C for ; Wed, 10 Jun 2020 11:55:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B6862074B for ; Wed, 10 Jun 2020 11:55:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PnNgr+Jz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728757AbgFJLzG (ORCPT ); Wed, 10 Jun 2020 07:55:06 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:50315 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728745AbgFJLzG (ORCPT ); Wed, 10 Jun 2020 07:55:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h8cmPlf+UhVmg+8r2ez85zjWAFR8fo9Gsptc/PSKHa0=; b=PnNgr+JzMDlYP+QoBKHgi1yYK2+kAvlUnyMjP958wLn8LO7ABbjbqBNFWtdIENEWvJvq7/ 4DdUgoIX7fn5rnVcXROYSqkM3vCVWICpxQXeoFjU2msVOMl3UDpPCdGTDyqC9e4j8NhN9e K5COg91elNkq0Fnc41q+Y3Yxxm3W02k= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-335-EaRoQXwoMNGYgNQZz7DgTw-1; Wed, 10 Jun 2020 07:54:59 -0400 X-MC-Unique: EaRoQXwoMNGYgNQZz7DgTw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AD028107ACCA; Wed, 10 Jun 2020 11:54:58 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7C9E5D9D3; Wed, 10 Jun 2020 11:54:56 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Juan Quintela Subject: [PATCH v4 05/21] virtio-balloon: Rip out qemu_balloon_inhibit() Date: Wed, 10 Jun 2020 13:54:03 +0200 Message-Id: <20200610115419.51688-6-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The only remaining special case is postcopy. It cannot handle concurrent discards yet, which would result in requesting already sent pages from the source. Special-case it in virtio-balloon instead. Introduce migration_in_incoming_postcopy(), to find out if incoming postcopy is active. Cc: "Michael S. Tsirkin" Cc: Juan Quintela Cc: "Dr. David Alan Gilbert" Signed-off-by: David Hildenbrand Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Michael S. Tsirkin --- balloon.c | 18 ------------------ hw/virtio/virtio-balloon.c | 8 +++++++- include/migration/misc.h | 2 ++ include/sysemu/balloon.h | 2 -- migration/migration.c | 7 +++++++ migration/postcopy-ram.c | 23 ----------------------- 6 files changed, 16 insertions(+), 44 deletions(-) diff --git a/balloon.c b/balloon.c index 5fff79523a..354408c6ea 100644 --- a/balloon.c +++ b/balloon.c @@ -36,24 +36,6 @@ static QEMUBalloonEvent *balloon_event_fn; static QEMUBalloonStatus *balloon_stat_fn; static void *balloon_opaque; -static int balloon_inhibit_count; - -bool qemu_balloon_is_inhibited(void) -{ - return atomic_read(&balloon_inhibit_count) > 0 || - ram_block_discard_is_disabled(); -} - -void qemu_balloon_inhibit(bool state) -{ - if (state) { - atomic_inc(&balloon_inhibit_count); - } else { - atomic_dec(&balloon_inhibit_count); - } - - assert(atomic_read(&balloon_inhibit_count) >= 0); -} static bool have_balloon(Error **errp) { diff --git a/hw/virtio/virtio-balloon.c b/hw/virtio/virtio-balloon.c index 065cd450f1..5ce2f956df 100644 --- a/hw/virtio/virtio-balloon.c +++ b/hw/virtio/virtio-balloon.c @@ -63,6 +63,12 @@ static bool virtio_balloon_pbp_matches(PartiallyBalloonedPage *pbp, return pbp->base_gpa == base_gpa; } +static bool virtio_balloon_inhibited(void) +{ + /* Postcopy cannot deal with concurrent discards, so it's special. */ + return ram_block_discard_is_disabled() || migration_in_incoming_postcopy(); +} + static void balloon_inflate_page(VirtIOBalloon *balloon, MemoryRegion *mr, hwaddr mr_offset, PartiallyBalloonedPage *pbp) @@ -360,7 +366,7 @@ static void virtio_balloon_handle_output(VirtIODevice *vdev, VirtQueue *vq) trace_virtio_balloon_handle_output(memory_region_name(section.mr), pa); - if (!qemu_balloon_is_inhibited()) { + if (!virtio_balloon_inhibited()) { if (vq == s->ivq) { balloon_inflate_page(s, section.mr, section.offset_within_region, &pbp); diff --git a/include/migration/misc.h b/include/migration/misc.h index d2762257aa..34e7d75713 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -69,6 +69,8 @@ bool migration_has_failed(MigrationState *); /* ...and after the device transmission */ bool migration_in_postcopy_after_devices(MigrationState *); void migration_global_dump(Monitor *mon); +/* True if incomming migration entered POSTCOPY_INCOMING_DISCARD */ +bool migration_in_incoming_postcopy(void); /* migration/block-dirty-bitmap.c */ void dirty_bitmap_mig_init(void); diff --git a/include/sysemu/balloon.h b/include/sysemu/balloon.h index aea0c44985..20a2defe3a 100644 --- a/include/sysemu/balloon.h +++ b/include/sysemu/balloon.h @@ -23,7 +23,5 @@ typedef void (QEMUBalloonStatus)(void *opaque, BalloonInfo *info); int qemu_add_balloon_handler(QEMUBalloonEvent *event_func, QEMUBalloonStatus *stat_func, void *opaque); void qemu_remove_balloon_handler(void *opaque); -bool qemu_balloon_is_inhibited(void); -void qemu_balloon_inhibit(bool state); #endif diff --git a/migration/migration.c b/migration/migration.c index b63ad91d34..14856cc930 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1772,6 +1772,13 @@ bool migration_in_postcopy_after_devices(MigrationState *s) return migration_in_postcopy() && s->postcopy_after_devices; } +bool migration_in_incoming_postcopy(void) +{ + PostcopyState ps = postcopy_state_get(); + + return ps >= POSTCOPY_INCOMING_DISCARD && ps < POSTCOPY_INCOMING_END; +} + bool migration_is_idle(void) { MigrationState *s = current_migration; diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index a36402722b..b41a9fe2fd 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -27,7 +27,6 @@ #include "qemu/notify.h" #include "qemu/rcu.h" #include "sysemu/sysemu.h" -#include "sysemu/balloon.h" #include "qemu/error-report.h" #include "trace.h" #include "hw/boards.h" @@ -520,20 +519,6 @@ int postcopy_ram_incoming_init(MigrationIncomingState *mis) return 0; } -/* - * Manage a single vote to the QEMU balloon inhibitor for all postcopy usage, - * last caller wins. - */ -static void postcopy_balloon_inhibit(bool state) -{ - static bool cur_state = false; - - if (state != cur_state) { - qemu_balloon_inhibit(state); - cur_state = state; - } -} - /* * At the end of a migration where postcopy_ram_incoming_init was called. */ @@ -565,8 +550,6 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) mis->have_fault_thread = false; } - postcopy_balloon_inhibit(false); - if (enable_mlock) { if (os_mlock() < 0) { error_report("mlock: %s", strerror(errno)); @@ -1160,12 +1143,6 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) } memset(mis->postcopy_tmp_zero_page, '\0', mis->largest_page_size); - /* - * Ballooning can mark pages as absent while we're postcopying - * that would cause false userfaults. - */ - postcopy_balloon_inhibit(true); - trace_postcopy_ram_enable_notify(); return 0; From patchwork Wed Jun 10 11:54:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597611 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6C2A14B7 for ; Wed, 10 Jun 2020 11:55:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C2E4F2074B for ; Wed, 10 Jun 2020 11:55:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gaiF9uKQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728769AbgFJLzL (ORCPT ); Wed, 10 Jun 2020 07:55:11 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:54041 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728774AbgFJLzK (ORCPT ); Wed, 10 Jun 2020 07:55:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790109; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/bRpzWo8mGWvq3enPFtqLN32egzNgjYQizck8xugJ+M=; b=gaiF9uKQOEnw/xpkr2H7jSIDPpUY0a8EkIT+HSl5WeYeZfUaeTaJrtDs4oFVQnp7sC/jqu B9WKyyjXA/b7VutLxn6fCUF2Z/gx2IWngFLBvakDUo9s1GVSXd4/uVCQsuEs7PtNrGOMrK uRFHXqVpIW89thfAaY8TvlbdOnF/jzE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-502-DLX1pjx1OeOHbokAJF5eVg-1; Wed, 10 Jun 2020 07:55:07 -0400 X-MC-Unique: DLX1pjx1OeOHbokAJF5eVg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7E40A461; Wed, 10 Jun 2020 11:55:06 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0AE695D9D3; Wed, 10 Jun 2020 11:54:58 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 06/21] target/i386: sev: Use ram_block_discard_disable() Date: Wed, 10 Jun 2020 13:54:04 +0200 Message-Id: <20200610115419.51688-7-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org AMD SEV will pin all guest memory, mark discarding of RAM broken. At the time this is called, we cannot have anyone active that relies on discards to work properly - let's still implement error handling. Reviewed-by: Dr. David Alan Gilbert Cc: "Michael S. Tsirkin" Cc: Paolo Bonzini Cc: Richard Henderson Cc: Eduardo Habkost Signed-off-by: David Hildenbrand --- target/i386/sev.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/target/i386/sev.c b/target/i386/sev.c index 51cdbe5496..4a4863db28 100644 --- a/target/i386/sev.c +++ b/target/i386/sev.c @@ -649,6 +649,12 @@ sev_guest_init(const char *id) uint32_t host_cbitpos; struct sev_user_data_status status = {}; + ret = ram_block_discard_disable(true); + if (ret) { + error_report("%s: cannot disable RAM discard", __func__); + return NULL; + } + sev_state = s = g_new0(SEVState, 1); s->sev_info = lookup_sev_guest_info(id); if (!s->sev_info) { @@ -724,6 +730,7 @@ sev_guest_init(const char *id) err: g_free(sev_state); sev_state = NULL; + ram_block_discard_disable(false); return NULL; } From patchwork Wed Jun 10 11:54:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597613 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7DC1B138C for ; Wed, 10 Jun 2020 11:55:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 62C852072E for ; Wed, 10 Jun 2020 11:55:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="a84Zw+vp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728783AbgFJLzP (ORCPT ); Wed, 10 Jun 2020 07:55:15 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:45106 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728779AbgFJLzP (ORCPT ); Wed, 10 Jun 2020 07:55:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K0bxEZTNG5tW1NNkUgKUn7jrte0My/UUwSbEJ+zpHfA=; b=a84Zw+vpW7dNbvZDqFH/x2jKk9eNdVXMhGnu+CGxLEQrrEGwQT+pLzPSzLs2ExiJ28Tx6o QBxjl2p3sNFELjLifG9U/abek9F5kl6Ps3VNJJp+6jh59bMaaFYXl42Wu+0Ct84v+fhWOi ueeJDh76D1BeSDpuhQVq/+BB2lHCZYQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-36-xRLBABodNWGxytLWjIPfRQ-1; Wed, 10 Jun 2020 07:55:09 -0400 X-MC-Unique: xRLBABodNWGxytLWjIPfRQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D210A193F562; Wed, 10 Jun 2020 11:55:08 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id CED915D9D3; Wed, 10 Jun 2020 11:55:06 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Juan Quintela Subject: [PATCH v4 07/21] migration/rdma: Use ram_block_discard_disable() Date: Wed, 10 Jun 2020 13:54:05 +0200 Message-Id: <20200610115419.51688-8-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org RDMA will pin all guest memory (as documented in docs/rdma.txt). We want to disable RAM block discards - however, to keep it simple use ram_block_discard_is_required() instead of inhibiting. Note: It is not sufficient to limit disabling to pin_all. Even when only conditionally pinning 1 MB chunks, as soon as one page within such a chunk was discarded and one page not, the discarded pages will be pinned as well. Reviewed-by: Dr. David Alan Gilbert Cc: "Michael S. Tsirkin" Cc: Juan Quintela Cc: "Dr. David Alan Gilbert" Signed-off-by: David Hildenbrand --- migration/rdma.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index ec45d33ba3..bbe6f36627 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -29,6 +29,7 @@ #include "qemu/sockets.h" #include "qemu/bitmap.h" #include "qemu/coroutine.h" +#include "exec/memory.h" #include #include #include @@ -4017,8 +4018,14 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp) Error *local_err = NULL; trace_rdma_start_incoming_migration(); - rdma = qemu_rdma_data_init(host_port, &local_err); + /* Avoid ram_block_discard_disable(), cannot change during migration. */ + if (ram_block_discard_is_required()) { + error_setg(errp, "RDMA: cannot disable RAM discard"); + return; + } + + rdma = qemu_rdma_data_init(host_port, &local_err); if (rdma == NULL) { goto err; } @@ -4067,10 +4074,17 @@ void rdma_start_outgoing_migration(void *opaque, const char *host_port, Error **errp) { MigrationState *s = opaque; - RDMAContext *rdma = qemu_rdma_data_init(host_port, errp); RDMAContext *rdma_return_path = NULL; + RDMAContext *rdma; int ret = 0; + /* Avoid ram_block_discard_disable(), cannot change during migration. */ + if (ram_block_discard_is_required()) { + error_setg(errp, "RDMA: cannot disable RAM discard"); + return; + } + + rdma = qemu_rdma_data_init(host_port, errp); if (rdma == NULL) { goto err; } From patchwork Wed Jun 10 11:54:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597615 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6FF09138C for ; Wed, 10 Jun 2020 11:55:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53D282072E for ; Wed, 10 Jun 2020 11:55:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TAnec0pD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728779AbgFJLzT (ORCPT ); Wed, 10 Jun 2020 07:55:19 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:26285 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728775AbgFJLzS (ORCPT ); Wed, 10 Jun 2020 07:55:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3bFU0P1MN91YjNPtnZUSniyH3X2xNqVX/h+DSb3PV28=; b=TAnec0pDqo0p3B1UI6RlDt4RiYoPSGhQrb1I1vnMp38qnfgXcP27YHdywt+3uhIbBXbdxb iSs0YsQQdovb0YFpm0OmVZ6kWj03y80Ad1WHO9o6RhP9hASafC+FoJu5rrkdtf2VCfmkln cT4iOiLCxcDK34/WhzvAlM/1Dhal1r8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-26-c7OyNNCUO5iwegenYJpFYg-1; Wed, 10 Jun 2020 07:55:12 -0400 X-MC-Unique: c7OyNNCUO5iwegenYJpFYg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 80386107ACCA; Wed, 10 Jun 2020 11:55:11 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2EC0B5D9D3; Wed, 10 Jun 2020 11:55:09 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Lukas Straub , Hailiang Zhang , Juan Quintela Subject: [PATCH v4 08/21] migration/colo: Use ram_block_discard_disable() Date: Wed, 10 Jun 2020 13:54:06 +0200 Message-Id: <20200610115419.51688-9-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org COLO will copy all memory in a RAM block, disable discarding of RAM. Reviewed-by: Dr. David Alan Gilbert Tested-by: Lukas Straub Cc: "Michael S. Tsirkin" Cc: Hailiang Zhang Cc: Juan Quintela Cc: "Dr. David Alan Gilbert" Signed-off-by: David Hildenbrand --- include/migration/colo.h | 2 +- migration/migration.c | 8 +++++++- migration/savevm.c | 11 +++++++++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/include/migration/colo.h b/include/migration/colo.h index 1636e6f907..768e1f04c3 100644 --- a/include/migration/colo.h +++ b/include/migration/colo.h @@ -25,7 +25,7 @@ void migrate_start_colo_process(MigrationState *s); bool migration_in_colo_state(void); /* loadvm */ -void migration_incoming_enable_colo(void); +int migration_incoming_enable_colo(void); void migration_incoming_disable_colo(void); bool migration_incoming_colo_enabled(void); void *colo_process_incoming_thread(void *opaque); diff --git a/migration/migration.c b/migration/migration.c index 14856cc930..0f6799f5d2 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -338,12 +338,18 @@ bool migration_incoming_colo_enabled(void) void migration_incoming_disable_colo(void) { + ram_block_discard_disable(false); migration_colo_enabled = false; } -void migration_incoming_enable_colo(void) +int migration_incoming_enable_colo(void) { + if (ram_block_discard_disable(true)) { + error_report("COLO: cannot disable RAM discard"); + return -EBUSY; + } migration_colo_enabled = true; + return 0; } void migrate_add_address(SocketAddress *address) diff --git a/migration/savevm.c b/migration/savevm.c index c00a6807d9..19b4f9600d 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2111,8 +2111,15 @@ static int loadvm_handle_recv_bitmap(MigrationIncomingState *mis, static int loadvm_process_enable_colo(MigrationIncomingState *mis) { - migration_incoming_enable_colo(); - return colo_init_ram_cache(); + int ret = migration_incoming_enable_colo(); + + if (!ret) { + ret = colo_init_ram_cache(); + if (ret) { + migration_incoming_disable_colo(); + } + } + return ret; } /* From patchwork Wed Jun 10 11:54:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597617 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4190F138C for ; Wed, 10 Jun 2020 11:55:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 230902072E for ; Wed, 10 Jun 2020 11:55:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="F60vBOTL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728792AbgFJLz1 (ORCPT ); Wed, 10 Jun 2020 07:55:27 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:51902 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728778AbgFJLz0 (ORCPT ); Wed, 10 Jun 2020 07:55:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790124; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1qjWuHyFh5mR0tjmbdCRWfZkFdOjq2WlRSygc05EIvo=; b=F60vBOTLVGP379NE8XI1mlFnrWeWT2C3jm11iuV/14vxM95gspKpbq7OLZW6TouCsdJ9RR zxFTh/8m6Vg7wxbnVcre+Q+TM2MUufCu0lqNPFLrsSlJjJLc8/kkQsMSqm+NtIYa41Ob2e YJpixqS/rEMLx35ZtG3NHemGweKzS1I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-403-ZPMKjspYOj6ELdAJhSbzdQ-1; Wed, 10 Jun 2020 07:55:22 -0400 X-MC-Unique: ZPMKjspYOj6ELdAJhSbzdQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 78E7C83DC0F; Wed, 10 Jun 2020 11:55:21 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id D22D75D9D3; Wed, 10 Jun 2020 11:55:11 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 09/21] linux-headers: update to contain virtio-mem Date: Wed, 10 Jun 2020 13:54:07 +0200 Message-Id: <20200610115419.51688-10-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org To be merged hopefully soon. Then, we can replace this by a proper header sync. Cc: "Michael S. Tsirkin" Signed-off-by: David Hildenbrand --- include/standard-headers/linux/virtio_ids.h | 1 + include/standard-headers/linux/virtio_mem.h | 211 ++++++++++++++++++++ 2 files changed, 212 insertions(+) create mode 100644 include/standard-headers/linux/virtio_mem.h diff --git a/include/standard-headers/linux/virtio_ids.h b/include/standard-headers/linux/virtio_ids.h index ecc27a1740..b052355ac7 100644 --- a/include/standard-headers/linux/virtio_ids.h +++ b/include/standard-headers/linux/virtio_ids.h @@ -44,6 +44,7 @@ #define VIRTIO_ID_VSOCK 19 /* virtio vsock transport */ #define VIRTIO_ID_CRYPTO 20 /* virtio crypto */ #define VIRTIO_ID_IOMMU 23 /* virtio IOMMU */ +#define VIRTIO_ID_MEM 24 /* virtio mem */ #define VIRTIO_ID_FS 26 /* virtio filesystem */ #define VIRTIO_ID_PMEM 27 /* virtio pmem */ #define VIRTIO_ID_MAC80211_HWSIM 29 /* virtio mac80211-hwsim */ diff --git a/include/standard-headers/linux/virtio_mem.h b/include/standard-headers/linux/virtio_mem.h new file mode 100644 index 0000000000..05e5ade75d --- /dev/null +++ b/include/standard-headers/linux/virtio_mem.h @@ -0,0 +1,211 @@ +/* SPDX-License-Identifier: BSD-3-Clause */ +/* + * Virtio Mem Device + * + * Copyright Red Hat, Inc. 2020 + * + * Authors: + * David Hildenbrand + * + * This header is BSD licensed so anyone can use the definitions + * to implement compatible drivers/servers: + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of IBM nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL IBM OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef _LINUX_VIRTIO_MEM_H +#define _LINUX_VIRTIO_MEM_H + +#include "standard-headers/linux/types.h" +#include "standard-headers/linux/virtio_types.h" +#include "standard-headers/linux/virtio_ids.h" +#include "standard-headers/linux/virtio_config.h" + +/* + * Each virtio-mem device manages a dedicated region in physical address + * space. Each device can belong to a single NUMA node, multiple devices + * for a single NUMA node are possible. A virtio-mem device is like a + * "resizable DIMM" consisting of small memory blocks that can be plugged + * or unplugged. The device driver is responsible for (un)plugging memory + * blocks on demand. + * + * Virtio-mem devices can only operate on their assigned memory region in + * order to (un)plug memory. A device cannot (un)plug memory belonging to + * other devices. + * + * The "region_size" corresponds to the maximum amount of memory that can + * be provided by a device. The "size" corresponds to the amount of memory + * that is currently plugged. "requested_size" corresponds to a request + * from the device to the device driver to (un)plug blocks. The + * device driver should try to (un)plug blocks in order to reach the + * "requested_size". It is impossible to plug more memory than requested. + * + * The "usable_region_size" represents the memory region that can actually + * be used to (un)plug memory. It is always at least as big as the + * "requested_size" and will grow dynamically. It will only shrink when + * explicitly triggered (VIRTIO_MEM_REQ_UNPLUG). + * + * There are no guarantees what will happen if unplugged memory is + * read/written. Such memory should, in general, not be touched. E.g., + * even writing might succeed, but the values will simply be discarded at + * random points in time. + * + * It can happen that the device cannot process a request, because it is + * busy. The device driver has to retry later. + * + * Usually, during system resets all memory will get unplugged, so the + * device driver can start with a clean state. However, in specific + * scenarios (if the device is busy) it can happen that the device still + * has memory plugged. The device driver can request to unplug all memory + * (VIRTIO_MEM_REQ_UNPLUG) - which might take a while to succeed if the + * device is busy. + */ + +/* --- virtio-mem: feature bits --- */ + +/* node_id is an ACPI PXM and is valid */ +#define VIRTIO_MEM_F_ACPI_PXM 0 + + +/* --- virtio-mem: guest -> host requests --- */ + +/* request to plug memory blocks */ +#define VIRTIO_MEM_REQ_PLUG 0 +/* request to unplug memory blocks */ +#define VIRTIO_MEM_REQ_UNPLUG 1 +/* request to unplug all blocks and shrink the usable size */ +#define VIRTIO_MEM_REQ_UNPLUG_ALL 2 +/* request information about the plugged state of memory blocks */ +#define VIRTIO_MEM_REQ_STATE 3 + +struct virtio_mem_req_plug { + __virtio64 addr; + __virtio16 nb_blocks; + __virtio16 padding[3]; +}; + +struct virtio_mem_req_unplug { + __virtio64 addr; + __virtio16 nb_blocks; + __virtio16 padding[3]; +}; + +struct virtio_mem_req_state { + __virtio64 addr; + __virtio16 nb_blocks; + __virtio16 padding[3]; +}; + +struct virtio_mem_req { + __virtio16 type; + __virtio16 padding[3]; + + union { + struct virtio_mem_req_plug plug; + struct virtio_mem_req_unplug unplug; + struct virtio_mem_req_state state; + } u; +}; + + +/* --- virtio-mem: host -> guest response --- */ + +/* + * Request processed successfully, applicable for + * - VIRTIO_MEM_REQ_PLUG + * - VIRTIO_MEM_REQ_UNPLUG + * - VIRTIO_MEM_REQ_UNPLUG_ALL + * - VIRTIO_MEM_REQ_STATE + */ +#define VIRTIO_MEM_RESP_ACK 0 +/* + * Request denied - e.g. trying to plug more than requested, applicable for + * - VIRTIO_MEM_REQ_PLUG + */ +#define VIRTIO_MEM_RESP_NACK 1 +/* + * Request cannot be processed right now, try again later, applicable for + * - VIRTIO_MEM_REQ_PLUG + * - VIRTIO_MEM_REQ_UNPLUG + * - VIRTIO_MEM_REQ_UNPLUG_ALL + */ +#define VIRTIO_MEM_RESP_BUSY 2 +/* + * Error in request (e.g. addresses/alignment), applicable for + * - VIRTIO_MEM_REQ_PLUG + * - VIRTIO_MEM_REQ_UNPLUG + * - VIRTIO_MEM_REQ_STATE + */ +#define VIRTIO_MEM_RESP_ERROR 3 + + +/* State of memory blocks is "plugged" */ +#define VIRTIO_MEM_STATE_PLUGGED 0 +/* State of memory blocks is "unplugged" */ +#define VIRTIO_MEM_STATE_UNPLUGGED 1 +/* State of memory blocks is "mixed" */ +#define VIRTIO_MEM_STATE_MIXED 2 + +struct virtio_mem_resp_state { + __virtio16 state; +}; + +struct virtio_mem_resp { + __virtio16 type; + __virtio16 padding[3]; + + union { + struct virtio_mem_resp_state state; + } u; +}; + +/* --- virtio-mem: configuration --- */ + +struct virtio_mem_config { + /* Block size and alignment. Cannot change. */ + uint64_t block_size; + /* Valid with VIRTIO_MEM_F_ACPI_PXM. Cannot change. */ + uint16_t node_id; + uint8_t padding[6]; + /* Start address of the memory region. Cannot change. */ + uint64_t addr; + /* Region size (maximum). Cannot change. */ + uint64_t region_size; + /* + * Currently usable region size. Can grow up to region_size. Can + * shrink due to VIRTIO_MEM_REQ_UNPLUG_ALL (in which case no config + * update will be sent). + */ + uint64_t usable_region_size; + /* + * Currently used size. Changes due to plug/unplug requests, but no + * config updates will be sent. + */ + uint64_t plugged_size; + /* Requested size. New plug requests cannot exceed it. Can change. */ + uint64_t requested_size; +}; + +#endif /* _LINUX_VIRTIO_MEM_H */ From patchwork Wed Jun 10 11:54:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597621 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD4E5138C for ; Wed, 10 Jun 2020 11:55:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8F81920760 for ; Wed, 10 Jun 2020 11:55:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XIRimxu8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728819AbgFJLzh (ORCPT ); Wed, 10 Jun 2020 07:55:37 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:30787 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728806AbgFJLzg (ORCPT ); Wed, 10 Jun 2020 07:55:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=T5v3hYYm9DIstRnaIeNybGzu7FzMeXYFYKVxfwtLhBE=; b=XIRimxu87TSg+MlM4XTXn7Pmjj9HyZPP9maI7zxfI7Tw0eu3aDlQ74adw8TlC+rm9ToeYX RZHfDN5GdwnuLcY2Q+iE78jNsBDaQ2XJTFFJjxS4ewtR6dfWMoHbM7oRWTBT5aeZjf+/u7 bELKa8UGL+QbCvSHaWer0/HVbdgsHoY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-175-kgMDn9g2NhuBjnXkg83unw-1; Wed, 10 Jun 2020 07:55:28 -0400 X-MC-Unique: kgMDn9g2NhuBjnXkg83unw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 34C88DC09; Wed, 10 Jun 2020 11:55:27 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id CFE295D9D3; Wed, 10 Jun 2020 11:55:21 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Eric Blake , Markus Armbruster , Igor Mammedov Subject: [PATCH v4 10/21] virtio-mem: Paravirtualized memory hot(un)plug Date: Wed, 10 Jun 2020 13:54:08 +0200 Message-Id: <20200610115419.51688-11-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is the very basic/initial version of virtio-mem. An introduction to virtio-mem can be found in the Linux kernel driver [1]. While it can be used in the current state for hotplug of a smaller amount of memory, it will heavily benefit from resizeable memory regions in the future. Each virtio-mem device manages a memory region (provided via a memory backend). After requested by the hypervisor ("requested-size"), the guest can try to plug/unplug blocks of memory within that region, in order to reach the requested size. Initially, and after a reboot, all memory is unplugged (except in special cases - reboot during postcopy). The guest may only try to plug/unplug blocks of memory within the usable region size. The usable region size is a little bigger than the requested size, to give the device driver some flexibility. The usable region size will only grow, except on reboots or when all memory is requested to get unplugged. The guest can never plug more memory than requested. Unplugged memory will get zapped/discarded, similar to in a balloon device. The block size is variable, however, it is always chosen in a way such that THP splits are avoided (e.g., 2MB). The state of each block (plugged/unplugged) is tracked in a bitmap. As virtio-mem devices (e.g., virtio-mem-pci) will be memory devices, we now expose "VirtioMEMDeviceInfo" via "query-memory-devices". -------------------------------------------------------------------------- There are two important follow-up items that are in the works: 1. Resizeable memory regions: Use resizeable allocations/RAM blocks to grow/shrink along with the usable region size. This avoids creating initially very big VMAs, RAM blocks, and KVM slots. 2. Protection of unplugged memory: Make sure the gust cannot actually make use of unplugged memory. Other follow-up items that are in the works: 1. Exclude unplugged memory during migration (via precopy notifier). 2. Handle remapping of memory. 3. Support for other architectures. -------------------------------------------------------------------------- Example usage (virtio-mem-pci is introduced in follow-up patches): Start QEMU with two virtio-mem devices (one per NUMA node): $ qemu-system-x86_64 -m 4G,maxmem=20G \ -smp sockets=2,cores=2 \ -numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 \ [...] -object memory-backend-ram,id=mem0,size=8G \ -device virtio-mem-pci,id=vm0,memdev=mem0,node=0,requested-size=0M \ -object memory-backend-ram,id=mem1,size=8G \ -device virtio-mem-pci,id=vm1,memdev=mem1,node=1,requested-size=1G Query the configuration: (qemu) info memory-devices Memory device [virtio-mem]: "vm0" memaddr: 0x140000000 node: 0 requested-size: 0 size: 0 max-size: 8589934592 block-size: 2097152 memdev: /objects/mem0 Memory device [virtio-mem]: "vm1" memaddr: 0x340000000 node: 1 requested-size: 1073741824 size: 1073741824 max-size: 8589934592 block-size: 2097152 memdev: /objects/mem1 Add some memory to node 0: (qemu) qom-set vm0 requested-size 500M Remove some memory from node 1: (qemu) qom-set vm1 requested-size 200M Query the configuration again: (qemu) info memory-devices Memory device [virtio-mem]: "vm0" memaddr: 0x140000000 node: 0 requested-size: 524288000 size: 524288000 max-size: 8589934592 block-size: 2097152 memdev: /objects/mem0 Memory device [virtio-mem]: "vm1" memaddr: 0x340000000 node: 1 requested-size: 209715200 size: 209715200 max-size: 8589934592 block-size: 2097152 memdev: /objects/mem1 [1] https://lkml.kernel.org/r/20200311171422.10484-1-david@redhat.com Cc: "Michael S. Tsirkin" Cc: Eric Blake Cc: Markus Armbruster Cc: "Dr. David Alan Gilbert" Cc: Igor Mammedov Signed-off-by: David Hildenbrand --- hw/virtio/Kconfig | 11 + hw/virtio/Makefile.objs | 1 + hw/virtio/virtio-mem.c | 724 +++++++++++++++++++++++++++++++++ include/hw/virtio/virtio-mem.h | 78 ++++ qapi/misc.json | 39 +- 5 files changed, 852 insertions(+), 1 deletion(-) create mode 100644 hw/virtio/virtio-mem.c create mode 100644 include/hw/virtio/virtio-mem.h diff --git a/hw/virtio/Kconfig b/hw/virtio/Kconfig index 83122424fa..0eda25c4e1 100644 --- a/hw/virtio/Kconfig +++ b/hw/virtio/Kconfig @@ -47,3 +47,14 @@ config VIRTIO_PMEM depends on VIRTIO depends on VIRTIO_PMEM_SUPPORTED select MEM_DEVICE + +config VIRTIO_MEM_SUPPORTED + bool + +config VIRTIO_MEM + bool + default y + depends on VIRTIO + depends on LINUX + depends on VIRTIO_MEM_SUPPORTED + select MEM_DEVICE diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs index 4e4d39a0a4..7df70e977e 100644 --- a/hw/virtio/Makefile.objs +++ b/hw/virtio/Makefile.objs @@ -18,6 +18,7 @@ common-obj-$(call land,$(CONFIG_VIRTIO_PMEM),$(CONFIG_VIRTIO_PCI)) += virtio-pme obj-$(call land,$(CONFIG_VHOST_USER_FS),$(CONFIG_VIRTIO_PCI)) += vhost-user-fs-pci.o obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock.o +obj-$(CONFIG_VIRTIO_MEM) += virtio-mem.o ifeq ($(CONFIG_VIRTIO_PCI),y) obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock-pci.o diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c new file mode 100644 index 0000000000..d8a0c974d3 --- /dev/null +++ b/hw/virtio/virtio-mem.c @@ -0,0 +1,724 @@ +/* + * Virtio MEM device + * + * Copyright (C) 2020 Red Hat, Inc. + * + * Authors: + * David Hildenbrand + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "qemu-common.h" +#include "qemu/iov.h" +#include "qemu/cutils.h" +#include "qemu/error-report.h" +#include "qemu/units.h" +#include "sysemu/numa.h" +#include "sysemu/sysemu.h" +#include "sysemu/reset.h" +#include "hw/virtio/virtio.h" +#include "hw/virtio/virtio-bus.h" +#include "hw/virtio/virtio-access.h" +#include "hw/virtio/virtio-mem.h" +#include "qapi/error.h" +#include "qapi/visitor.h" +#include "exec/ram_addr.h" +#include "migration/misc.h" +#include "hw/boards.h" +#include "hw/qdev-properties.h" +#include "config-devices.h" + +/* + * Use QEMU_VMALLOC_ALIGN, so no THP will have to be split when unplugging + * memory (e.g., 2MB on x86_64). + */ +#define VIRTIO_MEM_MIN_BLOCK_SIZE QEMU_VMALLOC_ALIGN +/* + * Size the usable region bigger than the requested size if possible. Esp. + * Linux guests will only add (aligned) memory blocks in case they fully + * fit into the usable region, but plug+online only a subset of the pages. + * The memory block size corresponds mostly to the section size. + * + * This allows e.g., to add 20MB with a section size of 128MB on x86_64, and + * a section size of 1GB on arm64 (as long as the start address is properly + * aligned, similar to ordinary DIMMs). + * + * We can change this at any time and maybe even make it configurable if + * necessary (as the section size can change). But it's more likely that the + * section size will rather get smaller and not bigger over time. + */ +#if defined(__x86_64__) +#define VIRTIO_MEM_USABLE_EXTENT (2 * (128 * MiB)) +#else +#error VIRTIO_MEM_USABLE_EXTENT not defined +#endif + +static bool virtio_mem_is_busy(void) +{ + /* + * Postcopy cannot handle concurrent discards and we don't want to migrate + * pages on-demand with stale content when plugging new blocks. + */ + return migration_in_incoming_postcopy(); +} + +static bool virtio_mem_test_bitmap(VirtIOMEM *vmem, uint64_t start_gpa, + uint64_t size, bool plugged) +{ + const unsigned long first_bit = (start_gpa - vmem->addr) / vmem->block_size; + const unsigned long last_bit = first_bit + (size / vmem->block_size) - 1; + unsigned long found_bit; + + /* We fake a shorter bitmap to avoid searching too far. */ + if (plugged) { + found_bit = find_next_zero_bit(vmem->bitmap, last_bit + 1, first_bit); + } else { + found_bit = find_next_bit(vmem->bitmap, last_bit + 1, first_bit); + } + return found_bit > last_bit; +} + +static void virtio_mem_set_bitmap(VirtIOMEM *vmem, uint64_t start_gpa, + uint64_t size, bool plugged) +{ + const unsigned long bit = (start_gpa - vmem->addr) / vmem->block_size; + const unsigned long nbits = size / vmem->block_size; + + if (plugged) { + bitmap_set(vmem->bitmap, bit, nbits); + } else { + bitmap_clear(vmem->bitmap, bit, nbits); + } +} + +static void virtio_mem_send_response(VirtIOMEM *vmem, VirtQueueElement *elem, + struct virtio_mem_resp *resp) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(vmem); + VirtQueue *vq = vmem->vq; + + iov_from_buf(elem->in_sg, elem->in_num, 0, resp, sizeof(*resp)); + + virtqueue_push(vq, elem, sizeof(*resp)); + virtio_notify(vdev, vq); +} + +static void virtio_mem_send_response_simple(VirtIOMEM *vmem, + VirtQueueElement *elem, + uint16_t type) +{ + struct virtio_mem_resp resp = { + .type = cpu_to_le16(type), + }; + + virtio_mem_send_response(vmem, elem, &resp); +} + +static bool virtio_mem_valid_range(VirtIOMEM *vmem, uint64_t gpa, uint64_t size) +{ + if (!QEMU_IS_ALIGNED(gpa, vmem->block_size)) { + return false; + } + if (gpa + size < gpa || !size) { + return false; + } + if (gpa < vmem->addr || gpa >= vmem->addr + vmem->usable_region_size) { + return false; + } + if (gpa + size > vmem->addr + vmem->usable_region_size) { + return false; + } + return true; +} + +static int virtio_mem_set_block_state(VirtIOMEM *vmem, uint64_t start_gpa, + uint64_t size, bool plug) +{ + const uint64_t offset = start_gpa - vmem->addr; + int ret; + + if (virtio_mem_is_busy()) { + return -EBUSY; + } + + if (!plug) { + ret = ram_block_discard_range(vmem->memdev->mr.ram_block, offset, size); + if (ret) { + error_report("Unexpected error discarding RAM: %s", + strerror(-ret)); + return -EBUSY; + } + } + virtio_mem_set_bitmap(vmem, start_gpa, size, plug); + return 0; +} + +static int virtio_mem_state_change_request(VirtIOMEM *vmem, uint64_t gpa, + uint16_t nb_blocks, bool plug) +{ + const uint64_t size = nb_blocks * vmem->block_size; + int ret; + + if (!virtio_mem_valid_range(vmem, gpa, size)) { + return VIRTIO_MEM_RESP_ERROR; + } + + if (plug && (vmem->size + size > vmem->requested_size)) { + return VIRTIO_MEM_RESP_NACK; + } + + /* test if really all blocks are in the opposite state */ + if (!virtio_mem_test_bitmap(vmem, gpa, size, !plug)) { + return VIRTIO_MEM_RESP_ERROR; + } + + ret = virtio_mem_set_block_state(vmem, gpa, size, plug); + if (ret) { + return VIRTIO_MEM_RESP_BUSY; + } + if (plug) { + vmem->size += size; + } else { + vmem->size -= size; + } + return VIRTIO_MEM_RESP_ACK; +} + +static void virtio_mem_plug_request(VirtIOMEM *vmem, VirtQueueElement *elem, + struct virtio_mem_req *req) +{ + const uint64_t gpa = le64_to_cpu(req->u.plug.addr); + const uint16_t nb_blocks = le16_to_cpu(req->u.plug.nb_blocks); + uint16_t type; + + type = virtio_mem_state_change_request(vmem, gpa, nb_blocks, true); + virtio_mem_send_response_simple(vmem, elem, type); +} + +static void virtio_mem_unplug_request(VirtIOMEM *vmem, VirtQueueElement *elem, + struct virtio_mem_req *req) +{ + const uint64_t gpa = le64_to_cpu(req->u.unplug.addr); + const uint16_t nb_blocks = le16_to_cpu(req->u.unplug.nb_blocks); + uint16_t type; + + type = virtio_mem_state_change_request(vmem, gpa, nb_blocks, false); + virtio_mem_send_response_simple(vmem, elem, type); +} + +static void virtio_mem_resize_usable_region(VirtIOMEM *vmem, + uint64_t requested_size, + bool can_shrink) +{ + uint64_t newsize = MIN(memory_region_size(&vmem->memdev->mr), + requested_size + VIRTIO_MEM_USABLE_EXTENT); + + if (!requested_size) { + newsize = 0; + } + + if (newsize < vmem->usable_region_size && !can_shrink) { + return; + } + + vmem->usable_region_size = newsize; +} + +static int virtio_mem_unplug_all(VirtIOMEM *vmem) +{ + RAMBlock *rb = vmem->memdev->mr.ram_block; + int ret; + + if (virtio_mem_is_busy()) { + return -EBUSY; + } + + ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb)); + if (ret) { + error_report("Unexpected error discarding RAM: %s", strerror(-ret)); + return -EBUSY; + } + bitmap_clear(vmem->bitmap, 0, vmem->bitmap_size); + vmem->size = 0; + + virtio_mem_resize_usable_region(vmem, vmem->requested_size, true); + return 0; +} + +static void virtio_mem_unplug_all_request(VirtIOMEM *vmem, + VirtQueueElement *elem) +{ + if (virtio_mem_unplug_all(vmem)) { + virtio_mem_send_response_simple(vmem, elem, VIRTIO_MEM_RESP_BUSY); + } else { + virtio_mem_send_response_simple(vmem, elem, VIRTIO_MEM_RESP_ACK); + } +} + +static void virtio_mem_state_request(VirtIOMEM *vmem, VirtQueueElement *elem, + struct virtio_mem_req *req) +{ + const uint16_t nb_blocks = le16_to_cpu(req->u.state.nb_blocks); + const uint64_t gpa = le64_to_cpu(req->u.state.addr); + const uint64_t size = nb_blocks * vmem->block_size; + struct virtio_mem_resp resp = { + .type = cpu_to_le16(VIRTIO_MEM_RESP_ACK), + }; + + if (!virtio_mem_valid_range(vmem, gpa, size)) { + virtio_mem_send_response_simple(vmem, elem, VIRTIO_MEM_RESP_ERROR); + return; + } + + if (virtio_mem_test_bitmap(vmem, gpa, size, true)) { + resp.u.state.state = cpu_to_le16(VIRTIO_MEM_STATE_PLUGGED); + } else if (virtio_mem_test_bitmap(vmem, gpa, size, false)) { + resp.u.state.state = cpu_to_le16(VIRTIO_MEM_STATE_UNPLUGGED); + } else { + resp.u.state.state = cpu_to_le16(VIRTIO_MEM_STATE_MIXED); + } + virtio_mem_send_response(vmem, elem, &resp); +} + +static void virtio_mem_handle_request(VirtIODevice *vdev, VirtQueue *vq) +{ + const int len = sizeof(struct virtio_mem_req); + VirtIOMEM *vmem = VIRTIO_MEM(vdev); + VirtQueueElement *elem; + struct virtio_mem_req req; + uint16_t type; + + while (true) { + elem = virtqueue_pop(vq, sizeof(VirtQueueElement)); + if (!elem) { + return; + } + + if (iov_to_buf(elem->out_sg, elem->out_num, 0, &req, len) < len) { + virtio_error(vdev, "virtio-mem protocol violation: invalid request" + " size: %d", len); + g_free(elem); + return; + } + + if (iov_size(elem->in_sg, elem->in_num) < + sizeof(struct virtio_mem_resp)) { + virtio_error(vdev, "virtio-mem protocol violation: not enough space" + " for response: %zu", + iov_size(elem->in_sg, elem->in_num)); + g_free(elem); + return; + } + + type = le16_to_cpu(req.type); + switch (type) { + case VIRTIO_MEM_REQ_PLUG: + virtio_mem_plug_request(vmem, elem, &req); + break; + case VIRTIO_MEM_REQ_UNPLUG: + virtio_mem_unplug_request(vmem, elem, &req); + break; + case VIRTIO_MEM_REQ_UNPLUG_ALL: + virtio_mem_unplug_all_request(vmem, elem); + break; + case VIRTIO_MEM_REQ_STATE: + virtio_mem_state_request(vmem, elem, &req); + break; + default: + virtio_error(vdev, "virtio-mem protocol violation: unknown request" + " type: %d", type); + g_free(elem); + return; + } + + g_free(elem); + } +} + +static void virtio_mem_get_config(VirtIODevice *vdev, uint8_t *config_data) +{ + VirtIOMEM *vmem = VIRTIO_MEM(vdev); + struct virtio_mem_config *config = (void *) config_data; + + config->block_size = cpu_to_le64(vmem->block_size); + config->node_id = cpu_to_le16(vmem->node); + config->requested_size = cpu_to_le64(vmem->requested_size); + config->plugged_size = cpu_to_le64(vmem->size); + config->addr = cpu_to_le64(vmem->addr); + config->region_size = cpu_to_le64(memory_region_size(&vmem->memdev->mr)); + config->usable_region_size = cpu_to_le64(vmem->usable_region_size); +} + +static uint64_t virtio_mem_get_features(VirtIODevice *vdev, uint64_t features, + Error **errp) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + if (ms->numa_state) { +#if defined(CONFIG_ACPI) + virtio_add_feature(&features, VIRTIO_MEM_F_ACPI_PXM); +#endif + } + return features; +} + +static void virtio_mem_system_reset(void *opaque) +{ + VirtIOMEM *vmem = VIRTIO_MEM(opaque); + + /* + * During usual resets, we will unplug all memory and shrink the usable + * region size. This is, however, not possible in all scenarios. Then, + * the guest has to deal with this manually (VIRTIO_MEM_REQ_UNPLUG_ALL). + */ + virtio_mem_unplug_all(vmem); +} + +static void virtio_mem_device_realize(DeviceState *dev, Error **errp) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + int nb_numa_nodes = ms->numa_state ? ms->numa_state->num_nodes : 0; + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VirtIOMEM *vmem = VIRTIO_MEM(dev); + uint64_t page_size; + RAMBlock *rb; + int ret; + + if (!vmem->memdev) { + error_setg(errp, "'%s' property is not set", VIRTIO_MEM_MEMDEV_PROP); + return; + } else if (host_memory_backend_is_mapped(vmem->memdev)) { + char *path = object_get_canonical_path_component(OBJECT(vmem->memdev)); + + error_setg(errp, "'%s' property specifies a busy memdev: %s", + VIRTIO_MEM_MEMDEV_PROP, path); + g_free(path); + return; + } else if (!memory_region_is_ram(&vmem->memdev->mr) || + memory_region_is_rom(&vmem->memdev->mr) || + !vmem->memdev->mr.ram_block) { + error_setg(errp, "'%s' property specifies an unsupported memdev", + VIRTIO_MEM_MEMDEV_PROP); + return; + } + + if ((nb_numa_nodes && vmem->node >= nb_numa_nodes) || + (!nb_numa_nodes && vmem->node)) { + error_setg(errp, "'%s' property has value '%" PRIu32 "', which exceeds" + "the number of numa nodes: %d", VIRTIO_MEM_NODE_PROP, + vmem->node, nb_numa_nodes ? nb_numa_nodes : 1); + return; + } + + if (enable_mlock) { + error_setg(errp, "Incompatible with mlock"); + return; + } + + rb = vmem->memdev->mr.ram_block; + page_size = qemu_ram_pagesize(rb); + + if (vmem->block_size < page_size) { + error_setg(errp, "'%s' property has to be at least the page size (0x%" + PRIx64 ")", VIRTIO_MEM_BLOCK_SIZE_PROP, page_size); + return; + } else if (!QEMU_IS_ALIGNED(vmem->requested_size, vmem->block_size)) { + error_setg(errp, "'%s' property has to be multiples of '%s' (0x%" PRIx64 + ")", VIRTIO_MEM_REQUESTED_SIZE_PROP, + VIRTIO_MEM_BLOCK_SIZE_PROP, vmem->block_size); + return; + } else if (!QEMU_IS_ALIGNED(memory_region_size(&vmem->memdev->mr), + vmem->block_size)) { + error_setg(errp, "'%s' property memdev size has to be multiples of" + "'%s' (0x%" PRIx64 ")", VIRTIO_MEM_MEMDEV_PROP, + VIRTIO_MEM_BLOCK_SIZE_PROP, vmem->block_size); + return; + } + + if (ram_block_discard_require(true)) { + error_setg(errp, "Discarding RAM is disabled"); + return; + } + + ret = ram_block_discard_range(rb, 0, qemu_ram_get_used_length(rb)); + if (ret) { + error_setg_errno(errp, -ret, "Unexpected error discarding RAM"); + ram_block_discard_require(false); + return; + } + + virtio_mem_resize_usable_region(vmem, vmem->requested_size, true); + + vmem->bitmap_size = memory_region_size(&vmem->memdev->mr) / + vmem->block_size; + vmem->bitmap = bitmap_new(vmem->bitmap_size); + + virtio_init(vdev, TYPE_VIRTIO_MEM, VIRTIO_ID_MEM, + sizeof(struct virtio_mem_config)); + vmem->vq = virtio_add_queue(vdev, 128, virtio_mem_handle_request); + + host_memory_backend_set_mapped(vmem->memdev, true); + vmstate_register_ram(&vmem->memdev->mr, DEVICE(vmem)); + qemu_register_reset(virtio_mem_system_reset, vmem); +} + +static void virtio_mem_device_unrealize(DeviceState *dev) +{ + VirtIODevice *vdev = VIRTIO_DEVICE(dev); + VirtIOMEM *vmem = VIRTIO_MEM(dev); + + qemu_unregister_reset(virtio_mem_system_reset, vmem); + vmstate_unregister_ram(&vmem->memdev->mr, DEVICE(vmem)); + host_memory_backend_set_mapped(vmem->memdev, false); + virtio_del_queue(vdev, 0); + virtio_cleanup(vdev); + g_free(vmem->bitmap); + ram_block_discard_require(false); +} + +static int virtio_mem_restore_unplugged(VirtIOMEM *vmem) +{ + RAMBlock *rb = vmem->memdev->mr.ram_block; + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, length; + int ret; + + /* Find consecutive unplugged blocks and discard the consecutive range. */ + first_zero_bit = find_first_zero_bit(vmem->bitmap, vmem->bitmap_size); + while (first_zero_bit < vmem->bitmap_size) { + offset = first_zero_bit * vmem->block_size; + last_zero_bit = find_next_bit(vmem->bitmap, vmem->bitmap_size, + first_zero_bit + 1) - 1; + length = (last_zero_bit - first_zero_bit + 1) * vmem->block_size; + + ret = ram_block_discard_range(rb, offset, length); + if (ret) { + error_report("Unexpected error discarding RAM: %s", + strerror(-ret)); + return -EINVAL; + } + first_zero_bit = find_next_zero_bit(vmem->bitmap, vmem->bitmap_size, + last_zero_bit + 2); + } + return 0; +} + +static int virtio_mem_post_load(void *opaque, int version_id) +{ + if (migration_in_incoming_postcopy()) { + return 0; + } + + return virtio_mem_restore_unplugged(VIRTIO_MEM(opaque)); +} + +static const VMStateDescription vmstate_virtio_mem_device = { + .name = "virtio-mem-device", + .minimum_version_id = 1, + .version_id = 1, + .post_load = virtio_mem_post_load, + .fields = (VMStateField[]) { + VMSTATE_UINT64(usable_region_size, VirtIOMEM), + VMSTATE_UINT64(size, VirtIOMEM), + VMSTATE_UINT64(requested_size, VirtIOMEM), + VMSTATE_BITMAP(bitmap, VirtIOMEM, 0, bitmap_size), + VMSTATE_END_OF_LIST() + }, +}; + +static const VMStateDescription vmstate_virtio_mem = { + .name = "virtio-mem", + .minimum_version_id = 1, + .version_id = 1, + .fields = (VMStateField[]) { + VMSTATE_VIRTIO_DEVICE, + VMSTATE_END_OF_LIST() + }, +}; + +static void virtio_mem_fill_device_info(const VirtIOMEM *vmem, + VirtioMEMDeviceInfo *vi) +{ + vi->memaddr = vmem->addr; + vi->node = vmem->node; + vi->requested_size = vmem->requested_size; + vi->size = vmem->size; + vi->max_size = memory_region_size(&vmem->memdev->mr); + vi->block_size = vmem->block_size; + vi->memdev = object_get_canonical_path(OBJECT(vmem->memdev)); +} + +static MemoryRegion *virtio_mem_get_memory_region(VirtIOMEM *vmem, Error **errp) +{ + if (!vmem->memdev) { + error_setg(errp, "'%s' property must be set", VIRTIO_MEM_MEMDEV_PROP); + return NULL; + } + + return &vmem->memdev->mr; +} + +static void virtio_mem_get_size(Object *obj, Visitor *v, const char *name, + void *opaque, Error **errp) +{ + const VirtIOMEM *vmem = VIRTIO_MEM(obj); + uint64_t value = vmem->size; + + visit_type_size(v, name, &value, errp); +} + +static void virtio_mem_get_requested_size(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + const VirtIOMEM *vmem = VIRTIO_MEM(obj); + uint64_t value = vmem->requested_size; + + visit_type_size(v, name, &value, errp); +} + +static void virtio_mem_set_requested_size(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + VirtIOMEM *vmem = VIRTIO_MEM(obj); + Error *err = NULL; + uint64_t value; + + visit_type_size(v, name, &value, &err); + if (err) { + error_propagate(errp, err); + return; + } + + /* + * The block size and memory backend are not fixed until the device was + * realized. realize() will verify these properties then. + */ + if (DEVICE(obj)->realized) { + if (!QEMU_IS_ALIGNED(value, vmem->block_size)) { + error_setg(errp, "'%s' has to be multiples of '%s' (0x%" PRIx64 + ")", name, VIRTIO_MEM_BLOCK_SIZE_PROP, + vmem->block_size); + return; + } else if (value > memory_region_size(&vmem->memdev->mr)) { + error_setg(errp, "'%s' cannot exceed the memory backend size" + "(0x%" PRIx64 ")", name, + memory_region_size(&vmem->memdev->mr)); + return; + } + + if (value != vmem->requested_size) { + virtio_mem_resize_usable_region(vmem, value, false); + vmem->requested_size = value; + } + /* + * Trigger a config update so the guest gets notified. We trigger + * even if the size didn't change (especially helpful for debugging). + */ + virtio_notify_config(VIRTIO_DEVICE(vmem)); + } else { + vmem->requested_size = value; + } +} + +static void virtio_mem_get_block_size(Object *obj, Visitor *v, const char *name, + void *opaque, Error **errp) +{ + const VirtIOMEM *vmem = VIRTIO_MEM(obj); + uint64_t value = vmem->block_size; + + visit_type_size(v, name, &value, errp); +} + +static void virtio_mem_set_block_size(Object *obj, Visitor *v, const char *name, + void *opaque, Error **errp) +{ + VirtIOMEM *vmem = VIRTIO_MEM(obj); + Error *err = NULL; + uint64_t value; + + if (DEVICE(obj)->realized) { + error_setg(errp, "'%s' cannot be changed", name); + return; + } + + visit_type_size(v, name, &value, &err); + if (err) { + error_propagate(errp, err); + return; + } + + if (value < VIRTIO_MEM_MIN_BLOCK_SIZE) { + error_setg(errp, "'%s' property has to be at least 0x%" PRIx32, name, + VIRTIO_MEM_MIN_BLOCK_SIZE); + return; + } else if (!is_power_of_2(value)) { + error_setg(errp, "'%s' property has to be a power of two", name); + return; + } + vmem->block_size = value; +} + +static void virtio_mem_instance_init(Object *obj) +{ + VirtIOMEM *vmem = VIRTIO_MEM(obj); + + vmem->block_size = VIRTIO_MEM_MIN_BLOCK_SIZE; + + object_property_add(obj, VIRTIO_MEM_SIZE_PROP, "size", virtio_mem_get_size, + NULL, NULL, NULL); + object_property_add(obj, VIRTIO_MEM_REQUESTED_SIZE_PROP, "size", + virtio_mem_get_requested_size, + virtio_mem_set_requested_size, NULL, NULL); + object_property_add(obj, VIRTIO_MEM_BLOCK_SIZE_PROP, "size", + virtio_mem_get_block_size, virtio_mem_set_block_size, + NULL, NULL); +} + +static Property virtio_mem_properties[] = { + DEFINE_PROP_UINT64(VIRTIO_MEM_ADDR_PROP, VirtIOMEM, addr, 0), + DEFINE_PROP_UINT32(VIRTIO_MEM_NODE_PROP, VirtIOMEM, node, 0), + DEFINE_PROP_LINK(VIRTIO_MEM_MEMDEV_PROP, VirtIOMEM, memdev, + TYPE_MEMORY_BACKEND, HostMemoryBackend *), + DEFINE_PROP_END_OF_LIST(), +}; + +static void virtio_mem_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass); + VirtIOMEMClass *vmc = VIRTIO_MEM_CLASS(klass); + + device_class_set_props(dc, virtio_mem_properties); + dc->vmsd = &vmstate_virtio_mem; + + set_bit(DEVICE_CATEGORY_MISC, dc->categories); + vdc->realize = virtio_mem_device_realize; + vdc->unrealize = virtio_mem_device_unrealize; + vdc->get_config = virtio_mem_get_config; + vdc->get_features = virtio_mem_get_features; + vdc->vmsd = &vmstate_virtio_mem_device; + + vmc->fill_device_info = virtio_mem_fill_device_info; + vmc->get_memory_region = virtio_mem_get_memory_region; +} + +static const TypeInfo virtio_mem_info = { + .name = TYPE_VIRTIO_MEM, + .parent = TYPE_VIRTIO_DEVICE, + .instance_size = sizeof(VirtIOMEM), + .instance_init = virtio_mem_instance_init, + .class_init = virtio_mem_class_init, + .class_size = sizeof(VirtIOMEMClass), +}; + +static void virtio_register_types(void) +{ + type_register_static(&virtio_mem_info); +} + +type_init(virtio_register_types) diff --git a/include/hw/virtio/virtio-mem.h b/include/hw/virtio/virtio-mem.h new file mode 100644 index 0000000000..6981096f7c --- /dev/null +++ b/include/hw/virtio/virtio-mem.h @@ -0,0 +1,78 @@ +/* + * Virtio MEM device + * + * Copyright (C) 2020 Red Hat, Inc. + * + * Authors: + * David Hildenbrand + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + */ + +#ifndef HW_VIRTIO_MEM_H +#define HW_VIRTIO_MEM_H + +#include "standard-headers/linux/virtio_mem.h" +#include "hw/virtio/virtio.h" +#include "qapi/qapi-types-misc.h" +#include "sysemu/hostmem.h" + +#define TYPE_VIRTIO_MEM "virtio-mem" + +#define VIRTIO_MEM(obj) \ + OBJECT_CHECK(VirtIOMEM, (obj), TYPE_VIRTIO_MEM) +#define VIRTIO_MEM_CLASS(oc) \ + OBJECT_CLASS_CHECK(VirtIOMEMClass, (oc), TYPE_VIRTIO_MEM) +#define VIRTIO_MEM_GET_CLASS(obj) \ + OBJECT_GET_CLASS(VirtIOMEMClass, (obj), TYPE_VIRTIO_MEM) + +#define VIRTIO_MEM_MEMDEV_PROP "memdev" +#define VIRTIO_MEM_NODE_PROP "node" +#define VIRTIO_MEM_SIZE_PROP "size" +#define VIRTIO_MEM_REQUESTED_SIZE_PROP "requested-size" +#define VIRTIO_MEM_BLOCK_SIZE_PROP "block-size" +#define VIRTIO_MEM_ADDR_PROP "memaddr" + +typedef struct VirtIOMEM { + VirtIODevice parent_obj; + + /* guest -> host request queue */ + VirtQueue *vq; + + /* bitmap used to track unplugged memory */ + int32_t bitmap_size; + unsigned long *bitmap; + + /* assigned memory backend and memory region */ + HostMemoryBackend *memdev; + + /* NUMA node */ + uint32_t node; + + /* assigned address of the region in guest physical memory */ + uint64_t addr; + + /* usable region size (<= region_size) */ + uint64_t usable_region_size; + + /* actual size (how much the guest plugged) */ + uint64_t size; + + /* requested size */ + uint64_t requested_size; + + /* block size and alignment */ + uint64_t block_size; +} VirtIOMEM; + +typedef struct VirtIOMEMClass { + /* private */ + VirtIODevice parent; + + /* public */ + void (*fill_device_info)(const VirtIOMEM *vmen, VirtioMEMDeviceInfo *vi); + MemoryRegion *(*get_memory_region)(VirtIOMEM *vmem, Error **errp); +} VirtIOMEMClass; + +#endif diff --git a/qapi/misc.json b/qapi/misc.json index 99b90ac80b..e1c5547b65 100644 --- a/qapi/misc.json +++ b/qapi/misc.json @@ -1354,19 +1354,56 @@ } } +## +# @VirtioMEMDeviceInfo: +# +# VirtioMEMDevice state information +# +# @id: device's ID +# +# @memaddr: physical address in memory, where device is mapped +# +# @requested-size: the user requested size of the device +# +# @size: the (current) size of memory that the device provides +# +# @max-size: the maximum size of memory that the device can provide +# +# @block-size: the block size of memory that the device provides +# +# @node: NUMA node number where device is assigned to +# +# @memdev: memory backend linked with the region +# +# Since: 5.1 +## +{ 'struct': 'VirtioMEMDeviceInfo', + 'data': { '*id': 'str', + 'memaddr': 'size', + 'requested-size': 'size', + 'size': 'size', + 'max-size': 'size', + 'block-size': 'size', + 'node': 'int', + 'memdev': 'str' + } +} + ## # @MemoryDeviceInfo: # # Union containing information about a memory device # # nvdimm is included since 2.12. virtio-pmem is included since 4.1. +# virtio-mem is included since 5.1. # # Since: 2.1 ## { 'union': 'MemoryDeviceInfo', 'data': { 'dimm': 'PCDIMMDeviceInfo', 'nvdimm': 'PCDIMMDeviceInfo', - 'virtio-pmem': 'VirtioPMEMDeviceInfo' + 'virtio-pmem': 'VirtioPMEMDeviceInfo', + 'virtio-mem': 'VirtioMEMDeviceInfo' } } From patchwork Wed Jun 10 11:54:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597619 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB8AC138C for ; Wed, 10 Jun 2020 11:55:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AE8472074B for ; Wed, 10 Jun 2020 11:55:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hGx1xhVD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728809AbgFJLzf (ORCPT ); Wed, 10 Jun 2020 07:55:35 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:44334 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728787AbgFJLzf (ORCPT ); Wed, 10 Jun 2020 07:55:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bKZX/PjV7rqt+noI+UEu1q64HuUlP/xu1O7Q0uMkpG0=; b=hGx1xhVDvVCrXqfQ2RtbO/mI4WU9feyXHqLHh+e3UhM7LDzyc8K2lLr2XgJNogabTKCZZ0 OnGGLzGneprB2mud+tsrAkGhIZuKesCSaSRNbI5klsDECLwxVx8m+No49fXkWeY1MbwgGn +oviY9y1Du9vR7MfTQui32td50fg66A= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-305-lTl_YHIHOw-E3K41iRuwyw-1; Wed, 10 Jun 2020 07:55:31 -0400 X-MC-Unique: lTl_YHIHOw-E3K41iRuwyw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E00DDDC27; Wed, 10 Jun 2020 11:55:29 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 85E8C5D9D3; Wed, 10 Jun 2020 11:55:27 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Pankaj Gupta , Marcel Apfelbaum , Igor Mammedov Subject: [PATCH v4 11/21] virtio-pci: Proxy for virtio-mem Date: Wed, 10 Jun 2020 13:54:09 +0200 Message-Id: <20200610115419.51688-12-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's add a proxy for virtio-mem, make it a memory device, and pass-through the properties. Reviewed-by: Pankaj Gupta Cc: "Michael S. Tsirkin" Cc: Marcel Apfelbaum Cc: "Dr. David Alan Gilbert" Cc: Igor Mammedov Signed-off-by: David Hildenbrand --- hw/virtio/Makefile.objs | 1 + hw/virtio/virtio-mem-pci.c | 129 +++++++++++++++++++++++++++++++++++++ hw/virtio/virtio-mem-pci.h | 33 ++++++++++ include/hw/pci/pci.h | 1 + 4 files changed, 164 insertions(+) create mode 100644 hw/virtio/virtio-mem-pci.c create mode 100644 hw/virtio/virtio-mem-pci.h diff --git a/hw/virtio/Makefile.objs b/hw/virtio/Makefile.objs index 7df70e977e..b9661f9c01 100644 --- a/hw/virtio/Makefile.objs +++ b/hw/virtio/Makefile.objs @@ -19,6 +19,7 @@ obj-$(call land,$(CONFIG_VHOST_USER_FS),$(CONFIG_VIRTIO_PCI)) += vhost-user-fs-p obj-$(CONFIG_VIRTIO_IOMMU) += virtio-iommu.o obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock.o obj-$(CONFIG_VIRTIO_MEM) += virtio-mem.o +common-obj-$(call land,$(CONFIG_VIRTIO_MEM),$(CONFIG_VIRTIO_PCI)) += virtio-mem-pci.o ifeq ($(CONFIG_VIRTIO_PCI),y) obj-$(CONFIG_VHOST_VSOCK) += vhost-vsock-pci.o diff --git a/hw/virtio/virtio-mem-pci.c b/hw/virtio/virtio-mem-pci.c new file mode 100644 index 0000000000..b325303b32 --- /dev/null +++ b/hw/virtio/virtio-mem-pci.c @@ -0,0 +1,129 @@ +/* + * Virtio MEM PCI device + * + * Copyright (C) 2020 Red Hat, Inc. + * + * Authors: + * David Hildenbrand + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "virtio-mem-pci.h" +#include "hw/mem/memory-device.h" +#include "qapi/error.h" + +static void virtio_mem_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) +{ + VirtIOMEMPCI *mem_pci = VIRTIO_MEM_PCI(vpci_dev); + DeviceState *vdev = DEVICE(&mem_pci->vdev); + + qdev_set_parent_bus(vdev, BUS(&vpci_dev->bus)); + object_property_set_bool(OBJECT(vdev), true, "realized", errp); +} + +static void virtio_mem_pci_set_addr(MemoryDeviceState *md, uint64_t addr, + Error **errp) +{ + object_property_set_uint(OBJECT(md), addr, VIRTIO_MEM_ADDR_PROP, errp); +} + +static uint64_t virtio_mem_pci_get_addr(const MemoryDeviceState *md) +{ + return object_property_get_uint(OBJECT(md), VIRTIO_MEM_ADDR_PROP, + &error_abort); +} + +static MemoryRegion *virtio_mem_pci_get_memory_region(MemoryDeviceState *md, + Error **errp) +{ + VirtIOMEMPCI *pci_mem = VIRTIO_MEM_PCI(md); + VirtIOMEM *vmem = VIRTIO_MEM(&pci_mem->vdev); + VirtIOMEMClass *vmc = VIRTIO_MEM_GET_CLASS(vmem); + + return vmc->get_memory_region(vmem, errp); +} + +static uint64_t virtio_mem_pci_get_plugged_size(const MemoryDeviceState *md, + Error **errp) +{ + return object_property_get_uint(OBJECT(md), VIRTIO_MEM_SIZE_PROP, + errp); +} + +static void virtio_mem_pci_fill_device_info(const MemoryDeviceState *md, + MemoryDeviceInfo *info) +{ + VirtioMEMDeviceInfo *vi = g_new0(VirtioMEMDeviceInfo, 1); + VirtIOMEMPCI *pci_mem = VIRTIO_MEM_PCI(md); + VirtIOMEM *vmem = VIRTIO_MEM(&pci_mem->vdev); + VirtIOMEMClass *vpc = VIRTIO_MEM_GET_CLASS(vmem); + DeviceState *dev = DEVICE(md); + + if (dev->id) { + vi->has_id = true; + vi->id = g_strdup(dev->id); + } + + /* let the real device handle everything else */ + vpc->fill_device_info(vmem, vi); + + info->u.virtio_mem.data = vi; + info->type = MEMORY_DEVICE_INFO_KIND_VIRTIO_MEM; +} + +static void virtio_mem_pci_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass); + PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass); + MemoryDeviceClass *mdc = MEMORY_DEVICE_CLASS(klass); + + k->realize = virtio_mem_pci_realize; + set_bit(DEVICE_CATEGORY_MISC, dc->categories); + pcidev_k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET; + pcidev_k->device_id = PCI_DEVICE_ID_VIRTIO_MEM; + pcidev_k->revision = VIRTIO_PCI_ABI_VERSION; + pcidev_k->class_id = PCI_CLASS_OTHERS; + + mdc->get_addr = virtio_mem_pci_get_addr; + mdc->set_addr = virtio_mem_pci_set_addr; + mdc->get_plugged_size = virtio_mem_pci_get_plugged_size; + mdc->get_memory_region = virtio_mem_pci_get_memory_region; + mdc->fill_device_info = virtio_mem_pci_fill_device_info; +} + +static void virtio_mem_pci_instance_init(Object *obj) +{ + VirtIOMEMPCI *dev = VIRTIO_MEM_PCI(obj); + + virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev), + TYPE_VIRTIO_MEM); + object_property_add_alias(obj, VIRTIO_MEM_BLOCK_SIZE_PROP, + OBJECT(&dev->vdev), VIRTIO_MEM_BLOCK_SIZE_PROP); + object_property_add_alias(obj, VIRTIO_MEM_SIZE_PROP, OBJECT(&dev->vdev), + VIRTIO_MEM_SIZE_PROP); + object_property_add_alias(obj, VIRTIO_MEM_REQUESTED_SIZE_PROP, + OBJECT(&dev->vdev), + VIRTIO_MEM_REQUESTED_SIZE_PROP); +} + +static const VirtioPCIDeviceTypeInfo virtio_mem_pci_info = { + .base_name = TYPE_VIRTIO_MEM_PCI, + .generic_name = "virtio-mem-pci", + .instance_size = sizeof(VirtIOMEMPCI), + .instance_init = virtio_mem_pci_instance_init, + .class_init = virtio_mem_pci_class_init, + .interfaces = (InterfaceInfo[]) { + { TYPE_MEMORY_DEVICE }, + { } + }, +}; + +static void virtio_mem_pci_register_types(void) +{ + virtio_pci_types_register(&virtio_mem_pci_info); +} +type_init(virtio_mem_pci_register_types) diff --git a/hw/virtio/virtio-mem-pci.h b/hw/virtio/virtio-mem-pci.h new file mode 100644 index 0000000000..8820cd6628 --- /dev/null +++ b/hw/virtio/virtio-mem-pci.h @@ -0,0 +1,33 @@ +/* + * Virtio MEM PCI device + * + * Copyright (C) 2020 Red Hat, Inc. + * + * Authors: + * David Hildenbrand + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + */ + +#ifndef QEMU_VIRTIO_MEM_PCI_H +#define QEMU_VIRTIO_MEM_PCI_H + +#include "hw/virtio/virtio-pci.h" +#include "hw/virtio/virtio-mem.h" + +typedef struct VirtIOMEMPCI VirtIOMEMPCI; + +/* + * virtio-mem-pci: This extends VirtioPCIProxy. + */ +#define TYPE_VIRTIO_MEM_PCI "virtio-mem-pci-base" +#define VIRTIO_MEM_PCI(obj) \ + OBJECT_CHECK(VirtIOMEMPCI, (obj), TYPE_VIRTIO_MEM_PCI) + +struct VirtIOMEMPCI { + VirtIOPCIProxy parent_obj; + VirtIOMEM vdev; +}; + +#endif /* QEMU_VIRTIO_MEM_PCI_H */ diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h index cfedf5a995..fec72d5a31 100644 --- a/include/hw/pci/pci.h +++ b/include/hw/pci/pci.h @@ -87,6 +87,7 @@ extern bool pci_available; #define PCI_DEVICE_ID_VIRTIO_VSOCK 0x1012 #define PCI_DEVICE_ID_VIRTIO_PMEM 0x1013 #define PCI_DEVICE_ID_VIRTIO_IOMMU 0x1014 +#define PCI_DEVICE_ID_VIRTIO_MEM 0x1015 #define PCI_VENDOR_ID_REDHAT 0x1b36 #define PCI_DEVICE_ID_REDHAT_BRIDGE 0x0001 From patchwork Wed Jun 10 11:54:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597623 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2658414F6 for ; Wed, 10 Jun 2020 11:55:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 044CA20760 for ; Wed, 10 Jun 2020 11:55:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bc7656Kz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728814AbgFJLzh (ORCPT ); Wed, 10 Jun 2020 07:55:37 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:39356 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728787AbgFJLzg (ORCPT ); Wed, 10 Jun 2020 07:55:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790135; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gfo8N/39HuNuGms67Iz3gfpzyYs5Q5599cbmKQTi7kc=; b=bc7656Kz+Cq7SRC7YwxTRETac5cfqCDtILdkWLAmx89/LHa+nkukOo4IoPlP7/dh7gpqcP ynPrM0hYlfqvTgZPwhHEgNFff+xmDReX0bWc0bZvf6/wk/jBnkli9CKe/dWeFtBFpvaqpS dZLZKLV+EDMaSS0m8lJ7T2FGYdWhYw4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-434-PRFe2t6NMSCLqb1PnzQQoQ-1; Wed, 10 Jun 2020 07:55:33 -0400 X-MC-Unique: PRFe2t6NMSCLqb1PnzQQoQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 67AF0100A8EB; Wed, 10 Jun 2020 11:55:32 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3D5E75D9D3; Wed, 10 Jun 2020 11:55:30 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Peter Maydell , Markus Armbruster Subject: [PATCH v4 12/21] MAINTAINERS: Add myself as virtio-mem maintainer Date: Wed, 10 Jun 2020 13:54:10 +0200 Message-Id: <20200610115419.51688-13-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's make sure patches/bug reports find the right person. Reviewed-by: Dr. David Alan Gilbert Cc: "Michael S. Tsirkin" Cc: Peter Maydell Cc: Markus Armbruster Signed-off-by: David Hildenbrand --- MAINTAINERS | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 3abe3faa4e..4889485e6c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1762,6 +1762,14 @@ F: hw/virtio/virtio-crypto.c F: hw/virtio/virtio-crypto-pci.c F: include/hw/virtio/virtio-crypto.h +virtio-mem +M: David Hildenbrand +S: Supported +F: hw/virtio/virtio-mem.c +F: hw/virtio/virtio-mem-pci.h +F: hw/virtio/virtio-mem-pci.c +F: include/hw/virtio/virtio-mem.h + nvme M: Keith Busch L: qemu-block@nongnu.org From patchwork Wed Jun 10 11:54:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597625 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DFCF314B7 for ; Wed, 10 Jun 2020 11:55:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C95382074B for ; Wed, 10 Jun 2020 11:55:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Sjnxop7K" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728844AbgFJLzj (ORCPT ); Wed, 10 Jun 2020 07:55:39 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:34935 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728806AbgFJLzi (ORCPT ); Wed, 10 Jun 2020 07:55:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790137; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0Jehtk1iXtcCdpRGWLvzirxBkna3FHywxxRpKv/a8Ks=; b=Sjnxop7K0ckNwZtec2BTTOVtp+tK0k9erqXItUg3aasRQab57VbLyfyY0Z9mcVz4HNndDw risHwMhEr3oau4wSBFt/lg09SIkS+wHb1mqCDAg62YtkQqAtBKN1GeR7MoHujeZtDUQVdb c0efLHfu1LOaydqSK6cjkS4XbgMgO1I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-266-U1nnZ_RLNeuEhz2d-_9mkw-1; Wed, 10 Jun 2020 07:55:35 -0400 X-MC-Unique: U1nnZ_RLNeuEhz2d-_9mkw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 964E41005513; Wed, 10 Jun 2020 11:55:34 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id B89E65D9D3; Wed, 10 Jun 2020 11:55:32 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 13/21] hmp: Handle virtio-mem when printing memory device info Date: Wed, 10 Jun 2020 13:54:11 +0200 Message-Id: <20200610115419.51688-14-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Print the memory device info just like for other memory devices. Cc: "Dr. David Alan Gilbert" Cc: "Michael S. Tsirkin" Signed-off-by: David Hildenbrand Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Dr. David Alan Gilbert --- monitor/hmp-cmds.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c index 9c61e769ca..afc9a28069 100644 --- a/monitor/hmp-cmds.c +++ b/monitor/hmp-cmds.c @@ -1818,6 +1818,7 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict) MemoryDeviceInfoList *info_list = qmp_query_memory_devices(&err); MemoryDeviceInfoList *info; VirtioPMEMDeviceInfo *vpi; + VirtioMEMDeviceInfo *vmi; MemoryDeviceInfo *value; PCDIMMDeviceInfo *di; @@ -1852,6 +1853,21 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict) monitor_printf(mon, " size: %" PRIu64 "\n", vpi->size); monitor_printf(mon, " memdev: %s\n", vpi->memdev); break; + case MEMORY_DEVICE_INFO_KIND_VIRTIO_MEM: + vmi = value->u.virtio_mem.data; + monitor_printf(mon, "Memory device [%s]: \"%s\"\n", + MemoryDeviceInfoKind_str(value->type), + vmi->id ? vmi->id : ""); + monitor_printf(mon, " memaddr: 0x%" PRIx64 "\n", vmi->memaddr); + monitor_printf(mon, " node: %" PRId64 "\n", vmi->node); + monitor_printf(mon, " requested-size: %" PRIu64 "\n", + vmi->requested_size); + monitor_printf(mon, " size: %" PRIu64 "\n", vmi->size); + monitor_printf(mon, " max-size: %" PRIu64 "\n", vmi->max_size); + monitor_printf(mon, " block-size: %" PRIu64 "\n", + vmi->block_size); + monitor_printf(mon, " memdev: %s\n", vmi->memdev); + break; default: g_assert_not_reached(); } From patchwork Wed Jun 10 11:54:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597627 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 10150138C for ; Wed, 10 Jun 2020 11:55:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA2DD2072E for ; Wed, 10 Jun 2020 11:55:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="a1VZUERH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728832AbgFJLzn (ORCPT ); Wed, 10 Jun 2020 07:55:43 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:48951 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728825AbgFJLzn (ORCPT ); Wed, 10 Jun 2020 07:55:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q646sUohGiaUOuqh4dFa0vnY8G1fxPWotwWGJuhM3Gg=; b=a1VZUERHVVyEPcV5sEU/YHslbHybnOXukxGnhA2RY5LAheInMs7aeJXSxuCWzd0d2GnxSe X0KbgDzQJ/y5kAzjBt+44XBrhWQcdq0PMUF7cyK1KfjopFlsm6x67AbJL2rYtD2d1AwFZe 0MvKZThAFqdAu6+Tm/AI17heS1w1HqI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-327-zqAp5p9cM8qZNzUbrb8Wlg-1; Wed, 10 Jun 2020 07:55:41 -0400 X-MC-Unique: zqAp5p9cM8qZNzUbrb8Wlg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CFFC01B18BC0; Wed, 10 Jun 2020 11:55:39 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id E748A5D9D3; Wed, 10 Jun 2020 11:55:34 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Pankaj Gupta , Marcel Apfelbaum Subject: [PATCH v4 14/21] numa: Handle virtio-mem in NUMA stats Date: Wed, 10 Jun 2020 13:54:12 +0200 Message-Id: <20200610115419.51688-15-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Account the memory to the configured nid. Reviewed-by: Pankaj Gupta Cc: Eduardo Habkost Cc: Marcel Apfelbaum Cc: "Michael S. Tsirkin" Signed-off-by: David Hildenbrand --- hw/core/numa.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hw/core/numa.c b/hw/core/numa.c index 316bc50d75..06960918e7 100644 --- a/hw/core/numa.c +++ b/hw/core/numa.c @@ -812,6 +812,7 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) MemoryDeviceInfoList *info; PCDIMMDeviceInfo *pcdimm_info; VirtioPMEMDeviceInfo *vpi; + VirtioMEMDeviceInfo *vmi; for (info = info_list; info; info = info->next) { MemoryDeviceInfo *value = info->value; @@ -832,6 +833,11 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[0].node_mem += vpi->size; node_mem[0].node_plugged_mem += vpi->size; break; + case MEMORY_DEVICE_INFO_KIND_VIRTIO_MEM: + vmi = value->u.virtio_mem.data; + node_mem[vmi->node].node_mem += vmi->size; + node_mem[vmi->node].node_plugged_mem += vmi->size; + break; default: g_assert_not_reached(); } From patchwork Wed Jun 10 11:54:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597631 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39FEC14B7 for ; Wed, 10 Jun 2020 11:55:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1B1C52074B for ; Wed, 10 Jun 2020 11:55:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="A+ffbR55" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728855AbgFJLzv (ORCPT ); Wed, 10 Jun 2020 07:55:51 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:34605 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728767AbgFJLzu (ORCPT ); Wed, 10 Jun 2020 07:55:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790148; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kV1TOZCW/a8NmT94QFWKHAqddj4smNK0o5RBU/7Yaho=; b=A+ffbR55qu/cW9WLCEa72GNa6SUwZeBxNjaRFTcQemHcqo23XTKBIy6BmrWu0NT2ktJozM +g1hwZ06m6gUPzldf2nk/SFqt2pjdkxmtrrLKnbCivgxI8bbnE9hkJHIIl4UOErYWUaWYl NGZitspVkmtrGPbDLu5hjfLm29DkCW4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-367-UTbbtVl4MaOp9Esn4ry03Q-1; Wed, 10 Jun 2020 07:55:46 -0400 X-MC-Unique: UTbbtVl4MaOp9Esn4ry03Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 71FBA835BBF; Wed, 10 Jun 2020 11:55:45 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2AA085D9D3; Wed, 10 Jun 2020 11:55:40 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Pankaj Gupta , Marcel Apfelbaum , Eric Blake , Markus Armbruster Subject: [PATCH v4 15/21] pc: Support for virtio-mem-pci Date: Wed, 10 Jun 2020 13:54:13 +0200 Message-Id: <20200610115419.51688-16-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's wire it up similar to virtio-pmem. Also disallow unplug, so it's harder for users to shoot themselves into the foot. Reviewed-by: Pankaj Gupta Cc: "Michael S. Tsirkin" Cc: Marcel Apfelbaum Cc: Paolo Bonzini Cc: Richard Henderson Cc: Eduardo Habkost Cc: Eric Blake Cc: Markus Armbruster Signed-off-by: David Hildenbrand --- hw/i386/Kconfig | 1 + hw/i386/pc.c | 49 ++++++++++++++++++++++++++++--------------------- 2 files changed, 29 insertions(+), 21 deletions(-) diff --git a/hw/i386/Kconfig b/hw/i386/Kconfig index c93f32f657..03e347b207 100644 --- a/hw/i386/Kconfig +++ b/hw/i386/Kconfig @@ -35,6 +35,7 @@ config PC select ACPI_PCI select ACPI_VMGENID select VIRTIO_PMEM_SUPPORTED + select VIRTIO_MEM_SUPPORTED config PC_PCI bool diff --git a/hw/i386/pc.c b/hw/i386/pc.c index c740495eb6..ee6368915b 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -86,6 +86,7 @@ #include "hw/net/ne2000-isa.h" #include "standard-headers/asm-x86/bootparam.h" #include "hw/virtio/virtio-pmem-pci.h" +#include "hw/virtio/virtio-mem-pci.h" #include "hw/mem/memory-device.h" #include "sysemu/replay.h" #include "qapi/qmp/qerror.h" @@ -1657,8 +1658,8 @@ static void pc_cpu_pre_plug(HotplugHandler *hotplug_dev, numa_cpu_pre_plug(cpu_slot, dev, errp); } -static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev, - DeviceState *dev, Error **errp) +static void pc_virtio_md_pci_pre_plug(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) { HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev); Error *local_err = NULL; @@ -1669,7 +1670,8 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev, * order. We should never reach this point when hotplugging on x86, * however, better add a safety net. */ - error_setg(errp, "virtio-pmem-pci hotplug not supported on this bus."); + error_setg(errp, "hotplug of virtio based memory devices not supported" + " on this bus."); return; } /* @@ -1684,8 +1686,8 @@ static void pc_virtio_pmem_pci_pre_plug(HotplugHandler *hotplug_dev, error_propagate(errp, local_err); } -static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev, - DeviceState *dev, Error **errp) +static void pc_virtio_md_pci_plug(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) { HotplugHandler *hotplug_dev2 = qdev_get_bus_hotplug_handler(dev); Error *local_err = NULL; @@ -1705,17 +1707,17 @@ static void pc_virtio_pmem_pci_plug(HotplugHandler *hotplug_dev, error_propagate(errp, local_err); } -static void pc_virtio_pmem_pci_unplug_request(HotplugHandler *hotplug_dev, - DeviceState *dev, Error **errp) +static void pc_virtio_md_pci_unplug_request(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) { - /* We don't support virtio pmem hot unplug */ - error_setg(errp, "virtio pmem device unplug not supported."); + /* We don't support hot unplug of virtio based memory devices */ + error_setg(errp, "virtio based memory devices cannot be unplugged."); } -static void pc_virtio_pmem_pci_unplug(HotplugHandler *hotplug_dev, - DeviceState *dev, Error **errp) +static void pc_virtio_md_pci_unplug(HotplugHandler *hotplug_dev, + DeviceState *dev, Error **errp) { - /* We don't support virtio pmem hot unplug */ + /* We don't support hot unplug of virtio based memory devices */ } static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, @@ -1725,8 +1727,9 @@ static void pc_machine_device_pre_plug_cb(HotplugHandler *hotplug_dev, pc_memory_pre_plug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { pc_cpu_pre_plug(hotplug_dev, dev, errp); - } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI)) { - pc_virtio_pmem_pci_pre_plug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) { + pc_virtio_md_pci_pre_plug(hotplug_dev, dev, errp); } } @@ -1737,8 +1740,9 @@ static void pc_machine_device_plug_cb(HotplugHandler *hotplug_dev, pc_memory_plug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { pc_cpu_plug(hotplug_dev, dev, errp); - } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI)) { - pc_virtio_pmem_pci_plug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) { + pc_virtio_md_pci_plug(hotplug_dev, dev, errp); } } @@ -1749,8 +1753,9 @@ static void pc_machine_device_unplug_request_cb(HotplugHandler *hotplug_dev, pc_memory_unplug_request(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { pc_cpu_unplug_request_cb(hotplug_dev, dev, errp); - } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI)) { - pc_virtio_pmem_pci_unplug_request(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) { + pc_virtio_md_pci_unplug_request(hotplug_dev, dev, errp); } else { error_setg(errp, "acpi: device unplug request for not supported device" " type: %s", object_get_typename(OBJECT(dev))); @@ -1764,8 +1769,9 @@ static void pc_machine_device_unplug_cb(HotplugHandler *hotplug_dev, pc_memory_unplug(hotplug_dev, dev, errp); } else if (object_dynamic_cast(OBJECT(dev), TYPE_CPU)) { pc_cpu_unplug_cb(hotplug_dev, dev, errp); - } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI)) { - pc_virtio_pmem_pci_unplug(hotplug_dev, dev, errp); + } else if (object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) { + pc_virtio_md_pci_unplug(hotplug_dev, dev, errp); } else { error_setg(errp, "acpi: device unplug for not supported device" " type: %s", object_get_typename(OBJECT(dev))); @@ -1777,7 +1783,8 @@ static HotplugHandler *pc_get_hotplug_handler(MachineState *machine, { if (object_dynamic_cast(OBJECT(dev), TYPE_PC_DIMM) || object_dynamic_cast(OBJECT(dev), TYPE_CPU) || - object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI)) { + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_PMEM_PCI) || + object_dynamic_cast(OBJECT(dev), TYPE_VIRTIO_MEM_PCI)) { return HOTPLUG_HANDLER(machine); } From patchwork Wed Jun 10 11:54:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597633 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E1D6138C for ; Wed, 10 Jun 2020 11:55:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 05EDC20760 for ; Wed, 10 Jun 2020 11:55:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MUZ48gCl" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728826AbgFJLzw (ORCPT ); Wed, 10 Jun 2020 07:55:52 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:44792 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728784AbgFJLzw (ORCPT ); Wed, 10 Jun 2020 07:55:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790150; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3peSn+MW315QQpa+fsZ3dboRuGSyq5a24dDEiIpLnsw=; b=MUZ48gClZqaq6T5H1Q4UxnSE0Hnvq7+VflmLOVCEAAcWysH0y7dI7KJalFaCkGLkDixepT O8tLvOR1BdMCEImfcbIB7GKzn30s1WIDwBgKkYfmLtQVaKOSW9uRItAFtg3J2ytbeo/qf7 ImgAgb9wZmszVTYFYUXqnwobrMWxSHM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-233-jC5yrNrNMdC27-0yN9xWlQ-1; Wed, 10 Jun 2020 07:55:48 -0400 X-MC-Unique: jC5yrNrNMdC27-0yN9xWlQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CC3616A2BF; Wed, 10 Jun 2020 11:55:47 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id C34005D9D3; Wed, 10 Jun 2020 11:55:45 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Igor Mammedov Subject: [PATCH v4 16/21] virtio-mem: Allow notifiers for size changes Date: Wed, 10 Jun 2020 13:54:14 +0200 Message-Id: <20200610115419.51688-17-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We want to send qapi events in case the size of a virtio-mem device changes. This allows upper layers to always know how much memory is actually currently consumed via a virtio-mem device. Unfortuantely, we have to report the id of our proxy device. Let's provide an easy way for our proxy device to register, so it can send the qapi events. Piggy-backing on the notifier infrastructure (although we'll only ever have one notifier registered) seems to be an easy way. Reviewed-by: Dr. David Alan Gilbert Cc: "Michael S. Tsirkin" Cc: "Dr. David Alan Gilbert" Cc: Igor Mammedov Signed-off-by: David Hildenbrand --- hw/virtio/virtio-mem.c | 21 ++++++++++++++++++++- include/hw/virtio/virtio-mem.h | 5 +++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c index d8a0c974d3..2df33f9125 100644 --- a/hw/virtio/virtio-mem.c +++ b/hw/virtio/virtio-mem.c @@ -184,6 +184,7 @@ static int virtio_mem_state_change_request(VirtIOMEM *vmem, uint64_t gpa, } else { vmem->size -= size; } + notifier_list_notify(&vmem->size_change_notifiers, &vmem->size); return VIRTIO_MEM_RESP_ACK; } @@ -242,7 +243,10 @@ static int virtio_mem_unplug_all(VirtIOMEM *vmem) return -EBUSY; } bitmap_clear(vmem->bitmap, 0, vmem->bitmap_size); - vmem->size = 0; + if (vmem->size) { + vmem->size = 0; + notifier_list_notify(&vmem->size_change_notifiers, &vmem->size); + } virtio_mem_resize_usable_region(vmem, vmem->requested_size, true); return 0; @@ -561,6 +565,18 @@ static MemoryRegion *virtio_mem_get_memory_region(VirtIOMEM *vmem, Error **errp) return &vmem->memdev->mr; } +static void virtio_mem_add_size_change_notifier(VirtIOMEM *vmem, + Notifier *notifier) +{ + notifier_list_add(&vmem->size_change_notifiers, notifier); +} + +static void virtio_mem_remove_size_change_notifier(VirtIOMEM *vmem, + Notifier *notifier) +{ + notifier_remove(notifier); +} + static void virtio_mem_get_size(Object *obj, Visitor *v, const char *name, void *opaque, Error **errp) { @@ -668,6 +684,7 @@ static void virtio_mem_instance_init(Object *obj) VirtIOMEM *vmem = VIRTIO_MEM(obj); vmem->block_size = VIRTIO_MEM_MIN_BLOCK_SIZE; + notifier_list_init(&vmem->size_change_notifiers); object_property_add(obj, VIRTIO_MEM_SIZE_PROP, "size", virtio_mem_get_size, NULL, NULL, NULL); @@ -705,6 +722,8 @@ static void virtio_mem_class_init(ObjectClass *klass, void *data) vmc->fill_device_info = virtio_mem_fill_device_info; vmc->get_memory_region = virtio_mem_get_memory_region; + vmc->add_size_change_notifier = virtio_mem_add_size_change_notifier; + vmc->remove_size_change_notifier = virtio_mem_remove_size_change_notifier; } static const TypeInfo virtio_mem_info = { diff --git a/include/hw/virtio/virtio-mem.h b/include/hw/virtio/virtio-mem.h index 6981096f7c..b74c77cd42 100644 --- a/include/hw/virtio/virtio-mem.h +++ b/include/hw/virtio/virtio-mem.h @@ -64,6 +64,9 @@ typedef struct VirtIOMEM { /* block size and alignment */ uint64_t block_size; + + /* notifiers to notify when "size" changes */ + NotifierList size_change_notifiers; } VirtIOMEM; typedef struct VirtIOMEMClass { @@ -73,6 +76,8 @@ typedef struct VirtIOMEMClass { /* public */ void (*fill_device_info)(const VirtIOMEM *vmen, VirtioMEMDeviceInfo *vi); MemoryRegion *(*get_memory_region)(VirtIOMEM *vmem, Error **errp); + void (*add_size_change_notifier)(VirtIOMEM *vmem, Notifier *notifier); + void (*remove_size_change_notifier)(VirtIOMEM *vmem, Notifier *notifier); } VirtIOMEMClass; #endif From patchwork Wed Jun 10 11:54:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597635 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A52C138C for ; Wed, 10 Jun 2020 11:55:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7049220760 for ; Wed, 10 Jun 2020 11:55:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="IEuyUOae" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728864AbgFJLz5 (ORCPT ); Wed, 10 Jun 2020 07:55:57 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:24022 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728647AbgFJLz4 (ORCPT ); Wed, 10 Jun 2020 07:55:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xQ9OHpqQtlGZ/43g83SVL65+JVka1hvS0ZA72pHlNA8=; b=IEuyUOaeWpbDrTqFFSrfYGD5k/cRnmkDkrO9rv5N2PmjG8AokVpVGTTX1YLiu/W+wcYDZP WPK0K/CpRLHh0ypT8+zq/hK3votHDDYBO967VyvyGTvgaU5p/WpUnNEyz4yOj9rLwoIe+Q 7ntMiRrqKQ4NzRw8wZ+klkh/mwoNffg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-218-CoYoXmsRM9-NFK78b6FMUQ-1; Wed, 10 Jun 2020 07:55:51 -0400 X-MC-Unique: CoYoXmsRM9-NFK78b6FMUQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7DE77872FE2; Wed, 10 Jun 2020 11:55:50 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 284825D9D3; Wed, 10 Jun 2020 11:55:48 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Markus Armbruster , Eric Blake , Igor Mammedov Subject: [PATCH v4 17/21] virtio-pci: Send qapi events when the virtio-mem size changes Date: Wed, 10 Jun 2020 13:54:15 +0200 Message-Id: <20200610115419.51688-18-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's register the notifier and trigger the qapi event with the right device id. MEMORY_DEVICE_SIZE_CHANGE is similar to BALLOON_CHANGE, however on a memory device level. Don't unregister the notifier (we neither have finalize() nor unrealize() for VirtIOPCIProxy, so it's not that simple to do it) - both devices are expected to vanish at the same time. Cc: "Michael S. Tsirkin" Cc: Markus Armbruster Cc: "Dr. David Alan Gilbert" Cc: Eric Blake Cc: Igor Mammedov Signed-off-by: David Hildenbrand --- hw/virtio/virtio-mem-pci.c | 28 ++++++++++++++++++++++++++++ hw/virtio/virtio-mem-pci.h | 1 + monitor/monitor.c | 1 + qapi/misc.json | 25 +++++++++++++++++++++++++ 4 files changed, 55 insertions(+) diff --git a/hw/virtio/virtio-mem-pci.c b/hw/virtio/virtio-mem-pci.c index b325303b32..1a8e854123 100644 --- a/hw/virtio/virtio-mem-pci.c +++ b/hw/virtio/virtio-mem-pci.c @@ -14,6 +14,7 @@ #include "virtio-mem-pci.h" #include "hw/mem/memory-device.h" #include "qapi/error.h" +#include "qapi/qapi-events-misc.h" static void virtio_mem_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { @@ -74,6 +75,21 @@ static void virtio_mem_pci_fill_device_info(const MemoryDeviceState *md, info->type = MEMORY_DEVICE_INFO_KIND_VIRTIO_MEM; } +static void virtio_mem_pci_size_change_notify(Notifier *notifier, void *data) +{ + VirtIOMEMPCI *pci_mem = container_of(notifier, VirtIOMEMPCI, + size_change_notifier); + DeviceState *dev = DEVICE(pci_mem); + const uint64_t * const size_p = data; + const char *id = NULL; + + if (dev->id) { + id = g_strdup(dev->id); + } + + qapi_event_send_memory_device_size_change(!!id, id, *size_p); +} + static void virtio_mem_pci_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); @@ -98,9 +114,21 @@ static void virtio_mem_pci_class_init(ObjectClass *klass, void *data) static void virtio_mem_pci_instance_init(Object *obj) { VirtIOMEMPCI *dev = VIRTIO_MEM_PCI(obj); + VirtIOMEMClass *vmc; + VirtIOMEM *vmem; virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev), TYPE_VIRTIO_MEM); + + dev->size_change_notifier.notify = virtio_mem_pci_size_change_notify; + vmem = VIRTIO_MEM(&dev->vdev); + vmc = VIRTIO_MEM_GET_CLASS(vmem); + /* + * We never remove the notifier again, as we expect both devices to + * disappear at the same time. + */ + vmc->add_size_change_notifier(vmem, &dev->size_change_notifier); + object_property_add_alias(obj, VIRTIO_MEM_BLOCK_SIZE_PROP, OBJECT(&dev->vdev), VIRTIO_MEM_BLOCK_SIZE_PROP); object_property_add_alias(obj, VIRTIO_MEM_SIZE_PROP, OBJECT(&dev->vdev), diff --git a/hw/virtio/virtio-mem-pci.h b/hw/virtio/virtio-mem-pci.h index 8820cd6628..b51a28b275 100644 --- a/hw/virtio/virtio-mem-pci.h +++ b/hw/virtio/virtio-mem-pci.h @@ -28,6 +28,7 @@ typedef struct VirtIOMEMPCI VirtIOMEMPCI; struct VirtIOMEMPCI { VirtIOPCIProxy parent_obj; VirtIOMEM vdev; + Notifier size_change_notifier; }; #endif /* QEMU_VIRTIO_MEM_PCI_H */ diff --git a/monitor/monitor.c b/monitor/monitor.c index 125494410a..19dcb8fbe3 100644 --- a/monitor/monitor.c +++ b/monitor/monitor.c @@ -235,6 +235,7 @@ static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT__MAX] = { [QAPI_EVENT_QUORUM_REPORT_BAD] = { 1000 * SCALE_MS }, [QAPI_EVENT_QUORUM_FAILURE] = { 1000 * SCALE_MS }, [QAPI_EVENT_VSERPORT_CHANGE] = { 1000 * SCALE_MS }, + [QAPI_EVENT_MEMORY_DEVICE_SIZE_CHANGE] = { 1000 * SCALE_MS }, }; /* diff --git a/qapi/misc.json b/qapi/misc.json index e1c5547b65..4b25daeadb 100644 --- a/qapi/misc.json +++ b/qapi/misc.json @@ -1432,6 +1432,31 @@ ## { 'command': 'query-memory-devices', 'returns': ['MemoryDeviceInfo'] } +## +# @MEMORY_DEVICE_SIZE_CHANGE: +# +# Emitted when the size of a memory device changes. Only emitted for memory +# devices that can actually change the size (e.g., virtio-mem due to guest +# action). +# +# @id: device's ID +# @size: the new size of memory that the device provides +# +# Note: this event is rate-limited. +# +# Since: 5.1 +# +# Example: +# +# <- { "event": "MEMORY_DEVICE_SIZE_CHANGE", +# "data": { "id": "vm0", "size": 1073741824}, +# "timestamp": { "seconds": 1588168529, "microseconds": 201316 } } +# +## +{ 'event': 'MEMORY_DEVICE_SIZE_CHANGE', + 'data': { '*id': 'str', 'size': 'size' } } + + ## # @MEM_UNPLUG_ERROR: # From patchwork Wed Jun 10 11:54:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597637 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A536138C for ; Wed, 10 Jun 2020 11:56:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3EBFC20760 for ; Wed, 10 Jun 2020 11:56:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="CmMmimT4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728874AbgFJL4A (ORCPT ); Wed, 10 Jun 2020 07:56:00 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:52158 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728647AbgFJL4A (ORCPT ); Wed, 10 Jun 2020 07:56:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790158; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oubLizbgIuDbm/gkAZ598rX03MG+mq/HEB7KocvYrmI=; b=CmMmimT4A8sbTWHgos3WiY6f/x/DRS71cINuf7LsdHeXNmEkNnIDmWueL+8OQeQaUDb8/N V9ojD0pT2z5VWbD2u5YUM0lJtlsYSJEJ9Pk97xuYXjXQhXVzCPXBU7S3wiK8VG84b8VaXo F8SFc1zkqmnHoupc0B9SbEmcW4Lva5g= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-345-K0izSG0sM0-0A2Xyi07Vzg-1; Wed, 10 Jun 2020 07:55:53 -0400 X-MC-Unique: K0izSG0sM0-0A2Xyi07Vzg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AE52C106B25F; Wed, 10 Jun 2020 11:55:52 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id D0CEF5D9D3; Wed, 10 Jun 2020 11:55:50 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 18/21] virtio-mem: Migration sanity checks Date: Wed, 10 Jun 2020 13:54:16 +0200 Message-Id: <20200610115419.51688-19-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We want to make sure that certain properties don't change during migration, especially to catch user errors in a nice way. Let's migrate a temporary structure and validate that the properties didn't change. Cc: "Michael S. Tsirkin" Cc: "Dr. David Alan Gilbert" Signed-off-by: David Hildenbrand Reviewed-by: Dr. David Alan Gilbert --- hw/virtio/virtio-mem.c | 69 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c index 2df33f9125..450b8dc49d 100644 --- a/hw/virtio/virtio-mem.c +++ b/hw/virtio/virtio-mem.c @@ -519,12 +519,81 @@ static int virtio_mem_post_load(void *opaque, int version_id) return virtio_mem_restore_unplugged(VIRTIO_MEM(opaque)); } +typedef struct VirtIOMEMMigSanityChecks { + VirtIOMEM *parent; + uint64_t addr; + uint64_t region_size; + uint64_t block_size; + uint32_t node; +} VirtIOMEMMigSanityChecks; + +static int virtio_mem_mig_sanity_checks_pre_save(void *opaque) +{ + VirtIOMEMMigSanityChecks *tmp = opaque; + VirtIOMEM *vmem = tmp->parent; + + tmp->addr = vmem->addr; + tmp->region_size = memory_region_size(&vmem->memdev->mr); + tmp->block_size = vmem->block_size; + tmp->node = vmem->node; + return 0; +} + +static int virtio_mem_mig_sanity_checks_post_load(void *opaque, int version_id) +{ + VirtIOMEMMigSanityChecks *tmp = opaque; + VirtIOMEM *vmem = tmp->parent; + const uint64_t new_region_size = memory_region_size(&vmem->memdev->mr); + + if (tmp->addr != vmem->addr) { + error_report("Property '%s' changed from 0x%" PRIx64 " to 0x%" PRIx64, + VIRTIO_MEM_ADDR_PROP, tmp->addr, vmem->addr); + return -EINVAL; + } + /* + * Note: Preparation for resizeable memory regions. The maximum size + * of the memory region must not change during migration. + */ + if (tmp->region_size != new_region_size) { + error_report("region size changed from 0x%" PRIx64 " to 0x%" PRIx64, + tmp->region_size, new_region_size); + return -EINVAL; + } + if (tmp->block_size != vmem->block_size) { + error_report("Property '%s' changed from 0x%" PRIx64 " to 0x%" PRIx64, + VIRTIO_MEM_BLOCK_SIZE_PROP, tmp->block_size, + vmem->block_size); + return -EINVAL; + } + if (tmp->node != vmem->node) { + error_report("Property '%s' changed from %" PRIu32 " to %" PRIu32, + VIRTIO_MEM_NODE_PROP, tmp->node, vmem->node); + return -EINVAL; + } + return 0; +} + +static const VMStateDescription vmstate_virtio_mem_sanity_checks = { + .name = "virtio-mem-device/sanity-checks", + .pre_save = virtio_mem_mig_sanity_checks_pre_save, + .post_load = virtio_mem_mig_sanity_checks_post_load, + .fields = (VMStateField[]) { + VMSTATE_UINT64(addr, VirtIOMEMMigSanityChecks), + VMSTATE_UINT64(region_size, VirtIOMEMMigSanityChecks), + VMSTATE_UINT64(block_size, VirtIOMEMMigSanityChecks), + VMSTATE_UINT32(node, VirtIOMEMMigSanityChecks), + VMSTATE_END_OF_LIST(), + }, +}; + static const VMStateDescription vmstate_virtio_mem_device = { .name = "virtio-mem-device", .minimum_version_id = 1, .version_id = 1, .post_load = virtio_mem_post_load, .fields = (VMStateField[]) { + VMSTATE_WITH_TMP(VirtIOMEM, VirtIOMEMMigSanityChecks, + vmstate_virtio_mem_sanity_checks), VMSTATE_UINT64(usable_region_size, VirtIOMEM), VMSTATE_UINT64(size, VirtIOMEM), VMSTATE_UINT64(requested_size, VirtIOMEM), From patchwork Wed Jun 10 11:54:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597641 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ABD30138C for ; Wed, 10 Jun 2020 11:56:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 939132074B for ; Wed, 10 Jun 2020 11:56:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QmPNMJ4V" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728880AbgFJL4E (ORCPT ); Wed, 10 Jun 2020 07:56:04 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:35456 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728859AbgFJL4E (ORCPT ); Wed, 10 Jun 2020 07:56:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LNpYa81sj5PjQyv9U3qiC+uKU04GSUhnVRclqJQiLmE=; b=QmPNMJ4VmwPgY370O6zCWVUNwseQgIBH8cTbcd2AJ8VH1g+IEwo3RXnM4UBJQhjzBdfpa0 w98WXcsZu6kCBUYuO+ue91o8+/dzcxcqNR0wjuDeihOZWRJkNuOS+KmpdDzNKTeTiZcUgE k8fLV4LGE2xVF+U87iwMvpsfNVfZ5fc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-257-WwlQQFjoPvGbDLkts7bPog-1; Wed, 10 Jun 2020 07:55:58 -0400 X-MC-Unique: WwlQQFjoPvGbDLkts7bPog-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 899CC461; Wed, 10 Jun 2020 11:55:57 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 075FA5D9D3; Wed, 10 Jun 2020 11:55:52 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 19/21] virtio-mem: Add trace events Date: Wed, 10 Jun 2020 13:54:17 +0200 Message-Id: <20200610115419.51688-20-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's add some trace events that might come in handy later. Cc: "Michael S. Tsirkin" Cc: "Dr. David Alan Gilbert" Signed-off-by: David Hildenbrand --- hw/virtio/trace-events | 10 ++++++++++ hw/virtio/virtio-mem.c | 10 +++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index e83500bee9..c40ad5ea27 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -73,3 +73,13 @@ virtio_iommu_get_domain(uint32_t domain_id) "Alloc domain=%d" virtio_iommu_put_domain(uint32_t domain_id) "Free domain=%d" virtio_iommu_translate_out(uint64_t virt_addr, uint64_t phys_addr, uint32_t sid) "0x%"PRIx64" -> 0x%"PRIx64 " for sid=%d" virtio_iommu_report_fault(uint8_t reason, uint32_t flags, uint32_t endpoint, uint64_t addr) "FAULT reason=%d flags=%d endpoint=%d address =0x%"PRIx64 + +# virtio-mem.c +virtio_mem_send_response(uint16_t type) "type=%" PRIu16 +virtio_mem_plug_request(uint64_t addr, uint16_t nb_blocks) "addr=0x%" PRIx64 " nb_blocks=%" PRIu16 +virtio_mem_unplug_request(uint64_t addr, uint16_t nb_blocks) "addr=0x%" PRIx64 " nb_blocks=%" PRIu16 +virtio_mem_unplugged_all(void) "" +virtio_mem_unplug_all_request(void) "" +virtio_mem_resized_usable_region(uint64_t old_size, uint64_t new_size) "old_size=0x%" PRIx64 "new_size=0x%" PRIx64 +virtio_mem_state_request(uint64_t addr, uint16_t nb_blocks) "addr=0x%" PRIx64 " nb_blocks=%" PRIu16 +virtio_mem_state_response(uint16_t state) "state=%" PRIu16 diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c index 450b8dc49d..468fb4170f 100644 --- a/hw/virtio/virtio-mem.c +++ b/hw/virtio/virtio-mem.c @@ -30,6 +30,7 @@ #include "hw/boards.h" #include "hw/qdev-properties.h" #include "config-devices.h" +#include "trace.h" /* * Use QEMU_VMALLOC_ALIGN, so no THP will have to be split when unplugging @@ -100,6 +101,7 @@ static void virtio_mem_send_response(VirtIOMEM *vmem, VirtQueueElement *elem, VirtIODevice *vdev = VIRTIO_DEVICE(vmem); VirtQueue *vq = vmem->vq; + trace_virtio_mem_send_response(le16_to_cpu(resp->type)); iov_from_buf(elem->in_sg, elem->in_num, 0, resp, sizeof(*resp)); virtqueue_push(vq, elem, sizeof(*resp)); @@ -195,6 +197,7 @@ static void virtio_mem_plug_request(VirtIOMEM *vmem, VirtQueueElement *elem, const uint16_t nb_blocks = le16_to_cpu(req->u.plug.nb_blocks); uint16_t type; + trace_virtio_mem_plug_request(gpa, nb_blocks); type = virtio_mem_state_change_request(vmem, gpa, nb_blocks, true); virtio_mem_send_response_simple(vmem, elem, type); } @@ -206,6 +209,7 @@ static void virtio_mem_unplug_request(VirtIOMEM *vmem, VirtQueueElement *elem, const uint16_t nb_blocks = le16_to_cpu(req->u.unplug.nb_blocks); uint16_t type; + trace_virtio_mem_unplug_request(gpa, nb_blocks); type = virtio_mem_state_change_request(vmem, gpa, nb_blocks, false); virtio_mem_send_response_simple(vmem, elem, type); } @@ -225,6 +229,7 @@ static void virtio_mem_resize_usable_region(VirtIOMEM *vmem, return; } + trace_virtio_mem_resized_usable_region(vmem->usable_region_size, newsize); vmem->usable_region_size = newsize; } @@ -247,7 +252,7 @@ static int virtio_mem_unplug_all(VirtIOMEM *vmem) vmem->size = 0; notifier_list_notify(&vmem->size_change_notifiers, &vmem->size); } - + trace_virtio_mem_unplugged_all(); virtio_mem_resize_usable_region(vmem, vmem->requested_size, true); return 0; } @@ -255,6 +260,7 @@ static int virtio_mem_unplug_all(VirtIOMEM *vmem) static void virtio_mem_unplug_all_request(VirtIOMEM *vmem, VirtQueueElement *elem) { + trace_virtio_mem_unplug_all_request(); if (virtio_mem_unplug_all(vmem)) { virtio_mem_send_response_simple(vmem, elem, VIRTIO_MEM_RESP_BUSY); } else { @@ -272,6 +278,7 @@ static void virtio_mem_state_request(VirtIOMEM *vmem, VirtQueueElement *elem, .type = cpu_to_le16(VIRTIO_MEM_RESP_ACK), }; + trace_virtio_mem_state_request(gpa, nb_blocks); if (!virtio_mem_valid_range(vmem, gpa, size)) { virtio_mem_send_response_simple(vmem, elem, VIRTIO_MEM_RESP_ERROR); return; @@ -284,6 +291,7 @@ static void virtio_mem_state_request(VirtIOMEM *vmem, VirtQueueElement *elem, } else { resp.u.state.state = cpu_to_le16(VIRTIO_MEM_STATE_MIXED); } + trace_virtio_mem_state_response(le16_to_cpu(resp.u.state.state)); virtio_mem_send_response(vmem, elem, &resp); } From patchwork Wed Jun 10 11:54:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597639 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 032E7138C for ; Wed, 10 Jun 2020 11:56:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D79C62072E for ; Wed, 10 Jun 2020 11:56:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="K4/OGpYZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728891AbgFJL4F (ORCPT ); Wed, 10 Jun 2020 07:56:05 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:57527 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728870AbgFJL4E (ORCPT ); Wed, 10 Jun 2020 07:56:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wekDDw574QgT8J4I/42gBDbU1WxIpKSkHwqQsu+1Vwk=; b=K4/OGpYZOwc7etieWaUvihoepnkdwb81SYBtdghA+75wCf7NkAk7WKjQt9S2sUduYHdObT Oxd4Rda+5lF5+nPfTXgmx2o5Ahl9pFCd3oWo3bxn7evhf2V3Uu7+uw6+u/bf8flfxiehOv RlSFXIwhSadKHFvR3zrcF4sM8OsQwbw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-460-4QSvKPO-Mq6I8ILWbBNFhA-1; Wed, 10 Jun 2020 07:56:00 -0400 X-MC-Unique: 4QSvKPO-Mq6I8ILWbBNFhA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BA1961B18BC0; Wed, 10 Jun 2020 11:55:59 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE2CB5D9D3; Wed, 10 Jun 2020 11:55:57 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand Subject: [PATCH v4 20/21] virtio-mem: Exclude unplugged memory during migration Date: Wed, 10 Jun 2020 13:54:18 +0200 Message-Id: <20200610115419.51688-21-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The content of unplugged memory is undefined and should not be migrated, ever. Exclude all unplugged memory during precopy using the precopy notifier infrastructure introduced for free page hinting in virtio-balloon. Unplugged memory is marked as "not dirty", meaning it won't be considered for migration. Cc: "Michael S. Tsirkin" Cc: "Dr. David Alan Gilbert" Signed-off-by: David Hildenbrand --- hw/virtio/virtio-mem.c | 54 +++++++++++++++++++++++++++++++++- include/hw/virtio/virtio-mem.h | 3 ++ 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-mem.c b/hw/virtio/virtio-mem.c index 468fb4170f..21a23a3252 100644 --- a/hw/virtio/virtio-mem.c +++ b/hw/virtio/virtio-mem.c @@ -62,8 +62,14 @@ static bool virtio_mem_is_busy(void) /* * Postcopy cannot handle concurrent discards and we don't want to migrate * pages on-demand with stale content when plugging new blocks. + * + * For precopy, we don't want unplugged blocks in our migration stream, and + * when plugging new blocks, the page content might differ between source + * and destination (observable by the guest when not initializing pages + * after plugging them) until we're running on the destination (as we didn't + * migrate these blocks when they were unplugged). */ - return migration_in_incoming_postcopy(); + return migration_in_incoming_postcopy() || !migration_is_idle(); } static bool virtio_mem_test_bitmap(VirtIOMEM *vmem, uint64_t start_gpa, @@ -475,6 +481,7 @@ static void virtio_mem_device_realize(DeviceState *dev, Error **errp) host_memory_backend_set_mapped(vmem->memdev, true); vmstate_register_ram(&vmem->memdev->mr, DEVICE(vmem)); qemu_register_reset(virtio_mem_system_reset, vmem); + precopy_add_notifier(&vmem->precopy_notifier); } static void virtio_mem_device_unrealize(DeviceState *dev) @@ -482,6 +489,7 @@ static void virtio_mem_device_unrealize(DeviceState *dev) VirtIODevice *vdev = VIRTIO_DEVICE(dev); VirtIOMEM *vmem = VIRTIO_MEM(dev); + precopy_remove_notifier(&vmem->precopy_notifier); qemu_unregister_reset(virtio_mem_system_reset, vmem); vmstate_unregister_ram(&vmem->memdev->mr, DEVICE(vmem)); host_memory_backend_set_mapped(vmem->memdev, false); @@ -756,12 +764,56 @@ static void virtio_mem_set_block_size(Object *obj, Visitor *v, const char *name, vmem->block_size = value; } +static void virtio_mem_precopy_exclude_unplugged(VirtIOMEM *vmem) +{ + void * const host = qemu_ram_get_host_addr(vmem->memdev->mr.ram_block); + unsigned long first_zero_bit, last_zero_bit; + uint64_t offset, length; + + /* + * Find consecutive unplugged blocks and exclude them from migration. + * + * Note: Blocks cannot get (un)plugged during precopy, no locking needed. + */ + first_zero_bit = find_first_zero_bit(vmem->bitmap, vmem->bitmap_size); + while (first_zero_bit < vmem->bitmap_size) { + offset = first_zero_bit * vmem->block_size; + last_zero_bit = find_next_bit(vmem->bitmap, vmem->bitmap_size, + first_zero_bit + 1) - 1; + length = (last_zero_bit - first_zero_bit + 1) * vmem->block_size; + + qemu_guest_free_page_hint(host + offset, length); + first_zero_bit = find_next_zero_bit(vmem->bitmap, vmem->bitmap_size, + last_zero_bit + 2); + } +} + +static int virtio_mem_precopy_notify(NotifierWithReturn *n, void *data) +{ + VirtIOMEM *vmem = container_of(n, VirtIOMEM, precopy_notifier); + PrecopyNotifyData *pnd = data; + + switch (pnd->reason) { + case PRECOPY_NOTIFY_SETUP: + precopy_enable_free_page_optimization(); + break; + case PRECOPY_NOTIFY_AFTER_BITMAP_SYNC: + virtio_mem_precopy_exclude_unplugged(vmem); + break; + default: + break; + } + + return 0; +} + static void virtio_mem_instance_init(Object *obj) { VirtIOMEM *vmem = VIRTIO_MEM(obj); vmem->block_size = VIRTIO_MEM_MIN_BLOCK_SIZE; notifier_list_init(&vmem->size_change_notifiers); + vmem->precopy_notifier.notify = virtio_mem_precopy_notify; object_property_add(obj, VIRTIO_MEM_SIZE_PROP, "size", virtio_mem_get_size, NULL, NULL, NULL); diff --git a/include/hw/virtio/virtio-mem.h b/include/hw/virtio/virtio-mem.h index b74c77cd42..0778224964 100644 --- a/include/hw/virtio/virtio-mem.h +++ b/include/hw/virtio/virtio-mem.h @@ -67,6 +67,9 @@ typedef struct VirtIOMEM { /* notifiers to notify when "size" changes */ NotifierList size_change_notifiers; + + /* don't migrate unplugged memory */ + NotifierWithReturn precopy_notifier; } VirtIOMEM; typedef struct VirtIOMEMClass { From patchwork Wed Jun 10 11:54:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 11597647 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B218138C for ; Wed, 10 Jun 2020 11:56:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D93320760 for ; Wed, 10 Jun 2020 11:56:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Vc6+AD0A" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728902AbgFJL4O (ORCPT ); Wed, 10 Jun 2020 07:56:14 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:22393 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728893AbgFJL4O (ORCPT ); Wed, 10 Jun 2020 07:56:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1591790172; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OMm4of9YaGzegN5bSIFM+q+axawQAsBn39Dg4QO3Ssc=; b=Vc6+AD0AMdlrA2DwSOk6vaiZ0+BJ6swksuGsbIOqkd6I740uCsYXTe0H7sDHnrO+YSkvWN no/gsWZqJcHaAQn8aRi545YN4YcQZkCKnYjESCn63h+kR3c3eLW2zdicsA/4ZG67kuGqsG NhGY5FhwoaXtVPLzsZjXPBEbAeie1cU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-237-iZtHDbZuMtaMTb9TjcRE4w-1; Wed, 10 Jun 2020 07:56:10 -0400 X-MC-Unique: iZtHDbZuMtaMTb9TjcRE4w-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 607EF1B18BC2; Wed, 10 Jun 2020 11:56:08 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-42.ams2.redhat.com [10.36.114.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 188245D9D3; Wed, 10 Jun 2020 11:55:59 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Cc: kvm@vger.kernel.org, qemu-s390x@nongnu.org, Richard Henderson , Paolo Bonzini , "Dr . David Alan Gilbert" , Eduardo Habkost , "Michael S . Tsirkin" , David Hildenbrand , Alex Shi , Peter Maydell , Marcel Apfelbaum , Sergio Lopez , Igor Mammedov , "qemu-arm @ nongnu . org" Subject: [PATCH v4 21/21] numa: Auto-enable NUMA when any memory devices are possible Date: Wed, 10 Jun 2020 13:54:19 +0200 Message-Id: <20200610115419.51688-22-david@redhat.com> In-Reply-To: <20200610115419.51688-1-david@redhat.com> References: <20200610115419.51688-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's auto-enable it also when maxmem is specified but no slots are defined. This will result in us properly creating ACPI srat tables, indicating the maximum possible PFN to the guest OS. Based on this, e.g., Linux will enable the swiotlb properly. This avoids having to manually force the switolb on (swiotlb=force) in Linux in case we're booting only using DMA memory (e.g., 2GB on x86-64), and virtio-mem adds memory later on that really needs the swiotlb to be used for DMA. Let's take care of backwards compatibility if somebody has a setup that specifies "maxram" without "slots". Reported-by: Alex Shi Cc: Peter Maydell Cc: Eduardo Habkost Cc: Marcel Apfelbaum Cc: Sergio Lopez Cc: Paolo Bonzini Cc: Richard Henderson Cc: "Michael S. Tsirkin" Cc: Igor Mammedov Cc: qemu-arm@nongnu.org Signed-off-by: David Hildenbrand --- hw/arm/virt.c | 2 ++ hw/core/numa.c | 11 ++++++----- hw/i386/microvm.c | 1 + hw/i386/pc.c | 1 + hw/i386/pc_piix.c | 1 + hw/i386/pc_q35.c | 1 + include/hw/boards.h | 1 + 7 files changed, 13 insertions(+), 5 deletions(-) diff --git a/hw/arm/virt.c b/hw/arm/virt.c index 37462a6f78..d50ad6f841 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -2265,6 +2265,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data) mc->numa_mem_supported = true; mc->nvdimm_supported = true; mc->auto_enable_numa_with_memhp = true; + mc->auto_enable_numa_with_memdev = true; mc->default_ram_id = "mach-virt.ram"; object_class_property_add(oc, "acpi", "OnOffAuto", @@ -2375,6 +2376,7 @@ DEFINE_VIRT_MACHINE_AS_LATEST(5, 1) static void virt_machine_5_0_options(MachineClass *mc) { virt_machine_5_1_options(mc); + mc->auto_enable_numa_with_memdev = false; } DEFINE_VIRT_MACHINE(5, 0) diff --git a/hw/core/numa.c b/hw/core/numa.c index 06960918e7..b9923cdd45 100644 --- a/hw/core/numa.c +++ b/hw/core/numa.c @@ -681,8 +681,9 @@ void numa_complete_configuration(MachineState *ms) NodeInfo *numa_info = ms->numa_state->nodes; /* - * If memory hotplug is enabled (slots > 0) but without '-numa' - * options explicitly on CLI, guestes will break. + * If memory hotplug is enabled (slot > 0) or memory devices are enabled + * (ms->maxram_size > ram_size) but without '-numa' options explicitly on + * CLI, guests will break. * * Windows: won't enable memory hotplug without SRAT table at all * @@ -697,9 +698,9 @@ void numa_complete_configuration(MachineState *ms) * assume there is just one node with whole RAM. */ if (ms->numa_state->num_nodes == 0 && - ((ms->ram_slots > 0 && - mc->auto_enable_numa_with_memhp) || - mc->auto_enable_numa)) { + ((ms->ram_slots && mc->auto_enable_numa_with_memhp) || + (ms->maxram_size > ms->ram_size && mc->auto_enable_numa_with_memdev) || + mc->auto_enable_numa)) { NumaNodeOptions node = { }; parse_numa_node(ms, &node, &error_abort); numa_info[0].node_mem = ram_size; diff --git a/hw/i386/microvm.c b/hw/i386/microvm.c index 937db10ae6..2c052ee2e5 100644 --- a/hw/i386/microvm.c +++ b/hw/i386/microvm.c @@ -497,6 +497,7 @@ static void microvm_class_init(ObjectClass *oc, void *data) mc->max_cpus = 288; mc->has_hotpluggable_cpus = false; mc->auto_enable_numa_with_memhp = false; + mc->auto_enable_numa_with_memdev = false; mc->default_cpu_type = TARGET_DEFAULT_CPU_TYPE; mc->nvdimm_supported = false; mc->default_ram_id = "microvm.ram"; diff --git a/hw/i386/pc.c b/hw/i386/pc.c index ee6368915b..ec93870de5 100644 --- a/hw/i386/pc.c +++ b/hw/i386/pc.c @@ -1955,6 +1955,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data) mc->get_default_cpu_node_id = x86_get_default_cpu_node_id; mc->possible_cpu_arch_ids = x86_possible_cpu_arch_ids; mc->auto_enable_numa_with_memhp = true; + mc->auto_enable_numa_with_memdev = true; mc->has_hotpluggable_cpus = true; mc->default_boot_order = "cad"; mc->hot_add_cpu = pc_hot_add_cpu; diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c index f66e1d73ce..1bc68c3841 100644 --- a/hw/i386/pc_piix.c +++ b/hw/i386/pc_piix.c @@ -440,6 +440,7 @@ static void pc_i440fx_5_0_machine_options(MachineClass *m) m->is_default = false; compat_props_add(m->compat_props, hw_compat_5_0, hw_compat_5_0_len); compat_props_add(m->compat_props, pc_compat_5_0, pc_compat_5_0_len); + m->auto_enable_numa_with_memdev = false; } DEFINE_I440FX_MACHINE(v5_0, "pc-i440fx-5.0", NULL, diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c index 4ba8ac8774..a17459e37c 100644 --- a/hw/i386/pc_q35.c +++ b/hw/i386/pc_q35.c @@ -368,6 +368,7 @@ static void pc_q35_5_0_machine_options(MachineClass *m) m->alias = NULL; compat_props_add(m->compat_props, hw_compat_5_0, hw_compat_5_0_len); compat_props_add(m->compat_props, pc_compat_5_0, pc_compat_5_0_len); + m->auto_enable_numa_with_memhp = false; } DEFINE_Q35_MACHINE(v5_0, "pc-q35-5.0", NULL, diff --git a/include/hw/boards.h b/include/hw/boards.h index 18815d9be2..426ce5f625 100644 --- a/include/hw/boards.h +++ b/include/hw/boards.h @@ -207,6 +207,7 @@ struct MachineClass { const char **valid_cpu_types; strList *allowed_dynamic_sysbus_devices; bool auto_enable_numa_with_memhp; + bool auto_enable_numa_with_memdev; void (*numa_auto_assign_ram)(MachineClass *mc, NodeInfo *nodes, int nb_nodes, ram_addr_t size); bool ignore_boot_device_suffixes;