From patchwork Fri Jun 28 14:57:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Albert Esteve X-Patchwork-Id: 13716298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BCD9CC3064D for ; Fri, 28 Jun 2024 14:58:20 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sND2g-0007Bt-Qi; Fri, 28 Jun 2024 10:57:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2d-0007B3-VW for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:36 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2X-00038a-VO for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719586648; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uRhGKGbga7VigEUjruzxgi/GcgDz5SkHhCHAYDJnK7o=; b=MVZw3sBDUOO9Dy8Ns3DUbhzpDrj7PPPi2xB2RVt1hyB6soHKDO7ELH2bxEsuTpKJDTuJSH eB1SinpfR1p3NcTCnlokjZ64v0a0i9fvEofUV8F4yOB1GYzTjTtRWC+6xIEIeog+zIgAdE UdE1ocEoZaTQkgLYrtgsZ50dtXTgQ9Q= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-379-VadvQHbXPKSvEWFqGgc7GA-1; Fri, 28 Jun 2024 10:57:24 -0400 X-MC-Unique: VadvQHbXPKSvEWFqGgc7GA-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DDAA91956080; Fri, 28 Jun 2024 14:57:23 +0000 (UTC) Received: from localhost.localdomain (unknown [10.45.224.245]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 165A919560A3; Fri, 28 Jun 2024 14:57:20 +0000 (UTC) From: Albert Esteve To: qemu-devel@nongnu.org Cc: jasowang@redhat.com, david@redhat.com, slp@redhat.com, =?utf-8?q?Alex_Be?= =?utf-8?q?nn=C3=A9e?= , stefanha@redhat.com, "Michael S. Tsirkin" , Albert Esteve Subject: [RFC PATCH v2 1/5] vhost-user: Add VIRTIO Shared Memory map request Date: Fri, 28 Jun 2024 16:57:06 +0200 Message-ID: <20240628145710.1516121-2-aesteve@redhat.com> In-Reply-To: <20240628145710.1516121-1-aesteve@redhat.com> References: <20240628145710.1516121-1-aesteve@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass client-ip=170.10.129.124; envelope-from=aesteve@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.206, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add SHMEM_MAP/UNMAP requests to vhost-user to handle VIRTIO Shared Memory mappings. This request allows backends to dynamically map fds into a VIRTIO Shared Memory Region indentified by its `shmid`. Then, the fd memory is advertised to the driver as a base addres + offset, so it can be read/written (depending on the mmap flags requested) while its valid. The backend can munmap the memory range in a given VIRTIO Shared Memory Region (again, identified by its `shmid`), to free it. Upon receiving this message, the front-end must mmap the regions with PROT_NONE to reserve the virtual memory space. The device model needs to create MemoryRegion instances for the VIRTIO Shared Memory Regions and add them to the `VirtIODevice` instance. Signed-off-by: Albert Esteve --- docs/interop/vhost-user.rst | 27 +++++ hw/virtio/vhost-user.c | 122 ++++++++++++++++++++++ hw/virtio/virtio.c | 12 +++ include/hw/virtio/virtio.h | 5 + subprojects/libvhost-user/libvhost-user.c | 65 ++++++++++++ subprojects/libvhost-user/libvhost-user.h | 53 ++++++++++ 6 files changed, 284 insertions(+) diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst index d8419fd2f1..d52ba719d5 100644 --- a/docs/interop/vhost-user.rst +++ b/docs/interop/vhost-user.rst @@ -1859,6 +1859,33 @@ is sent by the front-end. when the operation is successful, or non-zero otherwise. Note that if the operation fails, no fd is sent to the backend. +``VHOST_USER_BACKEND_SHMEM_MAP`` + :id: 9 + :equivalent ioctl: N/A + :request payload: fd and ``struct VhostUserMMap`` + :reply payload: N/A + + This message can be submitted by the backends to advertise a new mapping + to be made in a given VIRTIO Shared Memory Region. Upon receiving the message, + The front-end will mmap the given fd into the VIRTIO Shared Memory Region + with the requested ``shmid``. A reply is generated indicating whether mapping + succeeded. + + Mapping over an already existing map is not allowed and request shall fail. + Therefore, the memory range in the request must correspond with a valid, + free region of the VIRTIO Shared Memory Region. + +``VHOST_USER_BACKEND_SHMEM_UNMAP`` + :id: 10 + :equivalent ioctl: N/A + :request payload: ``struct VhostUserMMap`` + :reply payload: N/A + + This message can be submitted by the backends so that the front-end un-mmap + a given range (``offset``, ``len``) in the VIRTIO Shared Memory Region with + the requested ``shmid``. + A reply is generated indicating whether unmapping succeeded. + .. _reply_ack: VHOST_USER_PROTOCOL_F_REPLY_ACK diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index cdf9af4a4b..7ee8a472c6 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -115,6 +115,8 @@ typedef enum VhostUserBackendRequest { VHOST_USER_BACKEND_SHARED_OBJECT_ADD = 6, VHOST_USER_BACKEND_SHARED_OBJECT_REMOVE = 7, VHOST_USER_BACKEND_SHARED_OBJECT_LOOKUP = 8, + VHOST_USER_BACKEND_SHMEM_MAP = 9, + VHOST_USER_BACKEND_SHMEM_UNMAP = 10, VHOST_USER_BACKEND_MAX } VhostUserBackendRequest; @@ -192,6 +194,24 @@ typedef struct VhostUserShared { unsigned char uuid[16]; } VhostUserShared; +/* For the flags field of VhostUserMMap */ +#define VHOST_USER_FLAG_MAP_R (1u << 0) +#define VHOST_USER_FLAG_MAP_W (1u << 1) + +typedef struct { + /* VIRTIO Shared Memory Region ID */ + uint8_t shmid; + uint8_t padding[7]; + /* File offset */ + uint64_t fd_offset; + /* Offset within the VIRTIO Shared Memory Region */ + uint64_t shm_offset; + /* Size of the mapping */ + uint64_t len; + /* Flags for the mmap operation, from VHOST_USER_FLAG_* */ + uint64_t flags; +} VhostUserMMap; + typedef struct { VhostUserRequest request; @@ -224,6 +244,7 @@ typedef union { VhostUserInflight inflight; VhostUserShared object; VhostUserTransferDeviceState transfer_state; + VhostUserMMap mmap; } VhostUserPayload; typedef struct VhostUserMsg { @@ -1748,6 +1769,100 @@ vhost_user_backend_handle_shared_object_lookup(struct vhost_user *u, return 0; } +static int +vhost_user_backend_handle_shmem_map(struct vhost_dev *dev, + VhostUserMMap *vu_mmap, + int fd) +{ + void *addr = 0; + MemoryRegion *mr = NULL; + + if (fd < 0) { + error_report("Bad fd for map"); + return -EBADF; + } + + if (!dev->vdev->shmem_list || + dev->vdev->n_shmem_regions <= vu_mmap->shmid) { + error_report("Device only has %d VIRTIO Shared Memory Regions. " + "Requested ID: %d", + dev->vdev->n_shmem_regions, vu_mmap->shmid); + return -EFAULT; + } + + mr = &dev->vdev->shmem_list[vu_mmap->shmid]; + + if (!mr) { + error_report("VIRTIO Shared Memory Region at " + "ID %d unitialized", vu_mmap->shmid); + return -EFAULT; + } + + if ((vu_mmap->shm_offset + vu_mmap->len) < vu_mmap->len || + (vu_mmap->shm_offset + vu_mmap->len) > mr->size) { + error_report("Bad offset/len for mmap %" PRIx64 "+%" PRIx64, + vu_mmap->shm_offset, vu_mmap->len); + return -EFAULT; + } + + void *shmem_ptr = memory_region_get_ram_ptr(mr); + + addr = mmap(shmem_ptr + vu_mmap->shm_offset, vu_mmap->len, + ((vu_mmap->flags & VHOST_USER_FLAG_MAP_R) ? PROT_READ : 0) | + ((vu_mmap->flags & VHOST_USER_FLAG_MAP_W) ? PROT_WRITE : 0), + MAP_SHARED | MAP_FIXED, fd, vu_mmap->fd_offset); + + if (addr == MAP_FAILED) { + error_report("Failed to mmap mem fd"); + return -EFAULT; + } + + return 0; +} + +static int +vhost_user_backend_handle_shmem_unmap(struct vhost_dev *dev, + VhostUserMMap *vu_mmap) +{ + void *addr = 0; + MemoryRegion *mr = NULL; + + if (!dev->vdev->shmem_list || + dev->vdev->n_shmem_regions <= vu_mmap->shmid) { + error_report("Device only has %d VIRTIO Shared Memory Regions. " + "Requested ID: %d", + dev->vdev->n_shmem_regions, vu_mmap->shmid); + return -EFAULT; + } + + mr = &dev->vdev->shmem_list[vu_mmap->shmid]; + + if (!mr) { + error_report("VIRTIO Shared Memory Region at " + "ID %d unitialized", vu_mmap->shmid); + return -EFAULT; + } + + if ((vu_mmap->shm_offset + vu_mmap->len) < vu_mmap->len || + (vu_mmap->shm_offset + vu_mmap->len) > mr->size) { + error_report("Bad offset/len for mmap %" PRIx64 "+%" PRIx64, + vu_mmap->shm_offset, vu_mmap->len); + return -EFAULT; + } + + void *shmem_ptr = memory_region_get_ram_ptr(mr); + + addr = mmap(shmem_ptr + vu_mmap->shm_offset, vu_mmap->len, + PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0); + + if (addr == MAP_FAILED) { + error_report("Failed to unmap memory"); + return -EFAULT; + } + + return 0; +} + static void close_backend_channel(struct vhost_user *u) { g_source_destroy(u->backend_src); @@ -1816,6 +1931,13 @@ static gboolean backend_read(QIOChannel *ioc, GIOCondition condition, ret = vhost_user_backend_handle_shared_object_lookup(dev->opaque, ioc, &hdr, &payload); break; + case VHOST_USER_BACKEND_SHMEM_MAP: + ret = vhost_user_backend_handle_shmem_map(dev, &payload.mmap, + fd ? fd[0] : -1); + break; + case VHOST_USER_BACKEND_SHMEM_UNMAP: + ret = vhost_user_backend_handle_shmem_unmap(dev, &payload.mmap); + break; default: error_report("Received unexpected msg type: %d.", hdr.request); ret = -EINVAL; diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c index 893a072c9d..9f2da5b11e 100644 --- a/hw/virtio/virtio.c +++ b/hw/virtio/virtio.c @@ -2856,6 +2856,16 @@ int virtio_save(VirtIODevice *vdev, QEMUFile *f) return vmstate_save_state(f, &vmstate_virtio, vdev, NULL); } +MemoryRegion *virtio_new_shmem_region(VirtIODevice *vdev) +{ + MemoryRegion *mr = g_new0(MemoryRegion, 1); + ++vdev->n_shmem_regions; + vdev->shmem_list = g_renew(MemoryRegion, vdev->shmem_list, + vdev->n_shmem_regions); + vdev->shmem_list[vdev->n_shmem_regions - 1] = *mr; + return mr; +} + /* A wrapper for use as a VMState .put function */ static int virtio_device_put(QEMUFile *f, void *opaque, size_t size, const VMStateField *field, JSONWriter *vmdesc) @@ -3264,6 +3274,8 @@ void virtio_init(VirtIODevice *vdev, uint16_t device_id, size_t config_size) virtio_vmstate_change, vdev); vdev->device_endian = virtio_default_endian(); vdev->use_guest_notifier_mask = true; + vdev->shmem_list = NULL; + vdev->n_shmem_regions = 0; } /* diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index 7d5ffdc145..16d598aadc 100644 --- a/include/hw/virtio/virtio.h +++ b/include/hw/virtio/virtio.h @@ -165,6 +165,9 @@ struct VirtIODevice */ EventNotifier config_notifier; bool device_iotlb_enabled; + /* Shared memory region for vhost-user mappings. */ + MemoryRegion *shmem_list; + int n_shmem_regions; }; struct VirtioDeviceClass { @@ -280,6 +283,8 @@ void virtio_notify(VirtIODevice *vdev, VirtQueue *vq); int virtio_save(VirtIODevice *vdev, QEMUFile *f); +MemoryRegion *virtio_new_shmem_region(VirtIODevice *vdev); + extern const VMStateInfo virtio_vmstate_info; #define VMSTATE_VIRTIO_DEVICE \ diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c index a879149fef..28556d183a 100644 --- a/subprojects/libvhost-user/libvhost-user.c +++ b/subprojects/libvhost-user/libvhost-user.c @@ -1586,6 +1586,71 @@ vu_rm_shared_object(VuDev *dev, unsigned char uuid[UUID_LEN]) return vu_send_message(dev, &msg); } +bool +vu_shmem_map(VuDev *dev, uint8_t shmid, uint64_t fd_offset, + uint64_t shm_offset, uint64_t len, uint64_t flags) +{ + bool result = false; + VhostUserMsg msg_reply; + VhostUserMsg vmsg = { + .request = VHOST_USER_BACKEND_SHMEM_MAP, + .size = sizeof(vmsg.payload.mmap), + .flags = VHOST_USER_VERSION, + .payload.mmap = { + .shmid = shmid, + .fd_offset = fd_offset, + .shm_offset = shm_offset, + .len = len, + .flags = flags, + }, + }; + + if (vu_has_protocol_feature(dev, VHOST_USER_PROTOCOL_F_REPLY_ACK)) { + vmsg.flags |= VHOST_USER_NEED_REPLY_MASK; + } + + pthread_mutex_lock(&dev->backend_mutex); + if (!vu_message_write(dev, dev->backend_fd, &vmsg)) { + pthread_mutex_unlock(&dev->backend_mutex); + return false; + } + + /* Also unlocks the backend_mutex */ + return vu_process_message_reply(dev, &vmsg); +} + +bool +vu_shmem_unmap(VuDev *dev, uint8_t shmid, uint64_t fd_offset, + uint64_t shm_offset, uint64_t len) +{ + bool result = false; + VhostUserMsg msg_reply; + VhostUserMsg vmsg = { + .request = VHOST_USER_BACKEND_SHMEM_UNMAP, + .size = sizeof(vmsg.payload.mmap), + .flags = VHOST_USER_VERSION, + .payload.mmap = { + .shmid = shmid, + .fd_offset = fd_offset, + .shm_offset = shm_offset, + .len = len, + }, + }; + + if (vu_has_protocol_feature(dev, VHOST_USER_PROTOCOL_F_REPLY_ACK)) { + vmsg.flags |= VHOST_USER_NEED_REPLY_MASK; + } + + pthread_mutex_lock(&dev->backend_mutex); + if (!vu_message_write(dev, dev->backend_fd, &vmsg)) { + pthread_mutex_unlock(&dev->backend_mutex); + return false; + } + + /* Also unlocks the backend_mutex */ + return vu_process_message_reply(dev, &vmsg); +} + static bool vu_set_vring_call_exec(VuDev *dev, VhostUserMsg *vmsg) { diff --git a/subprojects/libvhost-user/libvhost-user.h b/subprojects/libvhost-user/libvhost-user.h index deb40e77b3..7f6c22cc1a 100644 --- a/subprojects/libvhost-user/libvhost-user.h +++ b/subprojects/libvhost-user/libvhost-user.h @@ -127,6 +127,8 @@ typedef enum VhostUserBackendRequest { VHOST_USER_BACKEND_SHARED_OBJECT_ADD = 6, VHOST_USER_BACKEND_SHARED_OBJECT_REMOVE = 7, VHOST_USER_BACKEND_SHARED_OBJECT_LOOKUP = 8, + VHOST_USER_BACKEND_SHMEM_MAP = 9, + VHOST_USER_BACKEND_SHMEM_UNMAP = 10, VHOST_USER_BACKEND_MAX } VhostUserBackendRequest; @@ -186,6 +188,24 @@ typedef struct VhostUserShared { unsigned char uuid[UUID_LEN]; } VhostUserShared; +/* For the flags field of VhostUserMMap */ +#define VHOST_USER_FLAG_MAP_R (1u << 0) +#define VHOST_USER_FLAG_MAP_W (1u << 1) + +typedef struct { + /* VIRTIO Shared Memory Region ID */ + uint8_t shmid; + uint8_t padding[7]; + /* File offset */ + uint64_t fd_offset; + /* Offset within the VIRTIO Shared Memory Region */ + uint64_t shm_offset; + /* Size of the mapping */ + uint64_t len; + /* Flags for the mmap operation, from VHOST_USER_FLAG_* */ + uint64_t flags; +} VhostUserMMap; + #if defined(_WIN32) && (defined(__x86_64__) || defined(__i386__)) # define VU_PACKED __attribute__((gcc_struct, packed)) #else @@ -214,6 +234,7 @@ typedef struct VhostUserMsg { VhostUserVringArea area; VhostUserInflight inflight; VhostUserShared object; + VhostUserMMap mmap; } payload; int fds[VHOST_MEMORY_BASELINE_NREGIONS]; @@ -597,6 +618,38 @@ bool vu_add_shared_object(VuDev *dev, unsigned char uuid[UUID_LEN]); */ bool vu_rm_shared_object(VuDev *dev, unsigned char uuid[UUID_LEN]); +/** + * vu_shmem_map: + * @dev: a VuDev context + * @shmid: VIRTIO Shared Memory Region ID + * @fd_offset: File offset + * @shm_offset: Offset within the VIRTIO Shared Memory Region + * @len: Size of the mapping + * @flags: Flags for the mmap operation + * + * Advertises a new mapping to be made in a given VIRTIO Shared Memory Region. + * + * Returns: TRUE on success, FALSE on failure. + */ +bool vu_shmem_map(VuDev *dev, uint8_t shmid, uint64_t fd_offset, + uint64_t shm_offset, uint64_t len, uint64_t flags); + +/** + * vu_shmem_map: + * @dev: a VuDev context + * @shmid: VIRTIO Shared Memory Region ID + * @fd_offset: File offset + * @shm_offset: Offset within the VIRTIO Shared Memory Region + * @len: Size of the mapping + * + * The front-end un-mmaps a given range in the VIRTIO Shared Memory Region + * with the requested `shmid`. + * + * Returns: TRUE on success, FALSE on failure. + */ +bool vu_shmem_unmap(VuDev *dev, uint8_t shmid, uint64_t fd_offset, + uint64_t shm_offset, uint64_t len); + /** * vu_queue_set_notification: * @dev: a VuDev context From patchwork Fri Jun 28 14:57:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Albert Esteve X-Patchwork-Id: 13716302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F4EFC3064D for ; Fri, 28 Jun 2024 14:58:55 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sND2x-0007ER-Pg; Fri, 28 Jun 2024 10:57:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2k-0007CS-0H for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2c-00038k-Bw for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719586651; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QS94ZL/p/YN0VZ/axbd9A4oNExdDycoMo/iGTctAHkg=; b=VhcBdhcbrlrgKXjUJgBSjZVIKtKlB7UboZuuV6QEfF+awCjN0ikaVGOdkz5gwNw6DGPLnn EvIOhKWNTEZfTbnC/twHA06m6UKBCrygToDcxQeO228EkNlX6t9lai5QMGNqch5vNmgi5k VJjAwqhan90XvCH2Dnov/xRHWuMIqrs= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-3-06Qz26buMYqFqgASr6-08w-1; Fri, 28 Jun 2024 10:57:28 -0400 X-MC-Unique: 06Qz26buMYqFqgASr6-08w-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 28E7C1956058; Fri, 28 Jun 2024 14:57:27 +0000 (UTC) Received: from localhost.localdomain (unknown [10.45.224.245]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4FED01944DC5; Fri, 28 Jun 2024 14:57:24 +0000 (UTC) From: Albert Esteve To: qemu-devel@nongnu.org Cc: jasowang@redhat.com, david@redhat.com, slp@redhat.com, =?utf-8?q?Alex_Be?= =?utf-8?q?nn=C3=A9e?= , stefanha@redhat.com, "Michael S. Tsirkin" , Albert Esteve Subject: [RFC PATCH v2 2/5] vhost_user: Add frontend command for shmem config Date: Fri, 28 Jun 2024 16:57:07 +0200 Message-ID: <20240628145710.1516121-3-aesteve@redhat.com> In-Reply-To: <20240628145710.1516121-1-aesteve@redhat.com> References: <20240628145710.1516121-1-aesteve@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass client-ip=170.10.129.124; envelope-from=aesteve@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.206, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The frontend can use this command to retrieve VIRTIO Shared Memory Regions configuration from the backend. The response contains the number of shared memory regions, their size, and shmid. This is useful when the frontend is unaware of specific backend type and configuration, for example, in the `vhost-user-device` case. Signed-off-by: Albert Esteve --- docs/interop/vhost-user.rst | 31 +++++++++++++++++++++++ hw/virtio/vhost-user.c | 42 +++++++++++++++++++++++++++++++ include/hw/virtio/vhost-backend.h | 6 +++++ include/hw/virtio/vhost-user.h | 1 + 4 files changed, 80 insertions(+) diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst index d52ba719d5..51f01d1d84 100644 --- a/docs/interop/vhost-user.rst +++ b/docs/interop/vhost-user.rst @@ -348,6 +348,19 @@ Device state transfer parameters In the future, additional phases might be added e.g. to allow iterative migration while the device is running. +VIRTIO Shared Memory Region configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++-------------+---------+------------+----+------------+ +| num regions | padding | mem size 0 | .. | mem size 7 | ++-------------+---------+------------+----+------------+ + +:num regions: a 32-bit number of regions + +:padding: 32-bit + +:mem size: 64-bit size of VIRTIO Shared Memory Region + C structure ----------- @@ -369,6 +382,10 @@ In QEMU the vhost-user message is implemented with the following struct: VhostUserConfig config; VhostUserVringArea area; VhostUserInflight inflight; + VhostUserShared object; + VhostUserTransferDeviceState transfer_state; + VhostUserMMap mmap; + VhostUserShMemConfig shmem; }; } QEMU_PACKED VhostUserMsg; @@ -1051,6 +1068,7 @@ Protocol features #define VHOST_USER_PROTOCOL_F_XEN_MMAP 17 #define VHOST_USER_PROTOCOL_F_SHARED_OBJECT 18 #define VHOST_USER_PROTOCOL_F_DEVICE_STATE 19 + #define VHOST_USER_PROTOCOL_F_SHMEM 20 Front-end message types ----------------------- @@ -1725,6 +1743,19 @@ Front-end message types Using this function requires prior negotiation of the ``VHOST_USER_PROTOCOL_F_DEVICE_STATE`` feature. +``VHOST_USER_GET_SHMEM_CONFIG`` + :id: 44 + :equivalent ioctl: N/A + :request payload: N/A + :reply payload: ``struct VhostUserShMemConfig`` + + When the ``VHOST_USER_PROTOCOL_F_SHMEM`` protocol feature has been + successfully negotiated, this message can be submitted by the front-end + to gather the VIRTIO Shared Memory Region configuration. Back-end will respond + with the number of VIRTIO Shared Memory Regions it requires, and each shared memory + region size in an array. The shared memory IDs are represented by the index + of the array. + Back-end message types ---------------------- diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 7ee8a472c6..57406dc8b4 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -104,6 +104,7 @@ typedef enum VhostUserRequest { VHOST_USER_GET_SHARED_OBJECT = 41, VHOST_USER_SET_DEVICE_STATE_FD = 42, VHOST_USER_CHECK_DEVICE_STATE = 43, + VHOST_USER_GET_SHMEM_CONFIG = 44, VHOST_USER_MAX } VhostUserRequest; @@ -138,6 +139,12 @@ typedef struct VhostUserMemRegMsg { VhostUserMemoryRegion region; } VhostUserMemRegMsg; +typedef struct VhostUserShMemConfig { + uint32_t nregions; + uint32_t padding; + uint64_t memory_sizes[VHOST_MEMORY_BASELINE_NREGIONS]; +} VhostUserShMemConfig; + typedef struct VhostUserLog { uint64_t mmap_size; uint64_t mmap_offset; @@ -245,6 +252,7 @@ typedef union { VhostUserShared object; VhostUserTransferDeviceState transfer_state; VhostUserMMap mmap; + VhostUserShMemConfig shmem; } VhostUserPayload; typedef struct VhostUserMsg { @@ -3136,6 +3144,39 @@ static int vhost_user_check_device_state(struct vhost_dev *dev, Error **errp) return 0; } +static int vhost_user_get_shmem_config(struct vhost_dev *dev, + int *nregions, + uint64_t *memory_sizes, + Error **errp) +{ + int ret; + VhostUserMsg msg = { + .hdr.request = VHOST_USER_GET_SHMEM_CONFIG, + .hdr.flags = VHOST_USER_VERSION, + }; + + if (!virtio_has_feature(dev->protocol_features, + VHOST_USER_PROTOCOL_F_SHMEM)) { + return 0; + } + + ret = vhost_user_write(dev, &msg, NULL, 0); + if (ret < 0) { + return ret; + } + + ret = vhost_user_read(dev, &msg); + if (ret < 0) { + return ret; + } + + *nregions = msg.payload.shmem.nregions; + memcpy(memory_sizes, + &msg.payload.shmem.memory_sizes, + sizeof(uint64_t) * VHOST_MEMORY_BASELINE_NREGIONS); + return 0; +} + const VhostOps user_ops = { .backend_type = VHOST_BACKEND_TYPE_USER, .vhost_backend_init = vhost_user_backend_init, @@ -3174,4 +3215,5 @@ const VhostOps user_ops = { .vhost_supports_device_state = vhost_user_supports_device_state, .vhost_set_device_state_fd = vhost_user_set_device_state_fd, .vhost_check_device_state = vhost_user_check_device_state, + .vhost_get_shmem_config = vhost_user_get_shmem_config, }; diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h index 70c2e8ffee..f9c2955420 100644 --- a/include/hw/virtio/vhost-backend.h +++ b/include/hw/virtio/vhost-backend.h @@ -159,6 +159,11 @@ typedef int (*vhost_set_device_state_fd_op)(struct vhost_dev *dev, int *reply_fd, Error **errp); typedef int (*vhost_check_device_state_op)(struct vhost_dev *dev, Error **errp); +typedef int (*vhost_get_shmem_config_op)(struct vhost_dev *dev, + int *nregions, + uint64_t *memory_sizes, + Error **errp); + typedef struct VhostOps { VhostBackendType backend_type; @@ -214,6 +219,7 @@ typedef struct VhostOps { vhost_supports_device_state_op vhost_supports_device_state; vhost_set_device_state_fd_op vhost_set_device_state_fd; vhost_check_device_state_op vhost_check_device_state; + vhost_get_shmem_config_op vhost_get_shmem_config; } VhostOps; int vhost_backend_update_device_iotlb(struct vhost_dev *dev, diff --git a/include/hw/virtio/vhost-user.h b/include/hw/virtio/vhost-user.h index d7c09ffd34..e1b587a908 100644 --- a/include/hw/virtio/vhost-user.h +++ b/include/hw/virtio/vhost-user.h @@ -32,6 +32,7 @@ enum VhostUserProtocolFeature { /* Feature 17 reserved for VHOST_USER_PROTOCOL_F_XEN_MMAP. */ VHOST_USER_PROTOCOL_F_SHARED_OBJECT = 18, VHOST_USER_PROTOCOL_F_DEVICE_STATE = 19, + VHOST_USER_PROTOCOL_F_SHMEM = 20, VHOST_USER_PROTOCOL_F_MAX }; From patchwork Fri Jun 28 14:57:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Albert Esteve X-Patchwork-Id: 13716300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E54E9C2BD09 for ; Fri, 28 Jun 2024 14:58:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sND2v-0007E8-0Q; Fri, 28 Jun 2024 10:57:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2m-0007Cc-Bf for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2f-00039B-4x for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719586652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nIU+/MKeHKu2FXBeFupA7TBH7huRAp5sg/sRmWomYfg=; b=YoVNByAaqBgazMsAUz8pzuMdsPOsBT3uVpYH4OtjWQnbzS4UPEEtYBZ3dk7RpUeIgv9Uuu FxNdZENoJIqCvms+UNAjMocrPFCTXCpVHq70sj7Osd1KCvmD+gU5tSmDF+TztLAoUWwXQS /QRmL2V/2SddmwiXDhz29Pe3ZOdhimc= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-277-Yuu4rgIqPmeffH9Rh7YMcg-1; Fri, 28 Jun 2024 10:57:31 -0400 X-MC-Unique: Yuu4rgIqPmeffH9Rh7YMcg-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6117F194510D; Fri, 28 Jun 2024 14:57:30 +0000 (UTC) Received: from localhost.localdomain (unknown [10.45.224.245]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8F54119560A3; Fri, 28 Jun 2024 14:57:27 +0000 (UTC) From: Albert Esteve To: qemu-devel@nongnu.org Cc: jasowang@redhat.com, david@redhat.com, slp@redhat.com, =?utf-8?q?Alex_Be?= =?utf-8?q?nn=C3=A9e?= , stefanha@redhat.com, "Michael S. Tsirkin" , Albert Esteve Subject: [RFC PATCH v2 3/5] vhost-user-dev: Add cache BAR Date: Fri, 28 Jun 2024 16:57:08 +0200 Message-ID: <20240628145710.1516121-4-aesteve@redhat.com> In-Reply-To: <20240628145710.1516121-1-aesteve@redhat.com> References: <20240628145710.1516121-1-aesteve@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass client-ip=170.10.129.124; envelope-from=aesteve@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.206, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a cache BAR in the vhost-user-device into which files can be directly mapped. The number, shmid, and size of the VIRTIO Shared Memory subregions is retrieved through a get_shmem_config message sent by the vhost-user-base module on the realize step, after virtio_init(). By default, if VHOST_USER_PROTOCOL_F_SHMEM feature is not supported by the backend, there is no cache. Signed-off-by: Albert Esteve --- hw/virtio/vhost-user-base.c | 39 +++++++++++++++++++++++++++++-- hw/virtio/vhost-user-device-pci.c | 37 ++++++++++++++++++++++++++--- 2 files changed, 71 insertions(+), 5 deletions(-) diff --git a/hw/virtio/vhost-user-base.c b/hw/virtio/vhost-user-base.c index a83167191e..e47c568a55 100644 --- a/hw/virtio/vhost-user-base.c +++ b/hw/virtio/vhost-user-base.c @@ -268,7 +268,9 @@ static void vub_device_realize(DeviceState *dev, Error **errp) { VirtIODevice *vdev = VIRTIO_DEVICE(dev); VHostUserBase *vub = VHOST_USER_BASE(dev); - int ret; + uint64_t memory_sizes[8]; + void *cache_ptr; + int i, ret, nregions; if (!vub->chardev.chr) { error_setg(errp, "vhost-user-base: missing chardev"); @@ -311,7 +313,7 @@ static void vub_device_realize(DeviceState *dev, Error **errp) /* Allocate queues */ vub->vqs = g_ptr_array_sized_new(vub->num_vqs); - for (int i = 0; i < vub->num_vqs; i++) { + for (i = 0; i < vub->num_vqs; i++) { g_ptr_array_add(vub->vqs, virtio_add_queue(vdev, vub->vq_size, vub_handle_output)); @@ -328,6 +330,39 @@ static void vub_device_realize(DeviceState *dev, Error **errp) do_vhost_user_cleanup(vdev, vub); } + ret = vub->vhost_dev.vhost_ops->vhost_get_shmem_config(&vub->vhost_dev, + &nregions, + memory_sizes, + errp); + + if (ret < 0) { + do_vhost_user_cleanup(vdev, vub); + } + + for (i = 0; i < nregions; i++) { + if (memory_sizes[i]) { + if (!is_power_of_2(memory_sizes[i]) || + memory_sizes[i] < qemu_real_host_page_size()) { + error_setg(errp, "Shared memory %d size must be a power of 2 " + "no smaller than the page size", i); + return; + } + + cache_ptr = mmap(NULL, memory_sizes[i], PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (cache_ptr == MAP_FAILED) { + error_setg(errp, "Unable to mmap blank cache: %s", + strerror(errno)); + return; + } + + virtio_new_shmem_region(vdev); + memory_region_init_ram_ptr(&vdev->shmem_list[i], + OBJECT(vdev), "vub-shm-" + i, + memory_sizes[i], cache_ptr); + } + } + qemu_chr_fe_set_handlers(&vub->chardev, NULL, NULL, vub_event, NULL, dev, NULL, true); } diff --git a/hw/virtio/vhost-user-device-pci.c b/hw/virtio/vhost-user-device-pci.c index efaf55d3dd..314bacfb7a 100644 --- a/hw/virtio/vhost-user-device-pci.c +++ b/hw/virtio/vhost-user-device-pci.c @@ -8,14 +8,18 @@ */ #include "qemu/osdep.h" +#include "qapi/error.h" #include "hw/qdev-properties.h" #include "hw/virtio/vhost-user-base.h" #include "hw/virtio/virtio-pci.h" +#define VIRTIO_DEVICE_PCI_CACHE_BAR 2 + struct VHostUserDevicePCI { VirtIOPCIProxy parent_obj; VHostUserBase vub; + MemoryRegion cachebar; }; #define TYPE_VHOST_USER_DEVICE_PCI "vhost-user-device-pci-base" @@ -25,10 +29,37 @@ OBJECT_DECLARE_SIMPLE_TYPE(VHostUserDevicePCI, VHOST_USER_DEVICE_PCI) static void vhost_user_device_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp) { VHostUserDevicePCI *dev = VHOST_USER_DEVICE_PCI(vpci_dev); - DeviceState *vdev = DEVICE(&dev->vub); - + DeviceState *dev_state = DEVICE(&dev->vub); + VirtIODevice *vdev = VIRTIO_DEVICE(dev_state); + uint64_t offset = 0, cache_size = 0; + int i; + vpci_dev->nvectors = 1; - qdev_realize(vdev, BUS(&vpci_dev->bus), errp); + qdev_realize(dev_state, BUS(&vpci_dev->bus), errp); + + for (i = 0; i < vdev->n_shmem_regions; i++) { + if (vdev->shmem_list[i].size > UINT64_MAX - cache_size) { + error_setg(errp, "Total shared memory required overflow"); + return; + } + cache_size = cache_size + vdev->shmem_list[i].size; + } + if (cache_size) { + memory_region_init(&dev->cachebar, OBJECT(vpci_dev), + "vhost-device-pci-cachebar", cache_size); + for (i = 0; i < vdev->n_shmem_regions; i++) { + memory_region_add_subregion(&dev->cachebar, offset, + &vdev->shmem_list[i]); + virtio_pci_add_shm_cap(vpci_dev, VIRTIO_DEVICE_PCI_CACHE_BAR, + offset, vdev->shmem_list[i].size, i); + offset = offset + vdev->shmem_list[i].size; + } + pci_register_bar(&vpci_dev->pci_dev, VIRTIO_DEVICE_PCI_CACHE_BAR, + PCI_BASE_ADDRESS_SPACE_MEMORY | + PCI_BASE_ADDRESS_MEM_PREFETCH | + PCI_BASE_ADDRESS_MEM_TYPE_64, + &dev->cachebar); + } } static void vhost_user_device_pci_class_init(ObjectClass *klass, void *data) From patchwork Fri Jun 28 14:57:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Albert Esteve X-Patchwork-Id: 13716303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F2C0C30658 for ; Fri, 28 Jun 2024 14:58:57 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sND30-0007Hr-BX; Fri, 28 Jun 2024 10:57:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2m-0007Ce-U1 for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2g-00039Y-OU for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719586656; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Gx+RAdbQo8yE3JDHhoP3CBDorc03ZP3HfOnKT4YuyA0=; b=h3mQN8A5pHQX7SaZq0Gt2U/sUvP6n9M9SnRZsfXsy0L+hU/lURloh1111jg2vjMpINO3OM CrgJG0/DWT1s58T5GcpGdiGb9Te9LKLKfwkeOMMnIm1elJAulMeUZtXn+rk+ceeWId4iJO lSZdCT4ZmbUfAZZr9pCqEy+ZW+puy34= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-185-FPQrFTnhPNmTvmQ4r3rOmg-1; Fri, 28 Jun 2024 10:57:34 -0400 X-MC-Unique: FPQrFTnhPNmTvmQ4r3rOmg-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CBCFA19560A7; Fri, 28 Jun 2024 14:57:33 +0000 (UTC) Received: from localhost.localdomain (unknown [10.45.224.245]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D032D19560A3; Fri, 28 Jun 2024 14:57:30 +0000 (UTC) From: Albert Esteve To: qemu-devel@nongnu.org Cc: jasowang@redhat.com, david@redhat.com, slp@redhat.com, =?utf-8?q?Alex_Be?= =?utf-8?q?nn=C3=A9e?= , stefanha@redhat.com, "Michael S. Tsirkin" , Albert Esteve Subject: [RFC PATCH v2 4/5] vhost_user: Add MEM_READ/WRITE backend requests Date: Fri, 28 Jun 2024 16:57:09 +0200 Message-ID: <20240628145710.1516121-5-aesteve@redhat.com> In-Reply-To: <20240628145710.1516121-1-aesteve@redhat.com> References: <20240628145710.1516121-1-aesteve@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass client-ip=170.10.129.124; envelope-from=aesteve@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.206, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org With SHMEM_MAP messages, sharing descriptors between devices will cause that these devices do not see the mappings, and fail to access these memory regions. To solve this, introduce MEM_READ/WRITE requests that will get triggered as a fallback when vhost-user memory translation fails. Signed-off-by: Albert Esteve --- hw/virtio/vhost-user.c | 31 +++++++++ subprojects/libvhost-user/libvhost-user.c | 84 +++++++++++++++++++++++ subprojects/libvhost-user/libvhost-user.h | 38 ++++++++++ 3 files changed, 153 insertions(+) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 57406dc8b4..18cacb2d68 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -118,6 +118,8 @@ typedef enum VhostUserBackendRequest { VHOST_USER_BACKEND_SHARED_OBJECT_LOOKUP = 8, VHOST_USER_BACKEND_SHMEM_MAP = 9, VHOST_USER_BACKEND_SHMEM_UNMAP = 10, + VHOST_USER_BACKEND_MEM_READ = 11, + VHOST_USER_BACKEND_MEM_WRITE = 12, VHOST_USER_BACKEND_MAX } VhostUserBackendRequest; @@ -145,6 +147,12 @@ typedef struct VhostUserShMemConfig { uint64_t memory_sizes[VHOST_MEMORY_BASELINE_NREGIONS]; } VhostUserShMemConfig; +typedef struct VhostUserMemRWMsg { + uint64_t guest_address; + uint32_t size; + uint8_t data[]; +} VhostUserMemRWMsg; + typedef struct VhostUserLog { uint64_t mmap_size; uint64_t mmap_offset; @@ -253,6 +261,7 @@ typedef union { VhostUserTransferDeviceState transfer_state; VhostUserMMap mmap; VhostUserShMemConfig shmem; + VhostUserMemRWMsg mem_rw; } VhostUserPayload; typedef struct VhostUserMsg { @@ -1871,6 +1880,22 @@ vhost_user_backend_handle_shmem_unmap(struct vhost_dev *dev, return 0; } +static int +vhost_user_backend_handle_mem_read(struct vhost_dev *dev, + VhostUserMemRWMsg *mem_rw) +{ + /* TODO */ + return -EPERM; +} + +static int +vhost_user_backend_handle_mem_write(struct vhost_dev *dev, + VhostUserMemRWMsg *mem_rw) +{ + /* TODO */ + return -EPERM; +} + static void close_backend_channel(struct vhost_user *u) { g_source_destroy(u->backend_src); @@ -1946,6 +1971,12 @@ static gboolean backend_read(QIOChannel *ioc, GIOCondition condition, case VHOST_USER_BACKEND_SHMEM_UNMAP: ret = vhost_user_backend_handle_shmem_unmap(dev, &payload.mmap); break; + case VHOST_USER_BACKEND_MEM_READ: + ret = vhost_user_backend_handle_mem_read(dev, &payload.mem_rw); + break; + case VHOST_USER_BACKEND_MEM_WRITE: + ret = vhost_user_backend_handle_mem_write(dev, &payload.mem_rw); + break; default: error_report("Received unexpected msg type: %d.", hdr.request); ret = -EINVAL; diff --git a/subprojects/libvhost-user/libvhost-user.c b/subprojects/libvhost-user/libvhost-user.c index 28556d183a..b5184064b5 100644 --- a/subprojects/libvhost-user/libvhost-user.c +++ b/subprojects/libvhost-user/libvhost-user.c @@ -1651,6 +1651,90 @@ vu_shmem_unmap(VuDev *dev, uint8_t shmid, uint64_t fd_offset, return vu_process_message_reply(dev, &vmsg); } +bool +vu_send_mem_read(VuDev *dev, uint64_t guest_addr, uint32_t size, + uint8_t *data) +{ + VhostUserMsg msg_reply; + VhostUserMsg msg = { + .request = VHOST_USER_BACKEND_MEM_READ, + .size = sizeof(msg.payload.mem_rw), + .flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK, + .payload = { + .mem_rw = { + .guest_address = guest_addr, + .size = size, + } + } + }; + + pthread_mutex_lock(&dev->backend_mutex); + if (!vu_message_write(dev, dev->backend_fd, &msg)) { + goto out_err; + } + + if (!vu_message_read_default(dev, dev->backend_fd, &msg_reply)) { + goto out_err; + } + + if (msg_reply.request != msg.request) { + DPRINT("Received unexpected msg type. Expected %d, received %d", + msg.request, msg_reply.request); + goto out_err; + } + + if (msg_reply.payload.mem_rw.size != size) { + DPRINT("Received unexpected number of bytes in the response. " + "Expected %d, received %d", + size, msg_reply.payload.mem_rw.size); + goto out_err; + } + + data = malloc(msg_reply.payload.mem_rw.size); + if (!data) { + DPRINT("Failed to malloc read memory data"); + goto out_err; + } + + memcpy(data, msg_reply.payload.mem_rw.data, size); + pthread_mutex_unlock(&dev->backend_mutex); + return true; + +out_err: + pthread_mutex_unlock(&dev->backend_mutex); + return false; +} + +bool +vu_send_mem_write(VuDev *dev, uint64_t guest_addr, uint32_t size, + uint8_t *data) +{ + VhostUserMsg msg = { + .request = VHOST_USER_BACKEND_MEM_WRITE, + .size = sizeof(msg.payload.mem_rw), + .flags = VHOST_USER_VERSION, + .payload = { + .mem_rw = { + .guest_address = guest_addr, + .size = size, + } + } + }; + memcpy(msg.payload.mem_rw.data, data, size); + + if (vu_has_protocol_feature(dev, VHOST_USER_PROTOCOL_F_REPLY_ACK)) { + msg.flags |= VHOST_USER_NEED_REPLY_MASK; + } + + if (!vu_message_write(dev, dev->backend_fd, &msg)) { + pthread_mutex_unlock(&dev->backend_mutex); + return false; + } + + /* Also unlocks the backend_mutex */ + return vu_process_message_reply(dev, &msg); +} + static bool vu_set_vring_call_exec(VuDev *dev, VhostUserMsg *vmsg) { diff --git a/subprojects/libvhost-user/libvhost-user.h b/subprojects/libvhost-user/libvhost-user.h index 7f6c22cc1a..8ef794870d 100644 --- a/subprojects/libvhost-user/libvhost-user.h +++ b/subprojects/libvhost-user/libvhost-user.h @@ -129,6 +129,8 @@ typedef enum VhostUserBackendRequest { VHOST_USER_BACKEND_SHARED_OBJECT_LOOKUP = 8, VHOST_USER_BACKEND_SHMEM_MAP = 9, VHOST_USER_BACKEND_SHMEM_UNMAP = 10, + VHOST_USER_BACKEND_MEM_READ = 11, + VHOST_USER_BACKEND_MEM_WRITE = 12, VHOST_USER_BACKEND_MAX } VhostUserBackendRequest; @@ -152,6 +154,12 @@ typedef struct VhostUserMemRegMsg { VhostUserMemoryRegion region; } VhostUserMemRegMsg; +typedef struct VhostUserMemRWMsg { + uint64_t guest_address; + uint32_t size; + uint8_t data[]; +} VhostUserMemRWMsg; + typedef struct VhostUserLog { uint64_t mmap_size; uint64_t mmap_offset; @@ -235,6 +243,7 @@ typedef struct VhostUserMsg { VhostUserInflight inflight; VhostUserShared object; VhostUserMMap mmap; + VhostUserMemRWMsg mem_rw; } payload; int fds[VHOST_MEMORY_BASELINE_NREGIONS]; @@ -650,6 +659,35 @@ bool vu_shmem_map(VuDev *dev, uint8_t shmid, uint64_t fd_offset, bool vu_shmem_unmap(VuDev *dev, uint8_t shmid, uint64_t fd_offset, uint64_t shm_offset, uint64_t len); +/** + * vu_send_mem_read: + * @dev: a VuDev context + * @guest_addr: guest physical address to read + * @size: number of bytes to read + * @data: head of an unitialized bytes array + * + * Reads `size` bytes of `guest_addr` in the frontend and stores + * them in `data`. + * + * Returns: TRUE on success, FALSE on failure. + */ +bool vu_send_mem_read(VuDev *dev, uint64_t guest_addr, uint32_t size, + uint8_t *data); + +/** + * vu_send_mem_write: + * @dev: a VuDev context + * @guest_addr: guest physical address to write + * @size: number of bytes to write + * @data: head of an array with `size` bytes to write + * + * Writes `size` bytes from `data` into `guest_addr` in the frontend. + * + * Returns: TRUE on success, FALSE on failure. + */ +bool vu_send_mem_write(VuDev *dev, uint64_t guest_addr, uint32_t size, + uint8_t *data); + /** * vu_queue_set_notification: * @dev: a VuDev context From patchwork Fri Jun 28 14:57:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Albert Esteve X-Patchwork-Id: 13716299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E75A5C3064D for ; Fri, 28 Jun 2024 14:58:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sND2z-0007Hf-Me; Fri, 28 Jun 2024 10:57:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2t-0007E7-GF for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:51 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sND2m-0003A5-5K for qemu-devel@nongnu.org; Fri, 28 Jun 2024 10:57:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719586662; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=95ZBOAFeQ4dSgCRsZThJ9ANpiObHEVIWQMUOqGoa0os=; b=TM/ResZmtUac+ArjNAFZuxwYJ/mnKr8GgYCLY5hoz2y6mUfQghQlnjEPY4jjL1GlGomW6v rWGh4vShUvHT+clAHm2HlioGakSSwz3JHEnRyL9NDsY83LCzd7okz4H0TyYV8WZIdpbwWr 4l4VPPWWTpWxrB16gjn5C+fl6WOGUjc= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-214-noSeCbp3O_GosPSJKTvTJQ-1; Fri, 28 Jun 2024 10:57:39 -0400 X-MC-Unique: noSeCbp3O_GosPSJKTvTJQ-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 17ED41944DE4; Fri, 28 Jun 2024 14:57:38 +0000 (UTC) Received: from localhost.localdomain (unknown [10.45.224.245]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 740281944DC5; Fri, 28 Jun 2024 14:57:34 +0000 (UTC) From: Albert Esteve To: qemu-devel@nongnu.org Cc: jasowang@redhat.com, david@redhat.com, slp@redhat.com, =?utf-8?q?Alex_Be?= =?utf-8?q?nn=C3=A9e?= , stefanha@redhat.com, "Michael S. Tsirkin" , Albert Esteve Subject: [RFC PATCH v2 5/5] vhost_user: Implement mem_read/mem_write handlers Date: Fri, 28 Jun 2024 16:57:10 +0200 Message-ID: <20240628145710.1516121-6-aesteve@redhat.com> In-Reply-To: <20240628145710.1516121-1-aesteve@redhat.com> References: <20240628145710.1516121-1-aesteve@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 Received-SPF: pass client-ip=170.10.133.124; envelope-from=aesteve@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.206, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Implement function handlers for memory read and write operations. Signed-off-by: Albert Esteve --- hw/virtio/vhost-user.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 18cacb2d68..79becbc87b 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -1884,16 +1884,42 @@ static int vhost_user_backend_handle_mem_read(struct vhost_dev *dev, VhostUserMemRWMsg *mem_rw) { - /* TODO */ - return -EPERM; + ram_addr_t offset; + int fd; + MemoryRegion *mr; + + mr = vhost_user_get_mr_data(mem_rw->guest_address, &offset, &fd); + + if (!mr) { + error_report("Failed to get memory region with address %" PRIx64, + mem_rw->guest_address); + return -EFAULT; + } + + memcpy(mem_rw->data, memory_region_get_ram_ptr(mr) + offset, mem_rw->size); + + return 0; } static int vhost_user_backend_handle_mem_write(struct vhost_dev *dev, VhostUserMemRWMsg *mem_rw) { - /* TODO */ - return -EPERM; + ram_addr_t offset; + int fd; + MemoryRegion *mr; + + mr = vhost_user_get_mr_data(mem_rw->guest_address, &offset, &fd); + + if (!mr) { + error_report("Failed to get memory region with address %" PRIx64, + mem_rw->guest_address); + return -EFAULT; + } + + memcpy(memory_region_get_ram_ptr(mr) + offset, mem_rw->data, mem_rw->size); + + return 0; } static void close_backend_channel(struct vhost_user *u)