From patchwork Mon Dec 9 07:00:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Raphael Norwitz X-Patchwork-Id: 11296405 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 81B83112B for ; Tue, 17 Dec 2019 01:56:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 61E042082E for ; Tue, 17 Dec 2019 01:56:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 61E042082E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nutanix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:34164 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ih26c-0003Sf-7e for patchwork-qemu-devel@patchwork.kernel.org; Mon, 16 Dec 2019 20:56:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:38577) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ih25g-0002E9-Iu for qemu-devel@nongnu.org; Mon, 16 Dec 2019 20:56:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ih25f-0001SM-5h for qemu-devel@nongnu.org; Mon, 16 Dec 2019 20:56:00 -0500 Received: from [192.146.154.1] (port=53222 helo=mcp01.nutanix.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ih25e-0001Oj-UO for qemu-devel@nongnu.org; Mon, 16 Dec 2019 20:55:59 -0500 Received: from localhost.corp.nutanix.com (unknown [10.40.33.233]) by mcp01.nutanix.com (Postfix) with ESMTP id 563BE1008B6E; Tue, 17 Dec 2019 01:55:54 +0000 (UTC) From: Raphael Norwitz To: mst@redhat.com, qemu-devel@nongnu.org Subject: [RFC PATCH 0/3] vhost-user: Lift Max Ram Slots Limitation Date: Mon, 9 Dec 2019 02:00:44 -0500 Message-Id: <1575874847-5792-1-git-send-email-raphael.norwitz@nutanix.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 192.146.154.1 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: raphael.s.norwitz@gmail.com, Raphael Norwitz Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" In QEMU today, a VM with a vhost-user device can hot add memory a maximum of 8 times. See these threads, among others: [1] https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg01046.html https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg01236.html [2] https://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04656.html This RFC/patch set introduces a new protocol feature VHOST_USER_PROTOCOL_F_CONFIGURE_SLOTS which, when enabled, lifts the restriction on the maximum number RAM slots imposed by vhost-user. The patch consists of 3 changes: 1. Fixed Error Handling in vhost_user_set_mem_table_postcopy: This is a bug fix in the postcopy migration path 2. vhost-user: Refactor vhost_user_set_mem_table Functions: This is a non-functional change refractoring the vhost_user_set_mem_table and vhost_user_set_mem_table_postcopy functions such that the feature can be more cleanly added. 3. Introduce Configurable Number of Memory Slots Exposed by vhost-user: This change introduces the new protocol feature VHOST_USER_PROTOCOL_F_CONFIGURE_SLOTS. The implementation details are explained in more detail in the commit messages, but at a high level the new protocol feature works as follows: - If the VHOST_USER_PROTCOL_F_CONFIGURE_SLOTS feature is enabled, QEMU will send multiple VHOST_USER_ADD_MEM_REG and VHOST_USER_REM_MEM_REG messages to map and unmap individual memory regions instead of one large VHOST_USER_SET_MEM_TABLE message containing all memory regions. - The vhost-user struct maintains a ’shadow state’ of memory regions already sent to the guest. Each time vhost_user_set_mem_table is called, the shadow state is compared with the new device state. A VHOST_USER_REM_MEM_REG will be sent for each region in the shadow state not in the device state. Then, a VHOST_USER_ADD_MEM_REG will be sent for each region in the device state but not the shadow state. After these messages have been sent, the shadow state will be updated to reflect the new device state. The VHOST_USER_SET_MEM_TABLE message was not reused because as the number of regions grows, the message becomes very large. In practice, such large messages caused problems (truncated messages) and in the past it seems the community has opted for smaller fixed size messages where possible. VRINGs, for example, are sent to the backend individually instead of in one massive message. Current Limitations: - postcopy migration is not supported when the VHOST_USER_PROTOCOL_F_CONFIGURE_SLOTS has been negotiated. - VHOST_USER_PROTOCOL_F_CONFIGURE_SLOTS cannot be negotiated when VHOST_USER_PROTOCOL_F_REPLY_ACK has also been negotiated. Both of these limitations are due to resource contraints. They are not imposed for technical reasons. Questions: - In the event transmitting a VHOST_USER_ADD_MEM_REG or VHOST_USER_REM_REG message fails, is there any reason the error handling should differ from when transmitting VHOST_USER_SET_MEM_TABLE message fails? - Is there a cleaner way to ensure to ensure a postcopy migration cannot be started with this protocol feature enabled? Best, Raphael Raphael Norwitz (3): Fixed Error Handling in vhost_user_set_mem_table_postcopy vhost-user: Refactor vhost_user_set_mem_table Functions Introduce Configurable Number of Memory Slots Exposed by vhost-user: docs/interop/vhost-user.rst | 43 +++++ hw/virtio/vhost-user.c | 384 +++++++++++++++++++++++++++++++++----------- 2 files changed, 335 insertions(+), 92 deletions(-)