From patchwork Mon Aug 23 14:16:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB941C4320A for ; Mon, 23 Aug 2021 14:20:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4A41A61372 for ; Mon, 23 Aug 2021 14:20:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4A41A61372 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52752 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAoh-0006yH-Dl for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:20:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41426) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAmG-0000Zl-Sz for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:18:17 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:45442) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAmE-0001ky-Pj for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:18:16 -0400 Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE3hVg121939; Mon, 23 Aug 2021 10:18:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=+WDovjiY/NOTRnrC1TbX+SwWuPgPSM5HkRu5XduuT/k=; b=hUHGzyBcvqz+KUmDsiDILwYtx/rvpss2mWVZ1GPIil8rOBVcXlYp6xprdqS1BnQQQEig gjzGps4xh2ivXOnEVUMigoNy+5RKwDPqkZa400LBaIDCNqAB/lQ2tOg1DuOHw6aUA8y1 m/yxVNnO5f5yNNAnks4tC3YRv4YD4o/zYgGWrbCgJ9AjtnQRWxFe6DnKiXXEVEhfGcAE JMBGsAYm6OtnSI/Es6mD6JV+5C6B1oowyyYxqCQfw//JOKTmemIVx1YU/gRWemDArbBJ qYODP84GCMBmCTtiuZmNhyT2QHzLHXMCycve4rqQAzJQ5MelLxrDz96nHPx7lI7f7+1T uw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akeg0268g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:18:11 -0400 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE4MqH126096; Mon, 23 Aug 2021 10:18:11 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akeg0267v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:18:11 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDMv9016512; Mon, 23 Aug 2021 14:18:09 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma01dal.us.ibm.com with ESMTP id 3ajs4c3ntd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:17:53 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGkbb30146964 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:46 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B1A06112065; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6FE86112067; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 01/12] migration: Add helpers to save confidential RAM Date: Mon, 23 Aug 2021 10:16:25 -0400 Message-Id: <20210823141636.65975-2-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 5_t-pyuLWHlxuI2pUSxT9B1wJ41Y0D-9 X-Proofpoint-GUID: hULpaUsxnuU82D5VsYBd2crAOV3OFARx X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 clxscore=1015 phishscore=0 mlxscore=0 malwarescore=0 bulkscore=0 spamscore=0 impostorscore=0 mlxlogscore=999 lowpriorityscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.156.1; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" QEMU cannot read the memory of memory-encrypted guests, which is required for sending RAM to the migration target. Instead, QEMU asks a migration helper running on an auxiliary vcpu in the guest to extract pages from memory; these pages are encrypted with a transfer key that is known to the source and target guests, but not to both QEMUs. The interaction with the guest migration helper is performed using two shared (unencrypted) pages which both QEMU and guest can read from and write to. The details of the mailbox protocol are described in migration/confidential-ram.c. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 17 ++++ migration/confidential-ram.c | 184 +++++++++++++++++++++++++++++++++++ migration/meson.build | 2 +- migration/trace-events | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 migration/confidential-ram.h create mode 100644 migration/confidential-ram.c diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h new file mode 100644 index 0000000000..0d49718d31 --- /dev/null +++ b/migration/confidential-ram.h @@ -0,0 +1,17 @@ +/* + * QEMU migration for confidential guest's RAM + */ + +#ifndef QEMU_CONFIDENTIAL_RAM_H +#define QEMU_CONFIDENTIAL_RAM_H + +#include "exec/cpu-common.h" +#include "qemu-file.h" + +void cgs_mh_init(void); +void cgs_mh_cleanup(void); + +int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, + uint64_t *bytes_sent); + +#endif diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c new file mode 100644 index 0000000000..65a588e7f6 --- /dev/null +++ b/migration/confidential-ram.c @@ -0,0 +1,184 @@ +#include "qemu/osdep.h" +#include "cpu.h" +#include "qemu/osdep.h" +#include "qemu/error-report.h" +#include "qemu/rcu.h" +#include "qemu/coroutine.h" +#include "qemu/timer.h" +#include "io/channel.h" +#include "qapi/error.h" +#include "exec/memory.h" +#include "trace.h" +#include "confidential-ram.h" + +enum cgs_mig_helper_cmd { + /* Initialize migration helper in guest */ + CGS_MIG_HELPER_CMD_INIT = 0, + + /* + * Fetch a page from gpa, encrypt it, and save result into the shared page + */ + CGS_MIG_HELPER_CMD_ENCRYPT, + + /* Read the shared page, decrypt it, and save result into gpa */ + CGS_MIG_HELPER_CMD_DECRYPT, + + /* Reset migration helper in guest */ + CGS_MIG_HELPER_CMD_RESET, + + CGS_MIG_HELPER_CMD_MAX +}; + +struct QEMU_PACKED CGSMigHelperCmdParams { + uint64_t cmd_type; + uint64_t gpa; + int32_t prefetch; + int32_t ret; + int32_t go; + int32_t done; +}; +typedef struct CGSMigHelperCmdParams CGSMigHelperCmdParams; + +struct QEMU_PACKED CGSMigHelperPageHeader { + uint32_t len; + uint8_t data[0]; +}; +typedef struct CGSMigHelperPageHeader CGSMigHelperPageHeader; + +struct CGSMigHelperState { + CGSMigHelperCmdParams *cmd_params; + CGSMigHelperPageHeader *io_page_hdr; + uint8_t *io_page; + bool initialized; +}; +typedef struct CGSMigHelperState CGSMigHelperState; + +static CGSMigHelperState cmhs = {0}; + +#define MH_BUSYLOOP_TIMEOUT 100000000LL +#define MH_REQUEST_TIMEOUT_MS 100 +#define MH_REQUEST_TIMEOUT_NS (MH_REQUEST_TIMEOUT_MS * 1000 * 1000) + +/* + * The migration helper shared area is hard-coded at gpa 0x820000 with size of + * 2 pages (0x2000 bytes). Instead of hard-coding, the address and size may be + * fetched from OVMF itself using a pc_system_ovmf_table_find call to query + * OVMF's GUIDed structure for a migration helper GUID. + */ +#define MH_SHARED_CMD_PARAMS_ADDR 0x820000 +#define MH_SHARED_IO_PAGE_HDR_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x800) +#define MH_SHARED_IO_PAGE_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x1000) + +void cgs_mh_init(void) +{ + RCU_READ_LOCK_GUARD(); + cmhs.cmd_params = qemu_map_ram_ptr(NULL, MH_SHARED_CMD_PARAMS_ADDR); + cmhs.io_page_hdr = qemu_map_ram_ptr(NULL, MH_SHARED_IO_PAGE_HDR_ADDR); + cmhs.io_page = qemu_map_ram_ptr(NULL, MH_SHARED_IO_PAGE_ADDR); +} + +static int send_command_to_cgs_mig_helper(uint64_t cmd_type, uint64_t gpa) +{ + /* + * The cmd_params struct is on a page shared with the guest migration + * helper. We use a volatile struct to force writes to memory so that the + * guest can see them. + */ + volatile CGSMigHelperCmdParams *params = cmhs.cmd_params; + int64_t counter, request_timeout_at; + + /* + * At this point io_page and io_page_hdr should be already filled according + * to the requested cmd_type. + */ + + params->cmd_type = cmd_type; + params->gpa = gpa; + params->prefetch = 0; + params->ret = -1; + params->done = 0; + + /* + * Force writes of all command parameters before writing the 'go' flag. + * The guest migration handler waits for the go flag and then reads the + * command parameters. + */ + smp_wmb(); + + /* Tell the migration helper to start working on this command */ + params->go = 1; + + /* + * Wait for the guest migration helper to process the command and mark the + * done flag + */ + request_timeout_at = qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + + MH_REQUEST_TIMEOUT_NS; + do { + counter = 0; + while (!params->done && (counter < MH_BUSYLOOP_TIMEOUT)) { + counter++; + } + } while (!params->done && + qemu_clock_get_ns(QEMU_CLOCK_REALTIME) < request_timeout_at); + + if (!params->done) { + error_report("Migration helper command %" PRIu64 " timed-out for " + "gpa 0x%" PRIx64, cmd_type, gpa); + return -EIO; + } + + return params->ret; +} + +static void init_cgs_mig_helper_if_needed(void) +{ + int ret; + + if (cmhs.initialized) { + return; + } + + ret = send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_INIT, 0); + if (ret == 0) { + cmhs.initialized = true; + } +} + +void cgs_mh_cleanup(void) +{ + send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_RESET, 0); +} + +int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, + uint64_t *bytes_sent) +{ + int ret; + + init_cgs_mig_helper_if_needed(); + + /* Ask the migration helper to encrypt the page at src_gpa */ + trace_encrypted_ram_save_page(size, src_gpa); + ret = send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_ENCRYPT, src_gpa); + if (ret) { + error_report("Error cgs_mh_save_encrypted_page ret=%d", ret); + return -1; + } + + /* Sanity check for response header */ + if (cmhs.io_page_hdr->len > 1024) { + error_report("confidential-ram: migration helper response is too large " + "(len=%u)", cmhs.io_page_hdr->len); + return -EINVAL; + } + + qemu_put_be32(f, cmhs.io_page_hdr->len); + qemu_put_buffer(f, cmhs.io_page_hdr->data, cmhs.io_page_hdr->len); + *bytes_sent = 4 + cmhs.io_page_hdr->len; + + qemu_put_be32(f, size); + qemu_put_buffer(f, cmhs.io_page, size); + *bytes_sent += 4 + size; + + return ret; +} diff --git a/migration/meson.build b/migration/meson.build index f8714dcb15..774223c1a3 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -32,4 +32,4 @@ softmmu_ss.add(when: 'CONFIG_LIVE_BLOCK_MIGRATION', if_true: files('block.c')) softmmu_ss.add(when: zstd, if_true: files('multifd-zstd.c')) specific_ss.add(when: 'CONFIG_SOFTMMU', - if_true: files('dirtyrate.c', 'ram.c', 'target.c')) + if_true: files('dirtyrate.c', 'ram.c', 'target.c', 'confidential-ram.c')) diff --git a/migration/trace-events b/migration/trace-events index a1c0f034ab..3d442a767f 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -344,3 +344,6 @@ migration_block_save_pending(uint64_t pending) "Enter save live pending %" PRIu # page_cache.c migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRId64 migration_pagecache_insert(void) "Error allocating page" + +# confidential-ram.c +encrypted_ram_save_page(uint32_t size, uint64_t gpa) "size: %u, gpa: 0x%" PRIx64 From patchwork Mon Aug 23 14:16:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D9CDC4320A for ; Mon, 23 Aug 2021 14:18:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3040A61372 for ; Mon, 23 Aug 2021 14:18:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3040A61372 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:43666 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAmf-0000wr-7l for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:18:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41134) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl0-0006jp-Cu for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:58 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:22192) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAkw-0000tc-Vo for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:58 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE70YS154199; Mon, 23 Aug 2021 10:16:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=tGjOf/3bcu6EQF+Z5Hw4WgQQacNaqqY3qF2Lh/PjfPM=; b=ExDP9Hu8WrJz2+0kN+9tGXNDWfxA1DxbHP0OUSrbsxOiGsDI5WRkCKZxG5ADxUoJdm+g LvDsGBXlmghugC5jDJRgQQN+QAqbhbRdqPQkXh1mh9ts+KPYP4sDFzxDxl93G/Q3AXHo xdTLtgnWFQ6ZqRonAqMNF6H5dhzrSUJLtkcJqs8MODMc35ikkfrzd/eaxg+44cHkxwQH KrSpurZOhOoiazaFQ+Oc/4c32wBhhiRDBx79IQHjdM7UIk7effHcqiHIGB7UbEvZskmU i963g3IffGUXk6gij59snLuWmKH6lj7TE5b9QA2yzePG7Ry4ITKFekZP3fSoookg9ZA2 4w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akf28sdtx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:49 -0400 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE7CnD155544; Mon, 23 Aug 2021 10:16:49 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akf28sdtg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:49 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDHhf031972; Mon, 23 Aug 2021 14:16:48 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma01wdc.us.ibm.com with ESMTP id 3ajs4ax6e9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:48 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGlVK17367460 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:47 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EB437112065; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BB0D6112066; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 02/12] migration: Add helpers to load confidential RAM Date: Mon, 23 Aug 2021 10:16:26 -0400 Message-Id: <20210823141636.65975-3-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: RkQCRBHVK1VeMXIJeL9YvtQYxTvYiKt- X-Proofpoint-ORIG-GUID: InJLUgzJWzjd1Hpe-pz9l8kSfS1FXPJ- X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_02:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 priorityscore=1501 phishscore=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 impostorscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.158.5; envelope-from=dovmurik@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" QEMU cannot write directly to the memory of memory-encrypted guests; this breaks normal RAM-load in the migration target. Instead, QEMU asks a migration helper running on an auxiliary vcpu in the guest to restore encrypted pages as they were received from the source to a specific GPA. The migration helper running inside the guest can safely decrypt the pages arrived from the source and load them into their proper location in the guest's memory. Loading pages uses the same shared (unencrypted) pages which both QEMU and the guest can read from and write to. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 37 ++++++++++++++++++++++++++++++++++++ migration/trace-events | 1 + 3 files changed, 40 insertions(+) diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h index 0d49718d31..ebe4073bce 100644 --- a/migration/confidential-ram.h +++ b/migration/confidential-ram.h @@ -14,4 +14,6 @@ void cgs_mh_cleanup(void); int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, uint64_t *bytes_sent); +int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa); + #endif diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c index 65a588e7f6..053ecea1d4 100644 --- a/migration/confidential-ram.c +++ b/migration/confidential-ram.c @@ -182,3 +182,40 @@ int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, return ret; } + +int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa) +{ + int ret = 1; + uint32_t page_hdr_len, enc_page_len; + + init_cgs_mig_helper_if_needed(); + + assert((dest_gpa & TARGET_PAGE_MASK) == dest_gpa); + + /* Read page header */ + page_hdr_len = qemu_get_be32(f); + if (page_hdr_len > 1024) { + error_report("confidential-ram: page header is too large (%d bytes) " + "when loading gpa 0x%" PRIx64, page_hdr_len, dest_gpa); + return -EINVAL; + } + cmhs.io_page_hdr->len = page_hdr_len; + qemu_get_buffer(f, cmhs.io_page_hdr->data, page_hdr_len); + + /* Read encrypted page */ + enc_page_len = qemu_get_be32(f); + if (enc_page_len != TARGET_PAGE_SIZE) { + error_report("confidential-ram: encrypted page is too large (%d bytes) " + "when loading gpa 0x%" PRIx64, enc_page_len, dest_gpa); + return -EINVAL; + } + qemu_get_buffer(f, cmhs.io_page, enc_page_len); + + trace_encrypted_ram_load_page(page_hdr_len, enc_page_len, dest_gpa); + ret = send_command_to_cgs_mig_helper(CGS_MIG_HELPER_CMD_DECRYPT, dest_gpa); + if (ret) { + error_report("confidential-ram: failed loading page at gpa " + "0x%" PRIx64 ": ret=%d", dest_gpa, ret); + } + return ret; +} diff --git a/migration/trace-events b/migration/trace-events index 3d442a767f..5a6b5c8230 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -346,4 +346,5 @@ migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRI migration_pagecache_insert(void) "Error allocating page" # confidential-ram.c +encrypted_ram_load_page(uint32_t hdr_len, uint32_t trans_len, uint64_t gpa) "hdr_len: %u, trans_len: %u, gpa: 0x%" PRIx64 encrypted_ram_save_page(uint32_t size, uint64_t gpa) "size: %u, gpa: 0x%" PRIx64 From patchwork Mon Aug 23 14:16:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DB6BC4320E for ; Mon, 23 Aug 2021 14:18:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0CF1A613AD for ; Mon, 23 Aug 2021 14:18:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0CF1A613AD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:43610 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAmf-0000uj-00 for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:18:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41170) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl1-0006k5-JQ for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:59 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:36407 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000v6-7c for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:59 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NEBxjt147206; Mon, 23 Aug 2021 10:16:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=WZAqgdalblGZs59iyVHmRv3UBNpHs+vOrmPdnc1cRQw=; b=n02YSaZVkLPPvic2vdV66tCkjxgUaMpKOP5lOdswapDSaTLe3n+mqiWmwt1rRD5GT6Cw UyLFvOVDXwUag8JgCXf1UMoAj73AaAixwNyQaDx6HzVuhKhJIvi+TlexwSCc4gLcQPSv Xib04Z3Fqsy/LhsjYPQcZKU088bTz9uYnfNL9Jfk4bha+RAJZkJ6FNC8JmzOIJOPnA3X Q6gICr2CIqakcslyUkjIM7ATKHEu8eOdEI18LHcHMFnDTQm/efhHmtdyTJqlvpiQs6qA AgLUDn16o1z8H4KWISnFdNUIUB50IEwWX9ZeOcFKIJ1yLAeY98qZ3XVGpwHofld4RJOT Cw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3akexysrur-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:49 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE4bP1105622; Mon, 23 Aug 2021 10:16:49 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 3akexysru3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:49 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDD3A021205; Mon, 23 Aug 2021 14:16:48 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma04dal.us.ibm.com with ESMTP id 3ajs4bknsv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:48 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGlHM17236318 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:47 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4442C112066; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0A612112069; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:46 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 03/12] migration: Introduce gpa_inside_migration_helper_shared_area Date: Mon, 23 Aug 2021 10:16:27 -0400 Message-Id: <20210823141636.65975-4-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 2Bv8RuuXr51BjuvFT3rLmzWGDpGkNg28 X-Proofpoint-ORIG-GUID: 6JNsZZH2OU5I9YZRA3d1lslV4kdGNFMy X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 clxscore=1015 lowpriorityscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0 impostorscore=0 priorityscore=1501 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.158.5; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: 0 X-Spam_score: -0.1 X-Spam_bar: / X-Spam_report: (-0.1 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The gpa_inside_migration_helper_shared_area will be used to skip migrating RAM pages that are used by the migration helper at the target. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h index ebe4073bce..9a1027bdaf 100644 --- a/migration/confidential-ram.h +++ b/migration/confidential-ram.h @@ -8,6 +8,8 @@ #include "exec/cpu-common.h" #include "qemu-file.h" +bool gpa_inside_migration_helper_shared_area(ram_addr_t gpa); + void cgs_mh_init(void); void cgs_mh_cleanup(void); diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c index 053ecea1d4..30002448b9 100644 --- a/migration/confidential-ram.c +++ b/migration/confidential-ram.c @@ -68,6 +68,12 @@ static CGSMigHelperState cmhs = {0}; #define MH_SHARED_CMD_PARAMS_ADDR 0x820000 #define MH_SHARED_IO_PAGE_HDR_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x800) #define MH_SHARED_IO_PAGE_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x1000) +#define MH_SHARED_LAST_BYTE (MH_SHARED_CMD_PARAMS_ADDR + 0x1fff) + +bool gpa_inside_migration_helper_shared_area(ram_addr_t gpa) +{ + return gpa >= MH_SHARED_CMD_PARAMS_ADDR && gpa <= MH_SHARED_LAST_BYTE; +} void cgs_mh_init(void) { From patchwork Mon Aug 23 14:16:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 102ABC4338F for ; Mon, 23 Aug 2021 14:18:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 953AC613AC for ; Mon, 23 Aug 2021 14:18:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 953AC613AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44006 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAmk-00019y-PP for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:18:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41200) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl3-0006kY-Bf for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:03 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:50756 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000uB-7d for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:01 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE4INn102836; Mon, 23 Aug 2021 10:16:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=zrPVXWstP3zcqRv37D+U63kE6tg7L8Q5T1W8qwO8xSE=; b=AAbvFAfp6DaXJJA+C9RCUrvsaJ3meh2HbqZwKbWud1KfbJFQpazX+wfGQgk4TLOFhARx OjP2TQi3t4ciGwqrc2c3IHnmKXLokoRttp5SjTWT+fF9POQyR91YRRKLJDexkRifeh/E UHmdrXH/QkdNNiQDTZaX5oa7hCD4l+HkAF+I9t8aUyJwdLu+zU1hZlw5S49/bHFTQgJd j51/a2jjA6H3eUFXPYi0+JnlLKmG0QsicDa+5RqeXGY5APJJJ5iUWcuYGf6KZxT8duYd 6cChm5KISemKpc0Aj9djAqwL5nPGc8YdO7mpP+WDURo3/J/8VHpwXiUDKKo9f6A7HnlZ Wg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3akexysruv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:50 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE4N5D103555; Mon, 23 Aug 2021 10:16:49 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0b-001b2d01.pphosted.com with ESMTP id 3akexysruc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:49 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDDsG023036; Mon, 23 Aug 2021 14:16:49 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma02dal.us.ibm.com with ESMTP id 3ajs4bkk37-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:49 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGlro10224128 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:47 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 87289112061; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4D9A611206F; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 04/12] migration: Save confidential guest RAM using migration helper Date: Mon, 23 Aug 2021 10:16:28 -0400 Message-Id: <20210823141636.65975-5-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: PqFBxdy28VW5yN83PpUXrrTYvFmmFXuC X-Proofpoint-ORIG-GUID: _-T8wRSMkI9EDJ59C3Nvj9p9GdTuYS5z X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 clxscore=1015 lowpriorityscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0 impostorscore=0 priorityscore=1501 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.158.5; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When saving RAM pages of a confidential guest, check whether a page is encrypted. If it is, ask the in-guest migration helper to encrypt the page for transmission. This patch forces the use of in-guest migration handler instead of the PSP-based SEV migration; this is just a temporary example. TODO introduce migration parameter for choosing this migration mode. Signed-off-by: Dov Murik --- include/sysemu/sev.h | 1 + migration/ram.c | 109 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 101 insertions(+), 9 deletions(-) diff --git a/include/sysemu/sev.h b/include/sysemu/sev.h index d04890113c..ea52d2f41f 100644 --- a/include/sysemu/sev.h +++ b/include/sysemu/sev.h @@ -19,6 +19,7 @@ #define RAM_SAVE_ENCRYPTED_PAGE 0x1 #define RAM_SAVE_SHARED_REGIONS_LIST 0x2 +#define RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE 0x4 bool sev_enabled(void); int sev_kvm_init(ConfidentialGuestSupport *cgs, Error **errp); diff --git a/migration/ram.c b/migration/ram.c index 4eca90cceb..a1f89445d4 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -51,12 +51,14 @@ #include "migration/colo.h" #include "block.h" #include "sysemu/cpu-throttle.h" +#include "sysemu/kvm.h" #include "savevm.h" #include "qemu/iov.h" #include "multifd.h" #include "sysemu/runstate.h" #include "hw/boards.h" #include "exec/confidential-guest-support.h" +#include "confidential-ram.h" /* Defines RAM_SAVE_ENCRYPTED_PAGE and RAM_SAVE_SHARED_REGION_LIST */ #include "sysemu/sev.h" @@ -97,6 +99,13 @@ bool memcrypt_enabled(void) return ms->cgs->ready; } +static inline bool confidential_guest(void) +{ + MachineState *ms = MACHINE(qdev_get_machine()); + + return ms->cgs; +} + XBZRLECacheStats xbzrle_counters; /* struct contains XBZRLE cache and a static page @@ -2091,6 +2100,49 @@ static bool encrypted_test_list(RAMState *rs, RAMBlock *block, return ops->is_gfn_in_unshared_region(gfn); } +/** + * ram_save_mh_encrypted_page - use the guest migration handler to encrypt + * a page and send it to the stream. + * + * Return the number of pages written (=1). + */ +static int ram_save_mh_encrypted_page(RAMState *rs, PageSearchStatus *pss, + bool last_stage) +{ + int ret; + uint8_t *p; + RAMBlock *block = pss->block; + ram_addr_t offset = pss->page << TARGET_PAGE_BITS; + ram_addr_t gpa; + uint64_t bytes_sent; + + p = block->host + offset; + + /* Find the GPA of the page */ + if (!kvm_physical_memory_addr_from_host(kvm_state, p, &gpa)) { + error_report("%s failed to get gpa for offset %" PRIu64 " block %s", + __func__, offset, memory_region_name(block->mr)); + return -1; + } + + ram_counters.transferred += + save_page_header(rs, rs->f, block, + offset | RAM_SAVE_FLAG_ENCRYPTED_DATA); + + qemu_put_be32(rs->f, RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE); + ram_counters.transferred += sizeof(uint32_t); + + ret = cgs_mh_save_encrypted_page(rs->f, gpa, TARGET_PAGE_SIZE, &bytes_sent); + if (ret) { + return -1; + } + + ram_counters.transferred += bytes_sent; + ram_counters.normal++; + + return 1; +} + /** * ram_save_target_page: save one target page * @@ -2111,17 +2163,48 @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss, return res; } - /* - * If memory encryption is enabled then use memory encryption APIs - * to write the outgoing buffer to the wire. The encryption APIs - * will take care of accessing the guest memory and re-encrypt it - * for the transport purposes. - */ - if (memcrypt_enabled() && - encrypted_test_list(rs, pss->block, pss->page)) { - return ram_save_encrypted_page(rs, pss, last_stage); + if (confidential_guest()) { + /* + * TODO: We'd like to support two migration modes for SEV guests: + * PSP-based and guest-assisted. A possible solution is to add a new + * migration parameter ("use_guest_assistance") that will controlwhich + * mode should be used. + */ + bool guest_assisted_confidential_migration = true; + + if (guest_assisted_confidential_migration) { + /* + * If memory encryption is enabled then skip saving the data pages + * used by the migration handler. + */ + if (gpa_inside_migration_helper_shared_area(offset)) { + return 0; + } + + /* + * If memory encryption is enabled then use in-guest migration + * helper to write the outgoing buffer to the wire. The migration + * helper will take care of accessing the guest memory and + * re-encrypt it for the transport purposes. + */ + if (encrypted_test_list(rs, pss->block, pss->page)) { + return ram_save_mh_encrypted_page(rs, pss, last_stage); + } + } else { + /* + * If memory encryption is enabled then use memory encryption APIs + * to write the outgoing buffer to the wire. The encryption APIs + * will take care of accessing the guest memory and re-encrypt it + * for the transport purposes. + */ + if (memcrypt_enabled() && + encrypted_test_list(rs, pss->block, pss->page)) { + return ram_save_encrypted_page(rs, pss, last_stage); + } + } } + if (save_compress_page(rs, block, offset)) { return 1; } @@ -2959,6 +3042,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque) return -1; } + if (confidential_guest()) { + cgs_mh_init(); + } + /* migration has already setup the bitmap, reuse it. */ if (!migration_in_colo_state()) { if (ram_init_all(rsp) != 0) { @@ -3167,6 +3254,10 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } } + if (confidential_guest()) { + cgs_mh_cleanup(); + } + if (ret >= 0) { multifd_send_sync_main(rs->f); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); From patchwork Mon Aug 23 14:16:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 277F8C4338F for ; Mon, 23 Aug 2021 14:25:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 957C461372 for ; Mon, 23 Aug 2021 14:25:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 957C461372 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:39302 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAt7-0008RW-9M for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:25:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41212) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl4-0006kb-1b for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:03 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:58644) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000vE-7e for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:01 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE741a154591; Mon, 23 Aug 2021 10:16:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=XZDmJ0tDMaBAMxIIo406E/MbWwxOLYIAzQ9Ynr9WldE=; b=DbpyazF8DlruO6SnY162A+dYmsv4OSpIMXkJ4tlWSAiaGFv0wSCmGt2b+Hi1ZpaJ/1Aa vLdd7YJ1feMkkXQIjLvQsen5vztjG4rxMIamcCahKamZfoQlBA/gy7ko6tUfPcWG6+bU v/nwVw0HTMOVO8le7qFSeShcu8lGkU6D/hJ5zurrdH1sClSUOLAsod52azQ6mgO4rKV1 goKFd5yZ7U1o1Cuvi7wb0nmKwC65+wUqKDPsP2ERFznXpgBM2FysET6YsshPmcboA9jP /XXTX5muFWO46GidifPmTXadqiA3JuvZeYLh3X3utfDTbko6EdF43xqzO8j/ddapL6q4 nw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akf28sdua-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:50 -0400 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE71q1154229; Mon, 23 Aug 2021 10:16:50 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akf28sdty-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:50 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDDsH023036; Mon, 23 Aug 2021 14:16:49 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma02dal.us.ibm.com with ESMTP id 3ajs4bkk3b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:49 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGlA010224132 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:47 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C0AF311206D; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 91315112063; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 05/12] migration: Load confidential guest RAM using migration helper Date: Mon, 23 Aug 2021 10:16:29 -0400 Message-Id: <20210823141636.65975-6-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: fWTm2OU7Hy2TzwPtVzctzfcczSNoRpnd X-Proofpoint-ORIG-GUID: 2cDRxgUVOLoP4Yxg6WuUhDFdAZS0bFJz X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_02:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=952 clxscore=1011 lowpriorityscore=0 priorityscore=1501 phishscore=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 impostorscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.158.5; envelope-from=dovmurik@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When loading encrypted RAM pages of a confidential guest, ask the in-guest migration helper to decrypt the incoming page and place it correctly in the guest memory at the appropriate address. This way the page's plaintext content remains inaccessible to the host. Signed-off-by: Dov Murik --- migration/ram.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index a1f89445d4..2d5889f795 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1250,6 +1250,7 @@ static int load_encrypted_data(QEMUFile *f, uint8_t *ptr) cgs_class->memory_encryption_ops; int flag; + hwaddr gpa; flag = qemu_get_be32(f); @@ -1257,6 +1258,12 @@ static int load_encrypted_data(QEMUFile *f, uint8_t *ptr) return ops->load_incoming_page(f, ptr); } else if (flag == RAM_SAVE_SHARED_REGIONS_LIST) { return ops->load_incoming_shared_regions_list(f); + } else if (flag == RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE) { + if (!kvm_physical_memory_addr_from_host(kvm_state, ptr, &gpa)) { + error_report("%s: failed to get gpa for host ptr %p", __func__, ptr); + return -EINVAL; + } + return cgs_mh_load_encrypted_page(f, gpa); } else { error_report("unknown encrypted flag %x", flag); return 1; @@ -3728,6 +3735,10 @@ void colo_release_ram_cache(void) */ static int ram_load_setup(QEMUFile *f, void *opaque) { + if (confidential_guest()) { + cgs_mh_init(); + } + if (compress_threads_load_setup(f)) { return -1; } @@ -3754,6 +3765,10 @@ static int ram_load_cleanup(void *opaque) rb->receivedmap = NULL; } + if (confidential_guest()) { + cgs_mh_cleanup(); + } + return 0; } @@ -4024,6 +4039,7 @@ void colo_flush_ram_cache(void) static int ram_load_precopy(QEMUFile *f) { int flags = 0, ret = 0, invalid_flags = 0, len = 0, i = 0; + /* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised = postcopy_is_advised(); if (!migrate_use_compression()) { From patchwork Mon Aug 23 14:16:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5430C4338F for ; Mon, 23 Aug 2021 14:23:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7E6DF61372 for ; Mon, 23 Aug 2021 14:23:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7E6DF61372 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:60674 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIArR-0003vo-KW for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:23:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41174) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl1-0006kG-S6 for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:01 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:61082) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000v3-7D for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:59 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NEE97L002853; Mon, 23 Aug 2021 10:16:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=aHGICwWj8IRaXvb82rKRdwHvjUlCjM8CBobklKtLPa4=; b=KY+MSw8ecW4SodXR/bI10R9Kd4xAO2UcFYTLzo98/2vbpTg+zVgWTjN/b7a+Gue1jYeY I/+BEIunEk8TWLBBURl+1XO490M/Pa3Izmr8QiRhjAowwIPb8crzoPZNgqWJ3IFGZtnU vWvSwFaBknrEH0lSkj10RtM+/ac4vYB37D6SMSppBwf2wlNUk9eRDaYZBS7XwN8Jtxck XSUa2eaFCIfW74GnNnitLnWB7jnYnfkOzab0K5xUPOdmWYEgrOb/11u4Xb0kOgqEVcVN +epjcoBMDawKVnanIt5+29JTCCpOSPHLTGYuBHEsYiJyd3z62gshii5yV8O9ruUzOgyu RQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amd8m02pv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:51 -0400 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NEENWj003285; Mon, 23 Aug 2021 10:16:50 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amd8m02p9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:50 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDH26031979; Mon, 23 Aug 2021 14:16:49 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma01wdc.us.ibm.com with ESMTP id 3ajs4ax6ex-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:49 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGmnT44433896 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:48 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 22C19112065; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D3FB2112061; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:47 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 06/12] migration: Skip ROM, non-RAM, and vga.vram memory region during RAM migration Date: Mon, 23 Aug 2021 10:16:30 -0400 Message-Id: <20210823141636.65975-7-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: buiYW3XJDhPqLt12N0Gz-IrnQX-E0F8y X-Proofpoint-ORIG-GUID: KhsVE4bsTRjpmeBE2aLt_mveV86BiumE X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 impostorscore=0 lowpriorityscore=0 bulkscore=0 phishscore=0 mlxlogscore=998 spamscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 suspectscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.156.1; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Migrating these memory region hangs the in-guest migration handler. Signed-off-by: Dov Murik --- migration/ram.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 2d5889f795..f0df6780fb 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2086,7 +2086,9 @@ static bool encrypted_test_list(RAMState *rs, RAMBlock *block, unsigned long gfn; /* ROM devices contains the unencrypted data */ - if (memory_region_is_rom(block->mr)) { + if (memory_region_is_rom(block->mr) || + memory_region_is_romd(block->mr) || + !memory_region_is_ram(block->mr)) { return false; } @@ -2098,6 +2100,10 @@ static bool encrypted_test_list(RAMState *rs, RAMBlock *block, return false; } + if (!strcmp(memory_region_name(block->mr), "vga.vram")) { + return false; + } + /* * Translate page in ram_addr_t address space to GPA address * space using memory region. From patchwork Mon Aug 23 14:16:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7B4DC4320A for ; Mon, 23 Aug 2021 14:20:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6D085613AD for ; Mon, 23 Aug 2021 14:20:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6D085613AD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52190 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAoa-0006bG-Ge for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:20:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41120) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAkz-0006jo-O9 for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:58 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:29142) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000v9-87 for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:57 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE71de154214; Mon, 23 Aug 2021 10:16:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=aNbLUjXVHcRx5/8OM3kEp6CdowhfwRZeXlRqM2ngV9c=; b=cEqZmXqvxSweuxBbL88PbCDwM/7SxDQhc4rKXwcYzaTJvxbLmdR3kokL5/DUNntV6/Ci t4RRDNACnsLzzbdR7aOlXYxhBU3S+gkxubMcIj81Q2ZG6HV5EDDbhDvZkzAfSOaJX79j HOM/zMwuVlARGY3GHF3JDjZyiZlTXNhLZevi6nfSDB12PztT4m1ub/mZMSSq0HUuQr5U /bbxnXa6xaz8xPQcIZFFxXwmWiBak/31gHI3k+ZbtjSRLdJPGEgdMGJQ5Ymbg4/QeDQv UFFBXssDAVprAs1CAFMAoqDr8V4pgqeWPrBKWqRAIEpoHksEIG46muL4AfYlClpWr08u bw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akf28sdup-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:51 -0400 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE7JSV156367; Mon, 23 Aug 2021 10:16:50 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akf28sdu9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:50 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDDWp022984; Mon, 23 Aug 2021 14:16:50 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma02dal.us.ibm.com with ESMTP id 3ajs4bkk3q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:50 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGmgS44368236 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:48 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6DF05112063; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2BCFD112066; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 07/12] i386/kvm: Exclude mirror vcpu in kvm_synchronize_all_tsc Date: Mon, 23 Aug 2021 10:16:31 -0400 Message-Id: <20210823141636.65975-8-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: WvzrQgdnGeVDD9WRCkcbdFZNnWCZq03v X-Proofpoint-ORIG-GUID: 6JVcnO5Z22aIC4lGe3-MYmD8PwuYGtgF X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_02:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 priorityscore=1501 phishscore=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 impostorscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.158.5; envelope-from=dovmurik@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" If we don't exclude it there's a hang when stopping the VM during migration. Signed-off-by: Dov Murik --- target/i386/kvm/kvm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c index 6b20917fa5..04bbc89b48 100644 --- a/target/i386/kvm/kvm.c +++ b/target/i386/kvm/kvm.c @@ -241,7 +241,9 @@ void kvm_synchronize_all_tsc(void) if (kvm_enabled()) { CPU_FOREACH(cpu) { - run_on_cpu(cpu, do_kvm_synchronize_tsc, RUN_ON_CPU_NULL); + if (!cpu->mirror_vcpu) { + run_on_cpu(cpu, do_kvm_synchronize_tsc, RUN_ON_CPU_NULL); + } } } } From patchwork Mon Aug 23 14:16:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44D03C4338F for ; Mon, 23 Aug 2021 14:20:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C3237613AC for ; Mon, 23 Aug 2021 14:20:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C3237613AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52154 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAoZ-0006Zk-RZ for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:20:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41172) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl1-0006kB-Ot for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:59 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:29526) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000vC-82 for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:16:59 -0400 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE43Uh038923; Mon, 23 Aug 2021 10:16:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=o7ArqVFIJaqWyG1hnlql/TDzqNvAqFGSabuNt3jUimo=; b=ZUlWBfHnTngn1z0jGrqH4+XEvKRBQ/pDyAg6UeZM/WDlYAVGMritJ5KRruBvnYwp/J5K 2p+fuP/2LAuKpjQa9rNThQVts99UGg3HWU5oAtC1UQtmJWx6dWSR7fKxGg3+25UgRY3w BvdlMyh2fJUeCsuc4TOFFbDt8zDeOTn7y9FIR8d8HXAupkuyF1P+c8ZCqcLygpM47Zxi RTivXiIiH810DzBj8IkWTGUHFJhjWKwJBT2yMr4Aa+R3LjGTEMAWfmT3rwzJR7a7KrUw dOqvNbzufYmqFfIr4pYvEqNfxLDGkrKx8omD0KlgQoaLwplShvqt5cZBmOYJmvd9pa8i Fw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amb23cqe0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:51 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE4Owh041046; Mon, 23 Aug 2021 10:16:51 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amb23cqdk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:51 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDEPi004324; Mon, 23 Aug 2021 14:16:50 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma03dal.us.ibm.com with ESMTP id 3ajs4bkpsk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:50 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGmDd32112896 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:48 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AFF8111206D; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 773DA112065; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 08/12] migration: Allow resetting the mirror vcpu to the MH entry point Date: Mon, 23 Aug 2021 10:16:32 -0400 Message-Id: <20210823141636.65975-9-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: KLGBT5ldUhEL0BR4xrQz8qKGBhV2mrsc X-Proofpoint-ORIG-GUID: bWCoUe1krCpRm0x9ZAQil9B2J2xTY1hU X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 malwarescore=0 suspectscore=0 adultscore=0 phishscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.158.5; envelope-from=dovmurik@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: 0 X-Spam_score: -0.1 X-Spam_bar: / X-Spam_report: (-0.1 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add a function to reset the mirror vcpu so it'll start directly at the entry point of the migration handler. Note: In the patch below the GDT and EIP values are hard-coded to fit the OVMF migration handler entry point implementation we currently have. These values can be exposed in the OVMF GUID table and can be discovered from there instead of being hard-coded here. Signed-off-by: Dov Murik --- migration/confidential-ram.h | 2 + migration/confidential-ram.c | 112 +++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) diff --git a/migration/confidential-ram.h b/migration/confidential-ram.h index 9a1027bdaf..af046f95cc 100644 --- a/migration/confidential-ram.h +++ b/migration/confidential-ram.h @@ -18,4 +18,6 @@ int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size, int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa); +void cgs_mh_reset_mirror_vcpu(CPUState *s); + #endif diff --git a/migration/confidential-ram.c b/migration/confidential-ram.c index 30002448b9..6e41cba878 100644 --- a/migration/confidential-ram.c +++ b/migration/confidential-ram.c @@ -8,6 +8,8 @@ #include "io/channel.h" #include "qapi/error.h" #include "exec/memory.h" +#include "sysemu/kvm.h" +#include "kvm/kvm_i386.h" #include "trace.h" #include "confidential-ram.h" @@ -225,3 +227,113 @@ int cgs_mh_load_encrypted_page(QEMUFile *f, ram_addr_t dest_gpa) } return ret; } + +void cgs_mh_reset_mirror_vcpu(CPUState *s) +{ + X86CPU *cpu = X86_CPU(s); + CPUX86State *env = &cpu->env; + uint64_t xcr0; + int i; + + memset(env, 0, offsetof(CPUX86State, end_reset_fields)); + + env->old_exception = -1; + + /* init to reset state */ + + env->hflags2 |= HF2_GIF_MASK; + env->hflags &= ~HF_GUEST_MASK; + env->hflags |= HF_CS32_MASK | HF_SS32_MASK | HF_PE_MASK | HF_MP_MASK; + + cpu_x86_update_cr0(env, 0x00010033); + env->a20_mask = ~0x0; + env->smbase = 0x30000; + env->msr_smi_count = 0; + + /* The GDT is hard-coded to the one setup by OVMF */ + env->gdt.base = 0x823600; + env->gdt.limit = 0x0047; + env->ldt.limit = 0xffff; + env->ldt.flags = DESC_P_MASK | (2 << DESC_TYPE_SHIFT); + env->tr.limit = 0xffff; + env->tr.flags = DESC_P_MASK | (11 << DESC_TYPE_SHIFT); + + cpu_x86_load_seg_cache(env, R_CS, 0x38, 0, 0xffffffff, + DESC_B_MASK | DESC_P_MASK | DESC_S_MASK | + DESC_CS_MASK | DESC_R_MASK | DESC_A_MASK); + cpu_x86_load_seg_cache(env, R_DS, 0x30, 0, 0xffffffff, + DESC_B_MASK | DESC_P_MASK | DESC_S_MASK | + DESC_W_MASK | DESC_A_MASK); + cpu_x86_load_seg_cache(env, R_ES, 0x30, 0, 0xffffffff, + DESC_B_MASK | DESC_P_MASK | DESC_S_MASK | + DESC_W_MASK | DESC_A_MASK); + cpu_x86_load_seg_cache(env, R_SS, 0x30, 0, 0xffffffff, + DESC_B_MASK | DESC_P_MASK | DESC_S_MASK | + DESC_W_MASK | DESC_A_MASK); + cpu_x86_load_seg_cache(env, R_FS, 0x30, 0, 0xffffffff, + DESC_B_MASK | DESC_P_MASK | DESC_S_MASK | + DESC_W_MASK | DESC_A_MASK); + cpu_x86_load_seg_cache(env, R_GS, 0x30, 0, 0xffffffff, + DESC_B_MASK | DESC_P_MASK | DESC_S_MASK | + DESC_W_MASK | DESC_A_MASK); + + /* The EIP is hard-coded to the OVMF migration handler entry point */ + env->eip = 0x823000; + /* env->regs[R_EDX] = env->cpuid_version; */ + + env->eflags = 0x2; + + /* FPU init */ + for (i = 0; i < 8; i++) { + env->fptags[i] = 1; + } + cpu_set_fpuc(env, 0x37f); + + env->mxcsr = 0x1f80; + /* All units are in INIT state. */ + env->xstate_bv = 0; + + env->pat = 0x0007040600070406ULL; + env->msr_ia32_misc_enable = MSR_IA32_MISC_ENABLE_DEFAULT; + if (env->features[FEAT_1_ECX] & CPUID_EXT_MONITOR) { + env->msr_ia32_misc_enable |= MSR_IA32_MISC_ENABLE_MWAIT; + } + + memset(env->dr, 0, sizeof(env->dr)); + env->dr[6] = DR6_FIXED_1; + env->dr[7] = DR7_FIXED_1; + cpu_breakpoint_remove_all(s, BP_CPU); + cpu_watchpoint_remove_all(s, BP_CPU); + + xcr0 = XSTATE_FP_MASK; + env->xcr0 = xcr0; + cpu_x86_update_cr4(env, 0x00000668); + + /* + * SDM 11.11.5 requires: + * - IA32_MTRR_DEF_TYPE MSR.E = 0 + * - IA32_MTRR_PHYSMASKn.V = 0 + * All other bits are undefined. For simplification, zero it all. + */ + env->mtrr_deftype = 0; + memset(env->mtrr_var, 0, sizeof(env->mtrr_var)); + memset(env->mtrr_fixed, 0, sizeof(env->mtrr_fixed)); + + env->interrupt_injected = -1; + env->exception_nr = -1; + env->exception_pending = 0; + env->exception_injected = 0; + env->exception_has_payload = false; + env->exception_payload = 0; + env->nmi_injected = false; +#if !defined(CONFIG_USER_ONLY) + /* We hard-wire the BSP to the first CPU. */ + apic_designate_bsp(cpu->apic_state, s->cpu_index == 0); + + s->halted = !cpu_is_bsp(cpu); + + if (kvm_enabled()) { + kvm_arch_reset_vcpu(cpu); + } +#endif +} From patchwork Mon Aug 23 14:16:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5A52C4338F for ; Mon, 23 Aug 2021 14:24:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A1B9F61372 for ; Mon, 23 Aug 2021 14:24:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A1B9F61372 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34956 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAs9-0005aQ-Qt for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:24:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41216) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl5-0006ku-ER for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:05 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:57972) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl0-0000y6-FZ for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:03 -0400 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE5oVp148177; Mon, 23 Aug 2021 10:16:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=VpkCRa4LfOTodmoNQonwlPc/wVYK2Poeu+ab/W+CU1g=; b=RUzHZPw2avsM2KLRO4aa1DbYIWXdXGX5bRPBWxYehdUBnGscP4khYH6aGm2OulDwInOF MAEKmSCbIewq4ffxIMvKX3tTBBz4uruZay1YlSmUft+Patl6Dv2Rvp2fxFe3S5q1w/zJ 4G1FS86mdDZ/H6BfR/fYtkZp+IqEx5PdgxGmCu3Xkuckuu9D5g6Q1pJG8V/HQrZ0B2sH EHYw23vSjAVgdeWOLAcZHugkTHz3XF8+wyrcdWImDNCJ54W6eFuD+ANF6xMFQZp0ugLU 45BgWt1MlkytxzQygKihUMgGqqJD4cQ6zLivd885uDAABcEJzHhnI6MI7pQ674BUSswT 1w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akejst3fj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:54 -0400 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE60Ac148998; Mon, 23 Aug 2021 10:16:54 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akejst3eb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:54 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDD86022985; Mon, 23 Aug 2021 14:16:50 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma02dal.us.ibm.com with ESMTP id 3ajs4bkk47-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:50 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGn0P48038236 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:49 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F31DB112061; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B945711206E; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 09/12] migration: Add QMP command start-migration-handler Date: Mon, 23 Aug 2021 10:16:33 -0400 Message-Id: <20210823141636.65975-10-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: x-dwwu1DlM8t5Onq35vUxUNh1dt0qTiF X-Proofpoint-ORIG-GUID: kIMD-cbRgLUCFfOpaY3KdW_jGV9AN7Yn X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 mlxlogscore=999 impostorscore=0 adultscore=0 mlxscore=0 spamscore=0 priorityscore=1501 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.156.1; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The start-migration-handler QMP command starts the mirror vcpu directly at the migration handler entry point. This is a temporary workaround to start-up (resume) the mirror vcpu which runs the in-guest migration handler (both on the source and the target). A proper solution would be to start it automatically when the 'migrate' and 'migrate-incoming' QMP commands are executed. Signed-off-by: Dov Murik --- qapi/migration.json | 12 ++++++++++++ migration/migration.c | 12 ++++++++++++ 2 files changed, 24 insertions(+) diff --git a/qapi/migration.json b/qapi/migration.json index 69c615ec4d..baff3c6bf7 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1504,6 +1504,18 @@ ## { 'command': 'migrate-incoming', 'data': {'uri': 'str' } } +## +# @start-migration-handler: +# +# Start the mirror vcpu which runs the in-guest migration handler. +# +# Returns: nothing on success +# +# Since: 6.2 +# +## +{ 'command': 'start-migration-handler' } + ## # @xen-save-devices-state: # diff --git a/migration/migration.c b/migration/migration.c index c9bc33fb10..a9f3a79e4f 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -60,6 +60,7 @@ #include "qemu/yank.h" #include "sysemu/cpus.h" #include "yank_functions.h" +#include "confidential-ram.h" #define MAX_THROTTLE (128 << 20) /* Migration transfer speed throttling */ @@ -2161,6 +2162,17 @@ void qmp_migrate_incoming(const char *uri, Error **errp) once = false; } +void qmp_start_migration_handler(Error **errp) +{ + CPUState *cpu; + CPU_FOREACH(cpu) { + if (cpu->mirror_vcpu) { + cgs_mh_reset_mirror_vcpu(cpu); + cpu_resume(cpu); + } + } +} + void qmp_migrate_recover(const char *uri, Error **errp) { MigrationIncomingState *mis = migration_incoming_get_current(); From patchwork Mon Aug 23 14:16:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 309F1C4338F for ; Mon, 23 Aug 2021 14:18:43 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A080E613AC for ; Mon, 23 Aug 2021 14:18:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A080E613AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:43674 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAmf-0000xD-I5 for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:18:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41218) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAl5-0006l4-TC for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:05 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:23142) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAky-0000wW-7e for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:17:03 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE4OPZ052006; Mon, 23 Aug 2021 10:16:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=4dl7IEek6/lwWZ4YKlOYcAcEW3DcH1O+/qNqlMJptvU=; b=PI7JOkqgVmXA9urbwl9nCaIdJVqNNiMH5vLVVOZsK+eEbOsHxPQuSGnl/GX6CWMupIXL tIFhZ1w1xWIQo4YzICkwAb1H8ElKcMj/yqYO3Wh7PxR9tIW5IGFtNXeS3IGx32Ta8+WF Lddud4JP4wqNRd9clC+92V2aKVuYt+YH7c5kI72rHys/xElvhFj25we6Ttjhni4RdZDy s75Tv+kWXuvnD7N8233mG8C5fC55Huj/dPsSRgbl8I3HS/6IfS9fvZu5mhegcNTyD7Dy AKsImfk9UsFhn+ku5KfUVH7cZDd7SKqw9G6aoK/2EtGPcA3HwFm4We/7+akkRjj1J/Ff Nw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3am1eva5qp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:52 -0400 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE4O8w052073; Mon, 23 Aug 2021 10:16:51 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3am1eva5q7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:16:51 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDEtd004325; Mon, 23 Aug 2021 14:16:50 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma03dal.us.ibm.com with ESMTP id 3ajs4bkpst-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:16:50 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGntG40042790 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:49 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 41467112066; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 082CD112065; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:48 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 10/12] migration: Add start-migrate-incoming QMP command Date: Mon, 23 Aug 2021 10:16:34 -0400 Message-Id: <20210823141636.65975-11-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: MlpCwD9BhzFjmmZocjtzRf3o7kAItls6 X-Proofpoint-ORIG-GUID: HeB3_4r5IbIr1ffqrd2XsmmZbmAVkFGS X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 mlxscore=0 mlxlogscore=999 spamscore=0 adultscore=0 lowpriorityscore=0 clxscore=1015 priorityscore=1501 phishscore=0 suspectscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.156.1; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This command forces a running VM into a migrate-incoming state. When using guest-assisted migration (for confidential guests), the target must be started so that its memory has the necessary code for the migration helper. After it is ready we can start receiving the incoming migration connection. Signed-off-by: Dov Murik --- qapi/migration.json | 26 ++++++++++++++++++++++++++ migration/migration.c | 17 +++++++++++++++++ 2 files changed, 43 insertions(+) diff --git a/qapi/migration.json b/qapi/migration.json index baff3c6bf7..da47b8534f 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1516,6 +1516,32 @@ ## { 'command': 'start-migration-handler' } +## +# @start-migrate-incoming: +# +# Force start an incoming migration even in a running VM. This is used by the +# target VM in guest-assisted migration of a confidential guest. +# +# @uri: The Uniform Resource Identifier identifying the source or +# address to listen on +# +# Returns: nothing on success +# +# Since: 6.0 +# +# Notes: +# +# The uri format is the same as the -incoming command-line option. +# +# Example: +# +# -> { "execute": "start-migrate-incoming", +# "arguments": { "uri": "tcp::4446" } } +# <- { "return": {} } +# +## +{ 'command': 'start-migrate-incoming', 'data': {'uri': 'str' } } + ## # @xen-save-devices-state: # diff --git a/migration/migration.c b/migration/migration.c index a9f3a79e4f..0b9ab3decb 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2173,6 +2173,23 @@ void qmp_start_migration_handler(Error **errp) } } +void qmp_start_migrate_incoming(const char *uri, Error **errp) +{ + Error *local_err = NULL; + + if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) { + return; + } + + vm_stop(RUN_STATE_PAUSED); + qemu_start_incoming_migration(uri, &local_err); + + if (local_err) { + yank_unregister_instance(MIGRATION_YANK_INSTANCE); + error_propagate(errp, local_err); + } +} + void qmp_migrate_recover(const char *uri, Error **errp) { MigrationIncomingState *mis = migration_incoming_get_current(); From patchwork Mon Aug 23 14:16:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85ADFC4320A for ; Mon, 23 Aug 2021 14:20:45 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4E78E61372 for ; Mon, 23 Aug 2021 14:20:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4E78E61372 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAoe-0006nv-B2 for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:20:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41524) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAmr-0002Wd-8A for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:18:53 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:64768) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAmp-000287-Mt for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:18:53 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NEE7UJ002784; Mon, 23 Aug 2021 10:18:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=LrpnHHtZYVIjXDsnZhxNjXR8w1wAeZd63BLfZhGGStw=; b=o88QyCdo0OOInQwWb2ldd5vv/ea4bWQziqxoVu+3jRGIKDT7RR6bSRBLbI1mUvsgr382 hKKxz3xNiSq4FhxFyT0PRGYyy6Rk5fNynngh+nJWjJW1N4ZldCmhKkSOMmSXTXLAsCMV CxTZE7L1A5TrdxYnmyzA8YX1qWlgRT2xcFfLqjdETjHjtW+eEHy0v0H2Rqyl0O8k4hEd 6QXA8csglUdTK95qdaBuqiIQAp1Ir+AsN5E7vHmNawwGuchi+x/PwquhQqMHHKBmZMAM TJZdwM8NvJ18XjS2LTKezSe1aMQZD5i7FnsKmnDdY6AURP4acuN2TWbwf4sOBMJ5H/9r lQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amd8m051v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:18:48 -0400 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NEEpuI003937; Mon, 23 Aug 2021 10:18:48 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 3amd8m0515-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:18:48 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDMoI016503; Mon, 23 Aug 2021 14:18:47 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma01dal.us.ibm.com with ESMTP id 3ajs4c3nv7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:18:45 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGnOA54395300 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:49 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8D00A112065; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4A10A112061; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 11/12] hw/isa/lpc_ich9: Allow updating an already-running VM Date: Mon, 23 Aug 2021 10:16:35 -0400 Message-Id: <20210823141636.65975-12-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: jlybOgFEszvuQQWCFT2rD--gaKaeukOC X-Proofpoint-ORIG-GUID: GGE4ZUiLix-pTJhA7df1coNakN32NcZi X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 impostorscore=0 lowpriorityscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 spamscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 suspectscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.156.1; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The post_load function crashed when we were loading the device state in to an already-running guest. This was because an existing memory region as not deleted in ich9_lpc_rcba_update. Signed-off-by: Dov Murik --- hw/isa/lpc_ich9.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/isa/lpc_ich9.c b/hw/isa/lpc_ich9.c index 5f9de0239c..ea07709c14 100644 --- a/hw/isa/lpc_ich9.c +++ b/hw/isa/lpc_ich9.c @@ -527,9 +527,10 @@ ich9_lpc_pmcon_update(ICH9LPCState *lpc) static int ich9_lpc_post_load(void *opaque, int version_id) { ICH9LPCState *lpc = opaque; + uint32_t rcba_old = pci_get_long(lpc->d.config + ICH9_LPC_RCBA); ich9_lpc_pmbase_sci_update(lpc); - ich9_lpc_rcba_update(lpc, 0 /* disabled ICH9_LPC_RCBA_EN */); + ich9_lpc_rcba_update(lpc, rcba_old); ich9_lpc_pmcon_update(lpc); return 0; } From patchwork Mon Aug 23 14:16:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dov Murik X-Patchwork-Id: 12452945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98DC5C4338F for ; Mon, 23 Aug 2021 14:29:22 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07631613AD for ; Mon, 23 Aug 2021 14:29:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 07631613AD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:50278 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mIAwz-0007Nv-4Z for qemu-devel@archiver.kernel.org; Mon, 23 Aug 2021 10:29:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAmh-00026v-B7 for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:18:43 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:34842) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mIAmd-000206-KY for qemu-devel@nongnu.org; Mon, 23 Aug 2021 10:18:43 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17NE3HJu063370; Mon, 23 Aug 2021 10:18:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=IQvFrSW6ezLBSZWc/lcIlywfoZzdMo5CWk+jmEIiXSQ=; b=ZefstGhDJAwN7iAgGrap7dg4Mj2oui2RqFze3n8GRpfs2cfGKFyS1KCFFSuyivKUDp8A 9Wk5SbUjVPCnKUYgmK64IkB6/KAaoPRjv0gQ9cw6iT8WQSdZ0mXvGS5td56Nhal3PhlK Sf9pyrWFUTkmhUvAkpnm+EL9R1zSGcX9/eVXIXSCleIMihQAN1AyV+fPQ/Uabi9hIUyS fBLt1OI0CaJhIQEqmwCFjORep/01gamc0wlfxE2DW1LVZIJ3oBdJtNfnDXZeZ1XjhDIV LcxDLyEDOawKQF2MURG16/ySk9TH1i59U8w1h/ol7OGpBUR3v+SPddXPSW/WZ0HaXxBP iQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akehva3nh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:18:37 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 17NE3LbC066504; Mon, 23 Aug 2021 10:18:36 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 3akehva3n2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 10:18:36 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 17NEDMvq016512; Mon, 23 Aug 2021 14:18:35 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma01dal.us.ibm.com with ESMTP id 3ajs4c3nvc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 23 Aug 2021 14:18:35 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 17NEGnCW48759262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Aug 2021 14:16:50 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D808F112061; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 96378112066; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) Received: from amdrome1.watson.ibm.com (unknown [9.2.130.16]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 23 Aug 2021 14:16:49 +0000 (GMT) From: Dov Murik To: qemu-devel@nongnu.org Subject: [RFC PATCH v2 12/12] docs: Add confidential guest live migration documentation Date: Mon, 23 Aug 2021 10:16:36 -0400 Message-Id: <20210823141636.65975-13-dovmurik@linux.ibm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210823141636.65975-1-dovmurik@linux.ibm.com> References: <20210823141636.65975-1-dovmurik@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: E6vM-2vzL25CmGqEFho_RfDJXf2qp_5n X-Proofpoint-GUID: gv41NxLN0DU3Nd8J2_pCcGAHqhnCinlp X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-08-23_03:2021-08-23, 2021-08-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 malwarescore=0 mlxlogscore=999 lowpriorityscore=0 clxscore=1015 phishscore=0 suspectscore=0 spamscore=0 impostorscore=0 priorityscore=1501 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108230098 Received-SPF: pass client-ip=148.163.156.1; envelope-from=dovmurik@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tom Lendacky , Ashish Kalra , Brijesh Singh , "Michael S. Tsirkin" , Steve Rutherford , James Bottomley , Juan Quintela , "Dr. David Alan Gilbert" , Dov Murik , Hubertus Franke , Tobin Feldman-Fitzthum , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The new page is linked from the main index, otherwise sphinx complains that "document isn't included in any toctree"; I assume there would be a better place for it in the documentation tree. Signed-off-by: Dov Murik --- docs/confidential-guest-live-migration.rst | 145 +++++++++++++++++++++ docs/confidential-guest-support.txt | 5 + docs/index.rst | 1 + 3 files changed, 151 insertions(+) create mode 100644 docs/confidential-guest-live-migration.rst diff --git a/docs/confidential-guest-live-migration.rst b/docs/confidential-guest-live-migration.rst new file mode 100644 index 0000000000..65b6111ff1 --- /dev/null +++ b/docs/confidential-guest-live-migration.rst @@ -0,0 +1,145 @@ +================================= +Confidential Guest Live Migration +================================= + +When migrating regular QEMU guests, QEMU reads the guest's RAM and sends it +over to the migration target host, where QEMU there writes it into the target +guest's RAM and starts the VM. This mechanism doesn't work when the guest +memory is encrypted or QEMU is prevented from reading it in another way. + +In order to support live migration in such scenarios, QEMU relies on an +in-guest migration helper which can securely extract RAM content from the +guest in order to send it to the target. The migration helper is implemented as +part of the VM's firmware in OVMF. + + +Migration flow +============== + +Source VM +--------- + +The source VM is started with an extra mirror vcpu which is not part of the +main VM but shared the same memory mapping. This vcpu is started at a special +entry point which runs a dedicated migration helper; the migration helper +simply waits for commands from QEMU. When migration starts using the +``migrate`` command, QEMU starts saving the state of the different devices. +When it reaches saving RAM pages, it'll check for each page whether it is +encrypted or not; for encrypted pages, it'll send a command to the migration +helper to extract the given page. The migration helper receives this command, +reads the page content, encrypts it with a transport key, and returns the +newly-encrypted page to QEMU. QEMU saves those pages to the outgoing migration +stream using the ``RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE`` subtype of the +``RAM_SAVE_FLAG_ENCRYPTED_DATA`` page flag. + +When QEMU reaches the last stage of RAM migration, it stops the source VM to +avoid dirtying the last pages of RAM. However, the mirror vcpu must be kept +running so the migration helper can still extract pages from the guest memory. + +Target VM +--------- + +Usually QEMU migration target VMs are started with the ``-incoming`` +command-line option which starts the VM paused. However, in order to migrate +confidential guests we must have the migration helper running inside the guest; +in such a case, we start the target with a special ``-fw_cfg`` value that tells +OVMF to load the migration handler code into memory and then enter a CPU dead +loop. After this short "boot" completes, QEMU can switch to the "migration +incoming" mode; we do that with the new ``start-migrate-incoming`` QMP command +that makes the target VM listen for incoming migration connections. + +QEMU will load the state of VM devices as it arrives from the incoming +migration stream. When it encounters a RAM page with the +``RAM_SAVE_FLAG_ENCRYPTED_DATA`` flag and the +``RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE`` subtype, it will send its +transport-encrypted content and guest physical address to the migration helper. +The migration helper running inside the guest will decrypt the page using the +transport key and place the content in memory (again, that memory page is not +accessible to host due to the confidential guest properties; for example, in SEV +it is hardware-encrypted with a VM-specific key). + + +Usage +===== + +In order to start the source and target VMs with mirror vCPUs, the +``mirrorvcpus=`` option must be passed to ``-smp`` . For example:: + + # ${QEMU} -smp 5,mirrorvcpus=1 ... + +This command starts a VM with 5 vcpus of which 4 are main vcpus (available for +the guest OS) and 1 is mirror vcpu. + +Moreover, in both the source and target we need to instruct OVMF to start the +migration helper running in the auxiliary vcpu. This is achieved using the +following command-line option:: + + # ${QEMU} -fw_cfg name=opt/ovmf/PcdSevIsMigrationHelper,string=0 ... + +In the target VM we need to add another ``-fw_cfg`` entry to instruct OVMF to +start only the migration helepr, which will wait for incoming pages (the target +cannot be started with ``-incoming`` because that option completely pauses the +VM, not allowing the migration helper to run). Because the migration helper must +be running when the incoming RAM pages are received, starting the target VM with +the ``-incoming`` option doesn't work (with that option, the VM doesn't start +executing). Instead, start the target VM without ``-incoming`` but with the +following option:: + + # ${QEMU} -fw_cfg name=opt/ovmf/PcdSevIsMigrationTarget,string=1 ... + +After the VM boots into the migration helper, we instruct QEMU to listen for +incoming migration connections by sending the following QMP command:: + + { "execute": "start-migrate-incoming", + "arguments": { "uri": "tcp:0.0.0.0:6666" } } + +Now that the target is ready, we instruct the source VM to start migrating its +state using the regular ``migrate`` QMP command, supplying the target VMs +listening address:: + + { "execute": "migrate", + "arguments": { "uri": "tcp:192.168.111.222:6666" } } + + +Implementation details +====================== + +Migration helper <-> QEMU communication +--------------------------------------- + +The migration helper is running inside the guest (implemented as part of OVMF). +QEMU communicates with it using a mailbox protocol over two shared (unencrypted) +4K RAM pages. + +The first page contains a ``SevMigHelperCmdParams`` struct at offset 0x0 +(``cmd_params``) and a ``MigrationHelperHeader`` struct at offset 0x800 +(``io_hdr``). The second page (``io_page``) is dedicated for encrypted page +content. + +In order to save a confidential RAM page, QEMU will fill the ``cmd_params`` +struct to indicate the SEV_MIG_HELPER_CMD_ENCRYPT command and the requested gpa +(guest physical address), and then set the ``go`` field to 1. Meanwhile the +migration helper waits for the ``go`` field to become non-zero; after it notices +``go`` is 1 it'll read the gpa, read the content of the relevant page from the +guest's memory, encrypt it with the transport key, and store the +transport-encrypted page in the the ``io_page``. Additional envelope data like +encryption IV and other fields are stored in ``io_hdr``. After the migration is +done writing to ``io_page`` and ``io_hdr``, it sets the ``done`` field to 1. At +this point QEMU notices that the migration helper is done and can continue its +part, which is saving the header and page to the outgoing migration stream. + +Similar process is used when loading a confidential RAM from the incoming +migration stream. QEMU reads the header and the encrypted page from the stream, +and copies them into the shared areas ``io_hdr`` and ``io_page`` respectably. +It then fills the ``cmd_params`` struct to indicate the +SEV_MIG_HELPER_CMD_DECRYPT command and the gpa, and sets ``go`` to 1. The +migration helper will notice the command, will decrypt the page using the +transport key and will place the decrypted content in the requetsed gpa, and set +``done`` to 1 to allow QEMU to continue processing the next item in the incoming +migration stream. + +Shared pages address discovery +------------------------------ +In the current implementation the address of the two shared pages is hard-coded +in both OVMF and QEMU. We plan for OVMF to expose this address via its GUIDed +table and let QEMU discover it using ``pc_system_ovmf_table_find()``. diff --git a/docs/confidential-guest-support.txt b/docs/confidential-guest-support.txt index 71d07ba57a..bed1601fbb 100644 --- a/docs/confidential-guest-support.txt +++ b/docs/confidential-guest-support.txt @@ -47,3 +47,8 @@ s390x Protected Virtualization (PV) docs/system/s390x/protvirt.rst Other mechanisms may be supported in future. + +Live migration support +---------------------- +Details regarding confidential guest live migration are in: + docs/confidential-guest-live-migration.rst diff --git a/docs/index.rst b/docs/index.rst index 5f7eaaa632..f2015de814 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -17,3 +17,4 @@ Welcome to QEMU's documentation! interop/index specs/index devel/index + confidential-guest-live-migration