From patchwork Fri Feb 14 14:13:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DB7D8C021A6 for ; Fri, 14 Feb 2025 14:16:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSm-0004LM-Pg; Fri, 14 Feb 2025 09:14:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSl-0004Jn-9c for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:39 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSi-0004Qe-SB for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:39 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtTtK028522; Fri, 14 Feb 2025 14:14:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=9oVLojoVLLhWzzkSHfDhB/9yMWQHtFgvTEYgp18yT7Q=; b= mUIcq/CuWThR1fxtFfHy8LnsmHB/X/q9C5+vQgL6C+LtyYpNiMQ/nCaZeMGvrky8 PwyQCnMzQ+ObL/aWiN3J2ph2KG6INwTC9i/Letsydcs5bFDjfpBXcnaQw3znnJSG EMTBpTNapnLBjlkEIUUOHJ1ZohMPChSyN8kWcFzL6zHlPbcXUA7KKtx6XBFWvTtG yNShQ8GTuDXchyylgfs7f5tmyFTvFKaPdMlzdEP5FZSz+6IflndU7DlcXEKGYHRN yMoazLc+amlHzbmuBE+LMN+eBv8V8GiPXXnKtrZdmMNXsiD62Fxl39aEAJWhalwv Ep/olYxoSeTmIDa19STPLQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnmk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:31 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECfZO1025148; Fri, 14 Feb 2025 14:14:31 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg3j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:31 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPF006920; Fri, 14 Feb 2025 14:14:30 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-2; Fri, 14 Feb 2025 14:14:30 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 01/45] MAINTAINERS: Add reviewer for CPR Date: Fri, 14 Feb 2025 06:13:43 -0800 Message-Id: <1739542467-226739-2-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: aDA1EmxwxYiRajvdqOfjn-5NlwB3Kkm9 X-Proofpoint-ORIG-GUID: aDA1EmxwxYiRajvdqOfjn-5NlwB3Kkm9 Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org CPR is integrated with live migration, and has the same maintainers. But, add a CPR section to add a reviewer. Signed-off-by: Steve Sistare --- MAINTAINERS | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 3848d37..2f9a6da 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2951,6 +2951,17 @@ F: include/qemu/co-shared-resource.h T: git https://gitlab.com/jsnow/qemu.git jobs T: git https://gitlab.com/vsementsov/qemu.git block +CheckPoint and Restart (CPR) +M: Peter Xu +M: Fabiano Rosas +R: Steve Sistare +S: Supported +F: hw/vfio/cpr* +F: include/migration/cpr.h +F: migration/cpr* +F: tests/qtest/migration/cpr* +F: docs/devel/migration/CPR.rst + Compute Express Link M: Jonathan Cameron R: Fan Ni From patchwork Fri Feb 14 14:13:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F18E3C02198 for ; Fri, 14 Feb 2025 14:15:23 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSm-0004Kp-OD; Fri, 14 Feb 2025 09:14:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSk-0004IU-7a for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:38 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSi-0004Qk-0g for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:37 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtS7f028514; Fri, 14 Feb 2025 14:14:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=mEVPWwF+UfGeyFbKqf1xWpPBvuAXnhWIm3FJAkaKpMI=; b= kFDbSWYruligitQdjUU2emtdLh9rtr5G69DwQoTUNDMmLbeMWQRHYP+dNCuQyqx6 xila1niW4Out3nzVpo4LlnOq7Q8iR/9cuIOT1YE74UHX2R+ZeKqJ6jke0Q74QeR4 /f+iw885AwryjNzoZkJQTwKB8CeeHS6iIOqHCS2avb+60++feGZG2fb/h8ZMlasC rSxzMIY5dmwK6kOUNj7W2S+S3BcQc+/aHIuiE4oMT8jwiley8PG11F0GtYXPTipS gLuyUkthP2/bzG4IK/2Z5vy+F5CFJyBsp3fsLliRZESms6B29BP1hWclHR6XBpyA R9WqUWjZD90COfjG9pXMaw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnmm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:33 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EC9vVV025116; Fri, 14 Feb 2025 14:14:32 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg4c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:32 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPH006920; Fri, 14 Feb 2025 14:14:32 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-3; Fri, 14 Feb 2025 14:14:31 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 02/45] migration: cpr helpers Date: Fri, 14 Feb 2025 06:13:44 -0800 Message-Id: <1739542467-226739-3-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: ynDWCEE5sMpbu2o6pTGYT7jDRpIrL9QG X-Proofpoint-ORIG-GUID: ynDWCEE5sMpbu2o6pTGYT7jDRpIrL9QG Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add cpr_needed_for_reuse, cpr_resave_fd helpers, cpr_is_incoming, and cpr_open_fd, for use when adding cpr support for vfio and iommufd. Signed-off-by: Steve Sistare --- include/migration/cpr.h | 6 ++++++ migration/cpr.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/include/migration/cpr.h b/include/migration/cpr.h index 3a6deb7..6ad04d4 100644 --- a/include/migration/cpr.h +++ b/include/migration/cpr.h @@ -18,15 +18,21 @@ void cpr_save_fd(const char *name, int id, int fd); void cpr_delete_fd(const char *name, int id); int cpr_find_fd(const char *name, int id); +void cpr_resave_fd(const char *name, int id, int fd); +int cpr_open_fd(const char *path, int flags, const char *name, int id, + bool *reused, Error **errp); MigMode cpr_get_incoming_mode(void); void cpr_set_incoming_mode(MigMode mode); +bool cpr_is_incoming(void); int cpr_state_save(MigrationChannel *channel, Error **errp); int cpr_state_load(MigrationChannel *channel, Error **errp); void cpr_state_close(void); struct QIOChannel *cpr_state_ioc(void); +bool cpr_needed_for_reuse(void *opaque); + QEMUFile *cpr_transfer_output(MigrationChannel *channel, Error **errp); QEMUFile *cpr_transfer_input(MigrationChannel *channel, Error **errp); diff --git a/migration/cpr.c b/migration/cpr.c index 584b0b9..12c489b 100644 --- a/migration/cpr.c +++ b/migration/cpr.c @@ -95,6 +95,39 @@ int cpr_find_fd(const char *name, int id) trace_cpr_find_fd(name, id, fd); return fd; } + +void cpr_resave_fd(const char *name, int id, int fd) +{ + CprFd *elem = find_fd(&cpr_state.fds, name, id); + int old_fd = elem ? elem->fd : -1; + + if (old_fd < 0) { + cpr_save_fd(name, id, fd); + } else if (old_fd != fd) { + error_setg(&error_fatal, + "internal error: cpr fd '%s' id %d value %d " + "already saved with a different value %d", + name, id, fd, old_fd); + } +} + +int cpr_open_fd(const char *path, int flags, const char *name, int id, + bool *reused, Error **errp) +{ + int fd = cpr_find_fd(name, id); + + if (reused) { + *reused = (fd >= 0); + } + if (fd < 0) { + fd = qemu_open(path, flags, errp); + if (fd >= 0) { + cpr_save_fd(name, id, fd); + } + } + return fd; +} + /*************************************************************************/ #define CPR_STATE "CprState" @@ -128,6 +161,11 @@ void cpr_set_incoming_mode(MigMode mode) incoming_mode = mode; } +bool cpr_is_incoming(void) +{ + return incoming_mode != MIG_MODE_NONE; +} + int cpr_state_save(MigrationChannel *channel, Error **errp) { int ret; @@ -222,3 +260,9 @@ void cpr_state_close(void) cpr_state_file = NULL; } } + +bool cpr_needed_for_reuse(void *opaque) +{ + MigMode mode = migrate_mode(); + return mode == MIG_MODE_CPR_TRANSFER; +} From patchwork Fri Feb 14 14:13:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99AE5C02198 for ; Fri, 14 Feb 2025 14:20:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSn-0004M1-OE; Fri, 14 Feb 2025 09:14:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSl-0004K5-E4 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:39 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSj-0004Qw-Jz for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:39 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECu2MM018730; Fri, 14 Feb 2025 14:14:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=1oM7G1hEz2luvuleMT7bb9H1E+mKvfumKezqrbdtfqc=; b= kLgXisjQT5DsvAie+WtY2EMJCyytCb3swAbunCuRQ5IX3uz/yie/036q/mkoXnpa ejiV/FVDjhKaxkYzuSN0kdy6EGaIm+5PQk+JcRgq0b08l6u9XULKQ5wp9Cke3EJh TcPUXsObSN5eT33hTpojBcp+HGbXDeXPnvtLVY8PFs/ooCEaUCU+Kb7iu1sFbH4P jt7PSYRzUTkfr77LzFt3kScCJK8rdm3BjOLMw7qcs6QdNgPquS684nbSUI4zt+/c uLzn5hddACbVOueGmVLJ5kV20XxmB+G8q1DFW4qtVDRlvTvHl70GvfmE8V1cbgAp NpItPetFsE1I8Z/dJvyMlQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tgbtvq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:35 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EDSoBA025143; Fri, 14 Feb 2025 14:14:34 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg4t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:33 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPJ006920; Fri, 14 Feb 2025 14:14:33 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-4; Fri, 14 Feb 2025 14:14:33 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 03/45] migration: lower handler priority Date: Fri, 14 Feb 2025 06:13:45 -0800 Message-Id: <1739542467-226739-4-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: TJVbJn3ViBuJ2eksh1wFmteaUrIilX_a X-Proofpoint-ORIG-GUID: TJVbJn3ViBuJ2eksh1wFmteaUrIilX_a Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define a vmstate priority that is lower than the default, so its handlers run after all default priority handlers. Since 0 is no longer the default priority, translate an uninitialized priority of 0 to MIG_PRI_DEFAULT. CPR for vfio will use this to install handlers for containers that run after handlers for the devices that they contain. Signed-off-by: Steve Sistare Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu --- include/migration/vmstate.h | 6 +++++- migration/savevm.c | 4 ++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h index a1dfab4..1ff7bd9 100644 --- a/include/migration/vmstate.h +++ b/include/migration/vmstate.h @@ -155,7 +155,11 @@ enum VMStateFlags { }; typedef enum { - MIG_PRI_DEFAULT = 0, + MIG_PRI_UNINITIALIZED = 0, /* An uninitialized priority field maps to */ + /* MIG_PRI_DEFAULT in save_state_priority */ + + MIG_PRI_LOW, /* Must happen after default */ + MIG_PRI_DEFAULT, MIG_PRI_IOMMU, /* Must happen before PCI devices */ MIG_PRI_PCI_BUS, /* Must happen before IOMMU */ MIG_PRI_VIRTIO_MEM, /* Must happen before IOMMU */ diff --git a/migration/savevm.c b/migration/savevm.c index 85a3559..7eee90d 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -232,7 +232,7 @@ typedef struct SaveState { static SaveState savevm_state = { .handlers = QTAILQ_HEAD_INITIALIZER(savevm_state.handlers), - .handler_pri_head = { [MIG_PRI_DEFAULT ... MIG_PRI_MAX] = NULL }, + .handler_pri_head = { [0 ... MIG_PRI_MAX] = NULL }, .global_section_id = 0, }; @@ -704,7 +704,7 @@ static int calculate_compat_instance_id(const char *idstr) static inline MigrationPriority save_state_priority(SaveStateEntry *se) { - if (se->vmsd) { + if (se->vmsd && se->vmsd->priority) { return se->vmsd->priority; } return MIG_PRI_DEFAULT; From patchwork Fri Feb 14 14:13:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 212C2C021A6 for ; Fri, 14 Feb 2025 14:15:02 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSo-0004M5-B3; Fri, 14 Feb 2025 09:14:42 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSm-0004L4-B3 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:40 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSk-0004RB-Lq for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:40 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtTkw001915; Fri, 14 Feb 2025 14:14:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=njNadYh9BwFewtCdk6dpQNmsscOfrvqJ1q70vR+SGq0=; b= kzjfFq3+rb7AVIOg4itG1Dv16F2yK3uZCtRrmaNdjj0A1gI4mwxcob3ZtJ9h6F6G k1b//BHwcRRKhHnn04VyniA3Zh+pTWF8pqe6WMwS0RijM/xgtmg5WlVU7Jf5gtsO cWE2grqkejPKsTxmJBrcWk7xyhMY9o6ZWUuTCfnzcvKCqzVTRBUxq7tYVKyflE0g qCaY48tL5khEYKzPxwV0X5leON5PUMY0l7mJpjTB4RY/lO8gMNzW27O8SemDnr5F igBHzP8s7Nd4LF8Ks5drXrhttvwDzhWsoT7ZGohSGkXMei/5Xj+jHd+7nG5WE/CK ZhwDnDoK2X4Ql4LBZSpGrA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyusv4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:36 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EC5VgY025186; Fri, 14 Feb 2025 14:14:35 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg59-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:35 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPL006920; Fri, 14 Feb 2025 14:14:34 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-5; Fri, 14 Feb 2025 14:14:34 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 04/45] vfio: vfio_find_ram_discard_listener Date: Fri, 14 Feb 2025 06:13:46 -0800 Message-Id: <1739542467-226739-5-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: 3kM9mvITPRtAXSOj2mLZJi0hUIusO6tE X-Proofpoint-GUID: 3kM9mvITPRtAXSOj2mLZJi0hUIusO6tE Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define vfio_find_ram_discard_listener as a subroutine so additional calls to it may be added in a subsequent patch. Signed-off-by: Steve Sistare Reviewed-by: Cédric Le Goater --- hw/vfio/common.c | 35 ++++++++++++++++++++++------------- include/hw/vfio/vfio-common.h | 3 +++ 2 files changed, 25 insertions(+), 13 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index abbdc56..e3e1da0 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -567,6 +567,26 @@ static void vfio_device_error_append(VFIODevice *vbasedev, Error **errp) } } +VFIORamDiscardListener *vfio_find_ram_discard_listener( + VFIOContainerBase *bcontainer, MemoryRegionSection *section) +{ + VFIORamDiscardListener *vrdl = NULL; + + QLIST_FOREACH(vrdl, &bcontainer->vrdl_list, next) { + if (vrdl->mr == section->mr && + vrdl->offset_within_address_space == + section->offset_within_address_space) { + break; + } + } + + if (!vrdl) { + hw_error("vfio: Trying to sync missing RAM discard listener"); + /* does not return */ + } + return vrdl; +} + static void vfio_listener_region_add(MemoryListener *listener, MemoryRegionSection *section) { @@ -1284,19 +1304,8 @@ vfio_sync_ram_discard_listener_dirty_bitmap(VFIOContainerBase *bcontainer, MemoryRegionSection *section) { RamDiscardManager *rdm = memory_region_get_ram_discard_manager(section->mr); - VFIORamDiscardListener *vrdl = NULL; - - QLIST_FOREACH(vrdl, &bcontainer->vrdl_list, next) { - if (vrdl->mr == section->mr && - vrdl->offset_within_address_space == - section->offset_within_address_space) { - break; - } - } - - if (!vrdl) { - hw_error("vfio: Trying to sync missing RAM discard listener"); - } + VFIORamDiscardListener *vrdl = + vfio_find_ram_discard_listener(bcontainer, section); /* * We only want/can synchronize the bitmap for actually mapped parts - diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ac35136..d601eea 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -306,6 +306,9 @@ int vfio_devices_query_dirty_bitmap(const VFIOContainerBase *bcontainer, int vfio_get_dirty_bitmap(const VFIOContainerBase *bcontainer, uint64_t iova, uint64_t size, ram_addr_t ram_addr, Error **errp); +VFIORamDiscardListener *vfio_find_ram_discard_listener( + VFIOContainerBase *bcontainer, MemoryRegionSection *section); + /* Returns 0 on success, or a negative errno. */ bool vfio_device_get_name(VFIODevice *vbasedev, Error **errp); void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp); From patchwork Fri Feb 14 14:13:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 83021C02198 for ; Fri, 14 Feb 2025 14:15:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSp-0004Mu-UC; Fri, 14 Feb 2025 09:14:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSo-0004MD-4W for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:42 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSm-0004RZ-BB for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:41 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuZE5002760; Fri, 14 Feb 2025 14:14:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=HrQY6vH++M9THOQRIuVMVfpVnIKVdhcKiCxbjG7D3bU=; b= OMNJGug8w6xmxuC4gtEXM01uN+VPM9qbzRpRJYyy6kBy2GF6BjjNF1dypnGuKb0h cu+7F1LOcSverjNPKjfEQOoiTCGkOYJhgX+sUGa/HjMCTmJzXW9lmAlWZbuoQ4rI 32sbANCO8lTyjBkStHGaj7OizJbtRQC2O6cbAkoIYkuMvYJZzj1udgzLSXttHPfO 4wcx2YAeGv6roMRseuKpLuMnCEN7zA/agLvONuFgyLWPuqziHdtv3ZSOl3jNNxph yxgC8DWOAAGL4RDP7hw2rCo5Czgm0huCRWGVJ5LaEEmVqPXCtCTuIKLZkYT7qHIc pFD5KmxgwrgGsuvHCAY02g== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0q2krue-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:37 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EDoAnJ025161; Fri, 14 Feb 2025 14:14:36 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg5x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:36 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPN006920; Fri, 14 Feb 2025 14:14:36 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-6; Fri, 14 Feb 2025 14:14:35 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 05/45] vfio/container: ram discard disable helper Date: Fri, 14 Feb 2025 06:13:47 -0800 Message-Id: <1739542467-226739-6-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=966 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: Dx7rxOJsyWJvGafXHzU0GG--cCMHnXWf X-Proofpoint-ORIG-GUID: Dx7rxOJsyWJvGafXHzU0GG--cCMHnXWf Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define a helper to set ram discard disable, generate error messages, and cleanup on failure. The second vfio_ram_block_discard_disable call site now performs VFIO_GROUP_UNSET_CONTAINER immediately on failure, instead of relying on the close of the container fd to do so in the kernel, but this is equivalent. Signed-off-by: Steve Sistare Reviewed-by: Cédric Le Goater --- hw/vfio/container.c | 48 +++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 7c57bdd2..0f17d53 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -535,16 +535,10 @@ static bool vfio_legacy_setup(VFIOContainerBase *bcontainer, Error **errp) return true; } -static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, - Error **errp) +static bool vfio_attach_discard_disable(VFIOContainer *container, + VFIOGroup *group, Error **errp) { - VFIOContainer *container; - VFIOContainerBase *bcontainer; - int ret, fd; - VFIOAddressSpace *space; - VFIOIOMMUClass *vioc; - - space = vfio_get_address_space(as); + int ret; /* * VFIO is currently incompatible with discarding of RAM insofar as the @@ -577,18 +571,32 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, * details once we know which type of IOMMU we are using. */ + ret = vfio_ram_block_discard_disable(container, true); + if (ret) { + error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"); + if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, &container->fd)) { + error_report("vfio: error disconnecting group %d from" + " container", group->groupid); + } + } + return !ret; +} + +static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, + Error **errp) +{ + VFIOContainer *container; + VFIOContainerBase *bcontainer; + int ret, fd; + VFIOAddressSpace *space; + VFIOIOMMUClass *vioc; + + space = vfio_get_address_space(as); + QLIST_FOREACH(bcontainer, &space->containers, next) { container = container_of(bcontainer, VFIOContainer, bcontainer); if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { - ret = vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, - "Cannot set discarding of RAM broken"); - if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, - &container->fd)) { - error_report("vfio: error disconnecting group %d from" - " container", group->groupid); - } + if (!vfio_attach_discard_disable(container, group, errp)) { return false; } group->container = container; @@ -620,9 +628,7 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, goto free_container_exit; } - ret = vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"); + if (!vfio_attach_discard_disable(container, group, errp)) { goto unregister_container_exit; } From patchwork Fri Feb 14 14:13:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B321C021A9 for ; Fri, 14 Feb 2025 14:24:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSs-0004No-42; Fri, 14 Feb 2025 09:14:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSp-0004Mv-Mp for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:43 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSo-0004Ro-03 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:43 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtUQu017777; Fri, 14 Feb 2025 14:14:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=kea8JUzvCW8nODhxeDL5w/7NENARlC44BW5O1/xjKAQ=; b= ZPkNqfDro0sxe9XlgaYO+UnqgWV2RAgwQX91nsDvKfnzKvR2ilhn5Y8VYDqQdjyi RV/gT6Jw1KhlRQ3z6+1/yS0iMp6PxqfrFjffxokJT16DvB9z5mpIi1CUyP196qey YLTwgzL0Rzp/YcWD0KhDqaQBza9PfwfDyAQ+gPNmZJGSMKlRa4DB4arzML/TK9eL PhIsLtGaYbh/CYqtmV2lLATedG29Re+fw+Nw6M6zcbDyuCUKE3hr7VW1wL20gCPa 61Dxfg2VZGi5cC17Vnx6qj1HMM6Df828SaMpby15zproUICKtK99Wjaaz8UojAU9 ABKgaIk/C1O4qowFwDacMA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tgbtvu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:39 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECP2ko025273; Fri, 14 Feb 2025 14:14:37 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg6j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:37 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPP006920; Fri, 14 Feb 2025 14:14:37 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-7; Fri, 14 Feb 2025 14:14:37 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 06/45] vfio/container: reform vfio_connect_container cleanup Date: Fri, 14 Feb 2025 06:13:48 -0800 Message-Id: <1739542467-226739-7-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: DbU7bNdqkoIU_LdU3eSd9mOTzVOnn6xN X-Proofpoint-ORIG-GUID: DbU7bNdqkoIU_LdU3eSd9mOTzVOnn6xN Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Replace the proliferation of exit labels in vfio_connect_container with conditionals for cleaning each piece of state. No functional change. Signed-off-by: Steve Sistare Reviewed-by: Cédric Le Goater --- hw/vfio/container.c | 61 +++++++++++++++++++++++++++++------------------------ 1 file changed, 33 insertions(+), 28 deletions(-) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 0f17d53..c668d07 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -587,9 +587,12 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, { VFIOContainer *container; VFIOContainerBase *bcontainer; - int ret, fd; + int ret, fd = -1; VFIOAddressSpace *space; - VFIOIOMMUClass *vioc; + VFIOIOMMUClass *vioc = NULL; + bool new_container = false; + bool group_was_added = false; + bool discard_disabled = false; space = vfio_get_address_space(as); @@ -608,35 +611,37 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, fd = qemu_open("/dev/vfio/vfio", O_RDWR, errp); if (fd < 0) { - goto put_space_exit; + goto fail; } ret = ioctl(fd, VFIO_GET_API_VERSION); if (ret != VFIO_API_VERSION) { error_setg(errp, "supported vfio version: %d, " "reported version: %d", VFIO_API_VERSION, ret); - goto close_fd_exit; + goto fail; } container = vfio_create_container(fd, group, errp); if (!container) { - goto close_fd_exit; + goto fail; } + new_container = true; bcontainer = &container->bcontainer; if (!vfio_cpr_register_container(bcontainer, errp)) { - goto free_container_exit; + goto fail; } if (!vfio_attach_discard_disable(container, group, errp)) { - goto unregister_container_exit; + goto fail; } + discard_disabled = true; vioc = VFIO_IOMMU_GET_CLASS(bcontainer); assert(vioc->setup); if (!vioc->setup(bcontainer, errp)) { - goto enable_discards_exit; + goto fail; } vfio_kvm_device_add_group(group); @@ -645,6 +650,7 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, group->container = container; QLIST_INSERT_HEAD(&container->group_list, group, container_next); + group_was_added = true; bcontainer->listener = vfio_memory_listener; memory_listener_register(&bcontainer->listener, bcontainer->space->as); @@ -652,35 +658,34 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, if (bcontainer->error) { error_propagate_prepend(errp, bcontainer->error, "memory listener initialization failed: "); - goto listener_release_exit; + goto fail; } bcontainer->initialized = true; return true; -listener_release_exit: - QLIST_REMOVE(group, container_next); - vfio_kvm_device_del_group(group); + +fail: memory_listener_unregister(&bcontainer->listener); - if (vioc->release) { + + if (group_was_added) { + QLIST_REMOVE(group, container_next); + vfio_kvm_device_del_group(group); + } + if (vioc && vioc->release) { vioc->release(bcontainer); } - -enable_discards_exit: - vfio_ram_block_discard_disable(container, false); - -unregister_container_exit: - vfio_cpr_unregister_container(bcontainer); - -free_container_exit: - object_unref(container); - -close_fd_exit: - close(fd); - -put_space_exit: + if (discard_disabled) { + vfio_ram_block_discard_disable(container, false); + } + if (new_container) { + vfio_cpr_unregister_container(bcontainer); + object_unref(container); + } + if (fd >= 0) { + close(fd); + } vfio_put_address_space(space); - return false; } From patchwork Fri Feb 14 14:13:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975018 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A22AC021A6 for ; Fri, 14 Feb 2025 14:24:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSs-0004OV-Rt; Fri, 14 Feb 2025 09:14:46 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSq-0004NV-QK for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:44 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSo-0004Rx-Tm for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:44 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtqaR022708; Fri, 14 Feb 2025 14:14:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=BI7xHpN3QRFQfxooh96sk1xXdkB6bvKSbO1XC86F69Q=; b= N6KDb12HyhiQy3ccdt7YHvQi0sBQAGyCECtK73voIbZq/8WA3igA6gCuMh5lcWJw 4A3AP7m5jFzZd72PCU14yd/4PcxfwGgP9m3VtcbGmX/Wie1yJV5bX20QNNOk1bEn 7+Vlc7/2pReQLnIU+fmtHBhDO5edcd72M09EbNQ8b1cnexFmhdvmJay48YimcLR4 XjK4pwFPWldbwt431giMKR615GyY5dat113uiUn64zhiH9f5ZjIKfqREWmp31DbA ZX+XrMNlUajSDCJR2Th4IoYxJs+zQunUYaEGZMKcWmkKYnpubADVyTVNFxxtlE+L VwfkaFJyy6kYfdUc/+7STQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0s43q6h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:40 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECwW5o025139; Fri, 14 Feb 2025 14:14:39 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg76-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:39 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPR006920; Fri, 14 Feb 2025 14:14:38 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-8; Fri, 14 Feb 2025 14:14:38 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 07/45] vfio/container: vfio_container_group_add Date: Fri, 14 Feb 2025 06:13:49 -0800 Message-Id: <1739542467-226739-8-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: YKVcgwofVQIBWbk4x06DcaBUnYSsZKpN X-Proofpoint-ORIG-GUID: YKVcgwofVQIBWbk4x06DcaBUnYSsZKpN Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add vfio_container_group_add to de-dup some code. No functional change. Signed-off-by: Steve Sistare Reviewed-by: Cédric Le Goater --- hw/vfio/container.c | 47 +++++++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index c668d07..c5bbb03 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -582,6 +582,26 @@ static bool vfio_attach_discard_disable(VFIOContainer *container, return !ret; } +static bool vfio_container_group_add(VFIOContainer *container, VFIOGroup *group, + Error **errp) +{ + if (!vfio_attach_discard_disable(container, group, errp)) { + return false; + } + group->container = container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + vfio_kvm_device_add_group(group); + return true; +} + +static void vfio_container_group_del(VFIOContainer *container, VFIOGroup *group) +{ + QLIST_REMOVE(group, container_next); + group->container = NULL; + vfio_kvm_device_del_group(group); + vfio_ram_block_discard_disable(container, false); +} + static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, Error **errp) { @@ -592,20 +612,13 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, VFIOIOMMUClass *vioc = NULL; bool new_container = false; bool group_was_added = false; - bool discard_disabled = false; space = vfio_get_address_space(as); QLIST_FOREACH(bcontainer, &space->containers, next) { container = container_of(bcontainer, VFIOContainer, bcontainer); if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { - if (!vfio_attach_discard_disable(container, group, errp)) { - return false; - } - group->container = container; - QLIST_INSERT_HEAD(&container->group_list, group, container_next); - vfio_kvm_device_add_group(group); - return true; + return vfio_container_group_add(container, group, errp); } } @@ -632,11 +645,6 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, goto fail; } - if (!vfio_attach_discard_disable(container, group, errp)) { - goto fail; - } - discard_disabled = true; - vioc = VFIO_IOMMU_GET_CLASS(bcontainer); assert(vioc->setup); @@ -644,12 +652,11 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, goto fail; } - vfio_kvm_device_add_group(group); - vfio_address_space_insert(space, bcontainer); - group->container = container; - QLIST_INSERT_HEAD(&container->group_list, group, container_next); + if (!vfio_container_group_add(container, group, errp)) { + goto fail; + } group_was_added = true; bcontainer->listener = vfio_memory_listener; @@ -669,15 +676,11 @@ fail: memory_listener_unregister(&bcontainer->listener); if (group_was_added) { - QLIST_REMOVE(group, container_next); - vfio_kvm_device_del_group(group); + vfio_container_group_del(container, group); } if (vioc && vioc->release) { vioc->release(bcontainer); } - if (discard_disabled) { - vfio_ram_block_discard_disable(container, false); - } if (new_container) { vfio_cpr_unregister_container(bcontainer); object_unref(container); From patchwork Fri Feb 14 14:13:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92641C021A4 for ; Fri, 14 Feb 2025 14:27:09 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSt-0004Os-F4; Fri, 14 Feb 2025 09:14:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSs-0004Nw-Ew for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:46 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSq-0004S8-6W for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:46 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtUnj001530; Fri, 14 Feb 2025 14:14:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=25agqrVRl+Z/STHiqShSmbHb9RkMccnzJxBqfaWd/OY=; b= HD/Wtuq1oIt2xMD4tEOzBM9mgnCAR72NMJLarz7xpT8/1MfwUPfZD8Vp77TWYanm oi7KxsUKbvDCUfo2wjbHSLA2kHeKQpjrD88TShLD3/t+UxqyCkDEFRMDmEMtI1QV WyeC4cycoxdF4xSvexF6AsrK5HK9GM1Cxc48WSxyAZefQjVH7pMbB/NsfEzbygO+ Sz1OIwuxZ1DPunKD7SQL6tZWYUg2piwK1yOEtQ0MmzimU6pViwtPF4P0WxFsCtkZ rQPATXP0I/YojShZiNGXx1Xi00CH8oBqrJyPWprbJL5xJikb4h9ZUJqpO12t26Sy FSIRN4YQbrC/havgTW8Erw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0q2kruj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:41 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECxs2W025360; Fri, 14 Feb 2025 14:14:40 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg7y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:40 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPT006920; Fri, 14 Feb 2025 14:14:40 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-9; Fri, 14 Feb 2025 14:14:39 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 08/45] vfio/container: register container for cpr Date: Fri, 14 Feb 2025 06:13:50 -0800 Message-Id: <1739542467-226739-9-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: ML9W1fauc_XAXLYKOFeLQxL-gi3OEz2L X-Proofpoint-ORIG-GUID: ML9W1fauc_XAXLYKOFeLQxL-gi3OEz2L Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Register a legacy container for cpr-transfer, replacing the generic CPR register call with a more specific legacy container register call. Add a blocker if the kernel does not support VFIO_UPDATE_VADDR or VFIO_UNMAP_ALL. This is mostly boiler plate. The fields to to saved and restored are added in subsequent patches. Signed-off-by: Steve Sistare --- MAINTAINERS | 1 + hw/vfio/container.c | 6 ++-- hw/vfio/cpr-legacy.c | 69 +++++++++++++++++++++++++++++++++++++++++++ hw/vfio/cpr.c | 6 ++-- hw/vfio/meson.build | 3 +- include/hw/vfio/vfio-common.h | 2 ++ include/hw/vfio/vfio-cpr.h | 25 ++++++++++++++++ 7 files changed, 105 insertions(+), 7 deletions(-) create mode 100644 hw/vfio/cpr-legacy.c create mode 100644 include/hw/vfio/vfio-cpr.h diff --git a/MAINTAINERS b/MAINTAINERS index 2f9a6da..aee1342 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2957,6 +2957,7 @@ M: Fabiano Rosas R: Steve Sistare S: Supported F: hw/vfio/cpr* +F: include/hw/vfio/vfio-cpr.h F: include/migration/cpr.h F: migration/cpr* F: tests/qtest/migration/cpr* diff --git a/hw/vfio/container.c b/hw/vfio/container.c index c5bbb03..eca3362 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -641,7 +641,7 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, new_container = true; bcontainer = &container->bcontainer; - if (!vfio_cpr_register_container(bcontainer, errp)) { + if (!vfio_legacy_cpr_register_container(container, errp)) { goto fail; } @@ -682,7 +682,7 @@ fail: vioc->release(bcontainer); } if (new_container) { - vfio_cpr_unregister_container(bcontainer); + vfio_legacy_cpr_unregister_container(container); object_unref(container); } if (fd >= 0) { @@ -722,7 +722,7 @@ static void vfio_disconnect_container(VFIOGroup *group) VFIOAddressSpace *space = bcontainer->space; trace_vfio_disconnect_container(container->fd); - vfio_cpr_unregister_container(bcontainer); + vfio_legacy_cpr_unregister_container(container); close(container->fd); object_unref(container); diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c new file mode 100644 index 0000000..d0557af --- /dev/null +++ b/hw/vfio/cpr-legacy.c @@ -0,0 +1,69 @@ +/* + * Copyright (c) 2021-2025 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include +#include "qemu/osdep.h" +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio-cpr.h" +#include "migration/blocker.h" +#include "migration/cpr.h" +#include "migration/migration.h" +#include "migration/vmstate.h" +#include "qapi/error.h" + +static bool vfio_cpr_supported(VFIOContainer *container, Error **errp) +{ + if (!ioctl(container->fd, VFIO_CHECK_EXTENSION, VFIO_UPDATE_VADDR)) { + error_setg(errp, "VFIO container does not support VFIO_UPDATE_VADDR"); + return false; + + } else if (!ioctl(container->fd, VFIO_CHECK_EXTENSION, VFIO_UNMAP_ALL)) { + error_setg(errp, "VFIO container does not support VFIO_UNMAP_ALL"); + return false; + + } else { + return true; + } +} + +static const VMStateDescription vfio_container_vmstate = { + .name = "vfio-container", + .version_id = 0, + .minimum_version_id = 0, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; + +bool vfio_legacy_cpr_register_container(VFIOContainer *container, Error **errp) +{ + VFIOContainerBase *bcontainer = &container->bcontainer; + Error **cpr_blocker = &container->cpr.blocker; + + migration_add_notifier_mode(&bcontainer->cpr_reboot_notifier, + vfio_cpr_reboot_notifier, + MIG_MODE_CPR_REBOOT); + + if (!vfio_cpr_supported(container, cpr_blocker)) { + return migrate_add_blocker_modes(cpr_blocker, errp, + MIG_MODE_CPR_TRANSFER, -1) == 0; + } + + vmstate_register(NULL, -1, &vfio_container_vmstate, container); + + return true; +} + +void vfio_legacy_cpr_unregister_container(VFIOContainer *container) +{ + VFIOContainerBase *bcontainer = &container->bcontainer; + + migration_remove_notifier(&bcontainer->cpr_reboot_notifier); + migrate_del_blocker(&container->cpr.blocker); + vmstate_unregister(NULL, &vfio_container_vmstate, container); +} diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index 3d1c8d2..6790f8a 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -7,12 +7,12 @@ #include "qemu/osdep.h" #include "hw/vfio/vfio-common.h" -#include "migration/misc.h" +#include "hw/vfio/vfio-cpr.h" #include "qapi/error.h" #include "system/runstate.h" -static int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, - MigrationEvent *e, Error **errp) +int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, + MigrationEvent *e, Error **errp) { if (e->type == MIG_EVENT_PRECOPY_SETUP && !runstate_check(RUN_STATE_SUSPENDED) && !vm_get_suspended()) { diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index bba776f..5487815 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -5,13 +5,14 @@ vfio_ss.add(files( 'container-base.c', 'container.c', 'migration.c', - 'cpr.c', )) vfio_ss.add(when: 'CONFIG_PSERIES', if_true: files('spapr.c')) vfio_ss.add(when: 'CONFIG_IOMMUFD', if_true: files( 'iommufd.c', )) vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( + 'cpr.c', + 'cpr-legacy.c', 'display.c', 'pci-quirks.c', 'pci.c', diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index d601eea..c482364 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -31,6 +31,7 @@ #endif #include "system/system.h" #include "hw/vfio/vfio-container-base.h" +#include "hw/vfio/vfio-cpr.h" #include "system/host_iommu_device.h" #include "system/iommufd.h" @@ -85,6 +86,7 @@ typedef struct VFIOContainer { int fd; /* /dev/vfio/vfio, empowered by the attached groups */ unsigned iommu_type; QLIST_HEAD(, VFIOGroup) group_list; + VFIOContainerCPR cpr; } VFIOContainer; OBJECT_DECLARE_SIMPLE_TYPE(VFIOContainer, VFIO_IOMMU_LEGACY); diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h new file mode 100644 index 0000000..d4f8346 --- /dev/null +++ b/include/hw/vfio/vfio-cpr.h @@ -0,0 +1,25 @@ +/* + * Copyright (c) 2025 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#ifndef HW_VFIO_VFIO_CPR_H +#define HW_VFIO_VFIO_CPR_H + +#include "migration/misc.h" + +typedef struct VFIOContainerCPR { + Error *blocker; +} VFIOContainerCPR; + +struct VFIOContainer; + +int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, + Error **errp); + +bool vfio_legacy_cpr_register_container(struct VFIOContainer *container, + Error **errp); +void vfio_legacy_cpr_unregister_container(struct VFIOContainer *container); +#endif From patchwork Fri Feb 14 14:13:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 070B1C02198 for ; Fri, 14 Feb 2025 14:15:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSw-0004Q2-4E; Fri, 14 Feb 2025 09:14:50 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSt-0004P2-Ur for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:48 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSr-0004SK-Ix for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:47 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtpWp019370; Fri, 14 Feb 2025 14:14:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=Jtd5qFBF4U2CM51CoANTy5520jQ5bgifXFgTF7QEBVo=; b= CfXCemX3HUd1NVQqLse/jB6kjauGZb/00uG5cOs/PPU/IdLp80AxZ+0lHu7SFKSz 0ZIwY5aVCkj5pU2sATPT2ibNf3Ug8x87Swz/miAHx7V5Vpz8yBgUTTHgcD2Ipuyy 8KSTvai5JFn3ta+9kG2J1/x8IvZyOunqAf2IkbHkvR+Qe4hK69WIxiG3H2QBOWkS NAdxgph444fup2KJeE0blvqlowyuLXrkW+F+bHU48h/YFcyj8F+xG1RxD4Ay8Y3s bDRWW3PI4dMXZDIqABzghvB8/dNl0YKq0fbmSQ4CTn1O97QW7/8ArqdI/7AElVRK lR7JXUmK7S3iDcTXZ8bKUA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0t4bmum-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:43 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuXh1025256; Fri, 14 Feb 2025 14:14:41 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg8f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:41 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPV006920; Fri, 14 Feb 2025 14:14:41 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-10; Fri, 14 Feb 2025 14:14:41 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 09/45] vfio/container: preserve descriptors Date: Fri, 14 Feb 2025 06:13:51 -0800 Message-Id: <1739542467-226739-10-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: p-LjDkH3VTpyqEoNYiwuYSecgpuGGcva X-Proofpoint-GUID: p-LjDkH3VTpyqEoNYiwuYSecgpuGGcva Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org At vfio creation time, save the value of vfio container, group, and device descriptors in CPR state. On qemu restart, vfio_realize() finds and uses the saved descriptors, and remembers the reused status for subsequent patches. The reused status is cleared when vmstate load finishes. During reuse, device and iommu state is already configured, so operations in vfio_realize that would modify the configuration, such as vfio ioctl's, are skipped. The result is that vfio_realize constructs qemu data structures that reflect the current state of the device. Signed-off-by: Steve Sistare --- hw/vfio/container.c | 57 +++++++++++++++++++++++++++++++++++++------ hw/vfio/cpr-legacy.c | 45 ++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 1 + include/hw/vfio/vfio-cpr.h | 9 +++++++ 4 files changed, 104 insertions(+), 8 deletions(-) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index eca3362..21f2706 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -31,6 +31,8 @@ #include "system/reset.h" #include "trace.h" #include "qapi/error.h" +#include "migration/cpr.h" +#include "migration/blocker.h" #include "pci.h" VFIOGroupList vfio_group_list = @@ -413,7 +415,7 @@ static bool vfio_set_iommu(int container_fd, int group_fd, } static VFIOContainer *vfio_create_container(int fd, VFIOGroup *group, - Error **errp) + bool cpr_reused, Error **errp) { int iommu_type; const char *vioc_name; @@ -424,7 +426,11 @@ static VFIOContainer *vfio_create_container(int fd, VFIOGroup *group, return NULL; } - if (!vfio_set_iommu(fd, group->fd, &iommu_type, errp)) { + /* + * If container is reused, just set its type and skip the ioctls, as the + * container and group are already configured in the kernel. + */ + if (!cpr_reused && !vfio_set_iommu(fd, group->fd, &iommu_type, errp)) { return NULL; } @@ -432,6 +438,7 @@ static VFIOContainer *vfio_create_container(int fd, VFIOGroup *group, container = VFIO_IOMMU_LEGACY(object_new(vioc_name)); container->fd = fd; + container->cpr.reused = cpr_reused; container->iommu_type = iommu_type; return container; } @@ -591,6 +598,7 @@ static bool vfio_container_group_add(VFIOContainer *container, VFIOGroup *group, group->container = container; QLIST_INSERT_HEAD(&container->group_list, group, container_next); vfio_kvm_device_add_group(group); + cpr_resave_fd("vfio_container_for_group", group->groupid, container->fd); return true; } @@ -600,6 +608,7 @@ static void vfio_container_group_del(VFIOContainer *container, VFIOGroup *group) group->container = NULL; vfio_kvm_device_del_group(group); vfio_ram_block_discard_disable(container, false); + cpr_delete_fd("vfio_container_for_group", group->groupid); } static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, @@ -612,17 +621,37 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, VFIOIOMMUClass *vioc = NULL; bool new_container = false; bool group_was_added = false; + bool cpr_reused; space = vfio_get_address_space(as); + fd = cpr_find_fd("vfio_container_for_group", group->groupid); + cpr_reused = (fd > 0); + + /* + * If the container is reused, then the group is already attached in the + * kernel. If a container with matching fd is found, then update the + * userland group list and return. If not, then after the loop, create + * the container struct and group list. + */ QLIST_FOREACH(bcontainer, &space->containers, next) { container = container_of(bcontainer, VFIOContainer, bcontainer); - if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { - return vfio_container_group_add(container, group, errp); + + if (cpr_reused) { + if (!vfio_cpr_container_match(container, group, &fd)) { + continue; + } + } else if (ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { + continue; } + + return vfio_container_group_add(container, group, errp); + } + + if (!cpr_reused) { + fd = qemu_open("/dev/vfio/vfio", O_RDWR, errp); } - fd = qemu_open("/dev/vfio/vfio", O_RDWR, errp); if (fd < 0) { goto fail; } @@ -634,7 +663,7 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, goto fail; } - container = vfio_create_container(fd, group, errp); + container = vfio_create_container(fd, group, cpr_reused, errp); if (!container) { goto fail; } @@ -700,6 +729,7 @@ static void vfio_disconnect_container(VFIOGroup *group) QLIST_REMOVE(group, container_next); group->container = NULL; + cpr_delete_fd("vfio_container_for_group", group->groupid); /* * Explicitly release the listener first before unset container, @@ -753,7 +783,7 @@ static VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) group = g_malloc0(sizeof(*group)); snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); - group->fd = qemu_open(path, O_RDWR, errp); + group->fd = cpr_open_fd(path, O_RDWR, "vfio_group", groupid, NULL, errp); if (group->fd < 0) { goto free_group_exit; } @@ -785,6 +815,7 @@ static VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) return group; close_fd_exit: + cpr_delete_fd("vfio_group", groupid); close(group->fd); free_group_exit: @@ -806,6 +837,7 @@ static void vfio_put_group(VFIOGroup *group) vfio_disconnect_container(group); QLIST_REMOVE(group, next); trace_vfio_put_group(group->fd); + cpr_delete_fd("vfio_group", group->groupid); close(group->fd); g_free(group); } @@ -815,8 +847,14 @@ static bool vfio_get_device(VFIOGroup *group, const char *name, { g_autofree struct vfio_device_info *info = NULL; int fd; + bool cpr_reused; + + fd = cpr_find_fd(name, 0); + cpr_reused = (fd >= 0); + if (!cpr_reused) { + fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); + } - fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); if (fd < 0) { error_setg_errno(errp, errno, "error getting device from group %d", group->groupid); @@ -861,6 +899,8 @@ static bool vfio_get_device(VFIOGroup *group, const char *name, vbasedev->num_irqs = info->num_irqs; vbasedev->num_regions = info->num_regions; vbasedev->flags = info->flags; + vbasedev->cpr.reused = cpr_reused; + cpr_resave_fd(name, 0, fd); trace_vfio_get_device(name, info->flags, info->num_regions, info->num_irqs); @@ -877,6 +917,7 @@ static void vfio_put_base_device(VFIODevice *vbasedev) QLIST_REMOVE(vbasedev, next); vbasedev->group = NULL; trace_vfio_put_base_device(vbasedev->fd); + cpr_delete_fd(vbasedev->name, 0); close(vbasedev->fd); } diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c index d0557af..cee0f4e 100644 --- a/hw/vfio/cpr-legacy.c +++ b/hw/vfio/cpr-legacy.c @@ -30,10 +30,27 @@ static bool vfio_cpr_supported(VFIOContainer *container, Error **errp) } } +static int vfio_container_post_load(void *opaque, int version_id) +{ + VFIOContainer *container = opaque; + VFIOGroup *group; + VFIODevice *vbasedev; + + container->cpr.reused = false; + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + vbasedev->cpr.reused = false; + } + } + return 0; +} + static const VMStateDescription vfio_container_vmstate = { .name = "vfio-container", .version_id = 0, .minimum_version_id = 0, + .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { VMSTATE_END_OF_LIST() @@ -67,3 +84,31 @@ void vfio_legacy_cpr_unregister_container(VFIOContainer *container) migrate_del_blocker(&container->cpr.blocker); vmstate_unregister(NULL, &vfio_container_vmstate, container); } + +static bool same_device(int fd1, int fd2) +{ + struct stat st1, st2; + + return !fstat(fd1, &st1) && !fstat(fd2, &st2) && st1.st_dev == st2.st_dev; +} + +bool vfio_cpr_container_match(VFIOContainer *container, VFIOGroup *group, + int *pfd) +{ + if (container->fd == *pfd) { + return true; + } + if (!same_device(container->fd, *pfd)) { + return false; + } + /* + * Same device, different fd. This occurs when the container fd is + * cpr_save'd multiple times, once for each groupid, so SCM_RIGHTS + * produces duplicates. De-dup it. + */ + cpr_delete_fd("vfio_container_for_group", group->groupid); + close(*pfd); + cpr_save_fd("vfio_container_for_group", group->groupid, container->fd); + *pfd = container->fd; + return true; +} diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index c482364..780646e 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -152,6 +152,7 @@ typedef struct VFIODevice { IOMMUFDBackend *iommufd; VFIOIOASHwpt *hwpt; QLIST_ENTRY(VFIODevice) hwpt_next; + VFIODeviceCPR cpr; } VFIODevice; struct VFIODeviceOps { diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index d4f8346..1a3eee9 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -12,9 +12,15 @@ typedef struct VFIOContainerCPR { Error *blocker; + bool reused; } VFIOContainerCPR; +typedef struct VFIODeviceCPR { + bool reused; +} VFIODeviceCPR; + struct VFIOContainer; +struct VFIOGroup; int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp); @@ -22,4 +28,7 @@ int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, bool vfio_legacy_cpr_register_container(struct VFIOContainer *container, Error **errp); void vfio_legacy_cpr_unregister_container(struct VFIOContainer *container); + +bool vfio_cpr_container_match(struct VFIOContainer *container, + struct VFIOGroup *group, int *fd); #endif From patchwork Fri Feb 14 14:13:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A6DFC02198 for ; Fri, 14 Feb 2025 14:29:01 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSu-0004PU-KH; Fri, 14 Feb 2025 09:14:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSt-0004Ox-QR for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSs-0004SU-CD for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:47 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtRAx011445; Fri, 14 Feb 2025 14:14:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=Uq/w+DXMHomToGU2bYKJT7qBPRUqltM3NGgb7hJdbvA=; b= Q3V76ZSgefR0rs8plV3i6apVA/i1zhnLyaT3xm0pCtJHMSyxbv290rTdv3W5H0Bf 5WFzsoHlniYJQlarRfI0YYOGYXFSGofQ/toTDhy5XCXTXobLuSrjXxzlH+WYzDCf zvh3KZCSZqNxUcjWAD9S9iJhnjwsTBSAd41BTUdmYbjVdAwcAT12A/2Cw75GXXud XlNAg/BPBuk9ac/W0r5RxUnBv7oN5TAY+fFu1fMP0ibRblJs3dVqOhoiCKiYipQp bUlI71dSH0eBYJuHpTPueSJPGcV06tkNGJNkLQGxLADKsjzTWxIWm2vADX+aboTj tdM254GrxTgJ9I+eN2oQMQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwmc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:43 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECxs2X025360; Fri, 14 Feb 2025 14:14:43 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg90-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:43 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPX006920; Fri, 14 Feb 2025 14:14:42 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-11; Fri, 14 Feb 2025 14:14:42 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 10/45] vfio/container: export vfio_legacy_dma_map Date: Fri, 14 Feb 2025 06:13:52 -0800 Message-Id: <1739542467-226739-11-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: PmCt7Nza1NjjnMBDL9F7SkI-bQUmyz9_ X-Proofpoint-GUID: PmCt7Nza1NjjnMBDL9F7SkI-bQUmyz9_ Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Export vfio_legacy_dma_map so it may be referenced outside the file in a subsequent patch. Signed-off-by: Steve Sistare --- hw/vfio/container.c | 4 ++-- include/hw/vfio/vfio-common.h | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 21f2706..931c435 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -176,8 +176,8 @@ static int vfio_legacy_dma_unmap(const VFIOContainerBase *bcontainer, return 0; } -static int vfio_legacy_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, - ram_addr_t size, void *vaddr, bool readonly) +int vfio_legacy_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly) { const VFIOContainer *container = container_of(bcontainer, VFIOContainer, bcontainer); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 780646e..7f33476 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -280,6 +280,9 @@ void vfio_reset_bytes_transferred(void); bool vfio_device_state_is_running(VFIODevice *vbasedev); bool vfio_device_state_is_precopy(VFIODevice *vbasedev); +int vfio_legacy_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly); + #ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info); From patchwork Fri Feb 14 14:13:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3C370C02198 for ; Fri, 14 Feb 2025 14:17:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwSy-0004Qg-4v; Fri, 14 Feb 2025 09:14:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSw-0004QF-Fb for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:50 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSu-0004Sf-2o for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:49 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECu0IH017940; Fri, 14 Feb 2025 14:14:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=Vy3kzSYFDPhg7CqzpFKgkQ4v3AP386VeBEpPDGy23ik=; b= gJDmPezFs9jRH7WuKwY/tkZZFUBbNI3pUHiSYjXheEq0dHlto1PQsc+watqWCGyj czAQXCpNjodjZ0onlGQP13GVteVQwInFu4/UwSOfidgYqCwLf5H22MED4qy7JWVT bk72pwZlSfb8b00EFx1TLZxE5/RdCWUznRLI3ShgwWdTiKbZlO2Dhb3su1m6LmZu HVOJG/KiuycasJcfdKD2i0W/WwwMoEzvtZej9ZY97EuXVHvv4TonfrmlJtrXWyhO HgBQnkK2dMATz1fKCyi0aoYHhnC+/GZ5ZviZxPvnzIkyP5lOIlxITcy8AhngcvmK D8yDkHZpuRMhwV+piFq2/g== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tgbtw0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:45 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ED5ZWS025287; Fri, 14 Feb 2025 14:14:44 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksg9k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:44 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPZ006920; Fri, 14 Feb 2025 14:14:43 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-12; Fri, 14 Feb 2025 14:14:43 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 11/45] vfio/container: discard old DMA vaddr Date: Fri, 14 Feb 2025 06:13:53 -0800 Message-Id: <1739542467-226739-12-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: MbamVYu6jUcl0bKEP1Yk6Sof2NYLsg29 X-Proofpoint-ORIG-GUID: MbamVYu6jUcl0bKEP1Yk6Sof2NYLsg29 Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org In the container pre_save handler, discard the virtual addresses in DMA mappings with VFIO_DMA_UNMAP_FLAG_VADDR, because guest RAM will be remapped at a different VA after in new QEMU. DMA to already-mapped pages continues. Signed-off-by: Steve Sistare --- hw/vfio/cpr-legacy.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c index cee0f4e..97a64e7 100644 --- a/hw/vfio/cpr-legacy.c +++ b/hw/vfio/cpr-legacy.c @@ -15,6 +15,22 @@ #include "migration/vmstate.h" #include "qapi/error.h" +static bool vfio_dma_unmap_vaddr_all(VFIOContainer *container, Error **errp) +{ + struct vfio_iommu_type1_dma_unmap unmap = { + .argsz = sizeof(unmap), + .flags = VFIO_DMA_UNMAP_FLAG_VADDR | VFIO_DMA_UNMAP_FLAG_ALL, + .iova = 0, + .size = 0, + }; + if (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { + error_setg_errno(errp, errno, "vfio_dma_unmap_vaddr_all"); + return false; + } + return true; +} + + static bool vfio_cpr_supported(VFIOContainer *container, Error **errp) { if (!ioctl(container->fd, VFIO_CHECK_EXTENSION, VFIO_UPDATE_VADDR)) { @@ -30,6 +46,18 @@ static bool vfio_cpr_supported(VFIOContainer *container, Error **errp) } } +static int vfio_container_pre_save(void *opaque) +{ + VFIOContainer *container = opaque; + Error *err = NULL; + + if (!vfio_dma_unmap_vaddr_all(container, &err)) { + error_report_err(err); + return -1; + } + return 0; +} + static int vfio_container_post_load(void *opaque, int version_id) { VFIOContainer *container = opaque; @@ -50,6 +78,7 @@ static const VMStateDescription vfio_container_vmstate = { .name = "vfio-container", .version_id = 0, .minimum_version_id = 0, + .pre_save = vfio_container_pre_save, .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { From patchwork Fri Feb 14 14:13:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975020 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BD60BC02198 for ; Fri, 14 Feb 2025 14:25:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT0-0004XR-2x; Fri, 14 Feb 2025 09:14:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSy-0004Qe-1J for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:52 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSw-0004Sz-8R for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:51 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuZE8002760; Fri, 14 Feb 2025 14:14:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=6nVQvnw1f40pgwFP/V5go6owNUca75t/z4U59Azj1a4=; b= HebwY+Z77Toc773AjhjM5Wrn2yzzKlZ2h0tUiTNNEUc4t6VkdzHVbAxOOPAp6y/c yPDI01KLDcOMDc6LPv5ziJopyuvNdQSMnhs/pEyXae4GtnsiyUOllF6vThJYTd1/ E284CJu+0Eqct/mZKkTsVNuMRiIlpqyy1331CyIA2ddkhv7CGqE4IClU4flxUXpO lhjbV7of2j5r5CN4oWad4mLX5K/9uXOIy0hZpMb5tWyVxbI/fVialy81sBqdeQGd T+vBS4tG3v9gTT1st5r8l/ON7YoHFBvm34LmxY7X8zetiAsO2lFos9acz9Kpp7ny 3dJKHwDV/9af2jJNY226kw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0q2kruq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:47 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuXh5025256; Fri, 14 Feb 2025 14:14:45 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksga9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:45 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPb006920; Fri, 14 Feb 2025 14:14:45 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-13; Fri, 14 Feb 2025 14:14:45 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 12/45] vfio/container: restore DMA vaddr Date: Fri, 14 Feb 2025 06:13:54 -0800 Message-Id: <1739542467-226739-13-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: ZV2R_LX7S_QXfTDOxPGvj-iqSSW1_nAE X-Proofpoint-ORIG-GUID: ZV2R_LX7S_QXfTDOxPGvj-iqSSW1_nAE Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org In new QEMU, do not register the memory listener at device creation time. Register it later, in the container post_load handler, after all vmstate that may affect regions and mapping boundaries has been loaded. The post_load registration will cause the listener to invoke its callback on each flat section, and the calls will match the mappings remembered by the kernel. The listener calls a special dma_map handler that passes the new VA of each section to the kernel using VFIO_DMA_MAP_FLAG_VADDR. Restore the normal handler at the end. Signed-off-by: Steve Sistare --- hw/vfio/container.c | 15 +++++++++++++-- hw/vfio/cpr-legacy.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 2 deletions(-) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 931c435..d26f78e 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -133,6 +133,8 @@ static int vfio_legacy_dma_unmap(const VFIOContainerBase *bcontainer, int ret; Error *local_err = NULL; + assert(!container->cpr.reused); + if (iotlb && vfio_devices_all_dirty_tracking_started(bcontainer)) { if (!vfio_devices_all_device_dirty_tracking(bcontainer) && bcontainer->dirty_pages_supported) { @@ -688,8 +690,17 @@ static bool vfio_connect_container(VFIOGroup *group, AddressSpace *as, } group_was_added = true; - bcontainer->listener = vfio_memory_listener; - memory_listener_register(&bcontainer->listener, bcontainer->space->as); + /* + * If reused, register the listener later, after all state that may + * affect regions and mapping boundaries has been cpr load'ed. Later, + * the listener will invoke its callback on each flat section and call + * dma_map to supply the new vaddr, and the calls will match the mappings + * remembered by the kernel. + */ + if (!cpr_reused) { + bcontainer->listener = vfio_memory_listener; + memory_listener_register(&bcontainer->listener, bcontainer->space->as); + } if (bcontainer->error) { error_propagate_prepend(errp, bcontainer->error, diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c index 97a64e7..bb5f802 100644 --- a/hw/vfio/cpr-legacy.c +++ b/hw/vfio/cpr-legacy.c @@ -14,6 +14,7 @@ #include "migration/migration.h" #include "migration/vmstate.h" #include "qapi/error.h" +#include "qemu/error-report.h" static bool vfio_dma_unmap_vaddr_all(VFIOContainer *container, Error **errp) { @@ -30,6 +31,34 @@ static bool vfio_dma_unmap_vaddr_all(VFIOContainer *container, Error **errp) return true; } +/* + * Set the new @vaddr for any mappings registered during cpr load. + * Reused is cleared thereafter. + */ +static int vfio_legacy_cpr_dma_map(const VFIOContainerBase *bcontainer, + hwaddr iova, ram_addr_t size, void *vaddr, + bool readonly) +{ + const VFIOContainer *container = container_of(bcontainer, VFIOContainer, + bcontainer); + struct vfio_iommu_type1_dma_map map = { + .argsz = sizeof(map), + .flags = VFIO_DMA_MAP_FLAG_VADDR, + .vaddr = (__u64)(uintptr_t)vaddr, + .iova = iova, + .size = size, + }; + + assert(container->cpr.reused); + + if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map)) { + error_report("vfio_legacy_cpr_dma_map (iova %lu, size %ld, va %p): %s", + iova, size, vaddr, strerror(errno)); + return -errno; + } + + return 0; +} static bool vfio_cpr_supported(VFIOContainer *container, Error **errp) { @@ -61,12 +90,20 @@ static int vfio_container_pre_save(void *opaque) static int vfio_container_post_load(void *opaque, int version_id) { VFIOContainer *container = opaque; + VFIOContainerBase *bcontainer = &container->bcontainer; VFIOGroup *group; VFIODevice *vbasedev; + bcontainer->listener = vfio_memory_listener; + memory_listener_register(&bcontainer->listener, bcontainer->space->as); container->cpr.reused = false; QLIST_FOREACH(group, &container->group_list, container_next) { + VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + + /* Restore original dma_map function */ + vioc->dma_map = vfio_legacy_dma_map; + QLIST_FOREACH(vbasedev, &group->device_list, next) { vbasedev->cpr.reused = false; } @@ -78,6 +115,7 @@ static const VMStateDescription vfio_container_vmstate = { .name = "vfio-container", .version_id = 0, .minimum_version_id = 0, + .priority = MIG_PRI_LOW, /* Must happen after devices and groups */ .pre_save = vfio_container_pre_save, .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, @@ -102,6 +140,11 @@ bool vfio_legacy_cpr_register_container(VFIOContainer *container, Error **errp) vmstate_register(NULL, -1, &vfio_container_vmstate, container); + /* During incoming CPR, divert calls to dma_map. */ + if (container->cpr.reused) { + VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + vioc->dma_map = vfio_legacy_cpr_dma_map; + } return true; } From patchwork Fri Feb 14 14:13:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B175C02198 for ; Fri, 14 Feb 2025 14:25:30 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT0-0004XU-2m; Fri, 14 Feb 2025 09:14:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSy-0004TB-Bs for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:52 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSw-0004T6-Om for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:52 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECu5LP019631; Fri, 14 Feb 2025 14:14:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=EtO+YcVnjLnfL11cilvDpps4BNkrAowKYxHloaLP1mw=; b= RwuKO06CF3rlxK8PbuWs87zGQbIPY7vaAe9RroUIh5lp1cZtWdsdHd58xLIKgXdX u5im8NMtz40IrsdGpFj14Gu8zrG2vbzvDn7t8xpnArVMSLys9je9sRVjhKnyin1d cPi52x8XHiAxrl+iFgh+t93c4JJMX54ESJzEqHliZBpwJ2P5Ais0UTmqWQ3mAYae U1hsSj74HrNacVKQokzcUPpw/hPG9JOK7EUNrjkFyqdslAcc75rsnm6cEerOGuyj dxLwXr8LSbbPbCIQELBP5oodvaXLOfShZnT9HYXVworJNRIB/VmZnw5hI9kUu739 6BcX/0AIAi8zyHNiur6+EQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0t4bmuq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:48 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECO40Q025170; Fri, 14 Feb 2025 14:14:47 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgaq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:47 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPd006920; Fri, 14 Feb 2025 14:14:46 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-14; Fri, 14 Feb 2025 14:14:46 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 13/45] vfio/container: mdev cpr blocker Date: Fri, 14 Feb 2025 06:13:55 -0800 Message-Id: <1739542467-226739-14-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: 4QRSWsh1ny2SdqA5hi92l0ZweJZmPn8j X-Proofpoint-GUID: 4QRSWsh1ny2SdqA5hi92l0ZweJZmPn8j Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During CPR, after VFIO_DMA_UNMAP_FLAG_VADDR, the vaddr is temporarily invalid, so mediated devices cannot be supported. Add a blocker for them. This restriction will not apply to iommufd containers when CPR is added for them in a future patch. Signed-off-by: Steve Sistare --- hw/vfio/container.c | 8 ++++++++ include/hw/vfio/vfio-cpr.h | 1 + 2 files changed, 9 insertions(+) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index d26f78e..8130d1f 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -999,6 +999,13 @@ static bool vfio_legacy_attach_device(const char *name, VFIODevice *vbasedev, return false; } + if (vbasedev->mdev) { + error_setg(&vbasedev->cpr.mdev_blocker, + "CPR does not support vfio mdev %s", vbasedev->name); + migrate_add_blocker_modes(&vbasedev->cpr.mdev_blocker, &error_fatal, + MIG_MODE_CPR_TRANSFER, -1); + } + bcontainer = &group->container->bcontainer; vbasedev->bcontainer = bcontainer; QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); @@ -1015,6 +1022,7 @@ static void vfio_legacy_detach_device(VFIODevice *vbasedev) QLIST_REMOVE(vbasedev, container_next); vbasedev->bcontainer = NULL; trace_vfio_detach_device(vbasedev->name, group->groupid); + migrate_del_blocker(&vbasedev->cpr.mdev_blocker); vfio_put_base_device(vbasedev); vfio_put_group(group); } diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index 1a3eee9..25ac944 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -17,6 +17,7 @@ typedef struct VFIOContainerCPR { typedef struct VFIODeviceCPR { bool reused; + Error *mdev_blocker; } VFIODeviceCPR; struct VFIOContainer; From patchwork Fri Feb 14 14:13:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51E62C021A6 for ; Fri, 14 Feb 2025 14:17:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT2-0004hY-6J; Fri, 14 Feb 2025 09:14:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSz-0004XB-PW for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:53 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSx-0004TF-K1 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:53 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtT8P028517; Fri, 14 Feb 2025 14:14:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=RkREt5x2dAQoSdbgHirMskiKHaxedX30dO32juP6jHs=; b= QmDtgAHOy6u9PKcZVH+Tfsp3IC99Nue00iK8fHqBWt/MSjmKd7SWiGyEZ0+s0CpJ UsS5BmjgvmqUxmOgvFzHL9aGu9HHTrZUFLKbTw6tBzwhLAs/OHf4RvO9pPQAwUxl 9BHYPc8IMDkwa+hEA+LikISj6ASrmk97qhBKyNeWF7yhfLw67kyp02cTOBuhIw8G vjZkzO856fWpbzY1v4sZY8hidqDoSRVFJyS6lgdD2K/K/ziPZLfho/xucZ+T/bFD gRrmpQHx6vnFKDbeLGhRqyBT3OKYQ4rPAVc9XONXyh5hCBZySHIXRXEpOHxq8rQ8 6DkUBzM4XAjvo54e8swi2A== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnn4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:48 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuXh7025256; Fri, 14 Feb 2025 14:14:48 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgbq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:48 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPf006920; Fri, 14 Feb 2025 14:14:47 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-15; Fri, 14 Feb 2025 14:14:47 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 14/45] vfio/container: recover from unmap-all-vaddr failure Date: Fri, 14 Feb 2025 06:13:56 -0800 Message-Id: <1739542467-226739-15-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: GTBkuUE2zJ9ZF2UfgadzmZqfTsjWMjJp X-Proofpoint-ORIG-GUID: GTBkuUE2zJ9ZF2UfgadzmZqfTsjWMjJp Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org If there are multiple containers and unmap-all fails for some container, we need to remap vaddr for the other containers for which unmap-all succeeded. Recover by walking all address ranges of all containers to restore the vaddr for each. Do so by invoking the vfio listener callback, and passing a new "remap" flag that tells it to restore a mapping without re-allocating new userland data structures. Signed-off-by: Steve Sistare --- hw/vfio/common.c | 19 ++++++++- hw/vfio/cpr-legacy.c | 91 +++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 3 ++ include/hw/vfio/vfio-cpr.h | 11 ++++++ 4 files changed, 123 insertions(+), 1 deletion(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index e3e1da0..48663ad 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -592,6 +592,13 @@ static void vfio_listener_region_add(MemoryListener *listener, { VFIOContainerBase *bcontainer = container_of(listener, VFIOContainerBase, listener); + vfio_container_region_add(bcontainer, section, false); +} + +void vfio_container_region_add(VFIOContainerBase *bcontainer, + MemoryRegionSection *section, + bool cpr_remap) +{ hwaddr iova, end; Int128 llend, llsize; void *vaddr; @@ -627,6 +634,11 @@ static void vfio_listener_region_add(MemoryListener *listener, int iommu_idx; trace_vfio_listener_region_add_iommu(section->mr->name, iova, end); + + if (cpr_remap) { + vfio_cpr_giommu_remap(bcontainer, section); + } + /* * FIXME: For VFIO iommu types which have KVM acceleration to * avoid bouncing all map/unmaps through qemu this way, this @@ -669,7 +681,12 @@ static void vfio_listener_region_add(MemoryListener *listener, * about changes. */ if (memory_region_has_ram_discard_manager(section->mr)) { - vfio_register_ram_discard_listener(bcontainer, section); + if (!cpr_remap) { + vfio_register_ram_discard_listener(bcontainer, section); + } else if (!vfio_cpr_register_ram_discard_listener(bcontainer, + section)) { + goto fail; + } return; } diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c index bb5f802..90f9f14 100644 --- a/hw/vfio/cpr-legacy.c +++ b/hw/vfio/cpr-legacy.c @@ -28,6 +28,7 @@ static bool vfio_dma_unmap_vaddr_all(VFIOContainer *container, Error **errp) error_setg_errno(errp, errno, "vfio_dma_unmap_vaddr_all"); return false; } + container->cpr.vaddr_unmapped = true; return true; } @@ -60,6 +61,14 @@ static int vfio_legacy_cpr_dma_map(const VFIOContainerBase *bcontainer, return 0; } +static void vfio_region_remap(MemoryListener *listener, + MemoryRegionSection *section) +{ + VFIOContainer *container = container_of(listener, VFIOContainer, + cpr.remap_listener); + vfio_container_region_add(&container->bcontainer, section, true); +} + static bool vfio_cpr_supported(VFIOContainer *container, Error **errp) { if (!ioctl(container->fd, VFIO_CHECK_EXTENSION, VFIO_UPDATE_VADDR)) { @@ -124,6 +133,40 @@ static const VMStateDescription vfio_container_vmstate = { } }; +static int vfio_cpr_fail_notifier(NotifierWithReturn *notifier, + MigrationEvent *e, Error **errp) +{ + VFIOContainer *container = + container_of(notifier, VFIOContainer, cpr.transfer_notifier); + VFIOContainerBase *bcontainer = &container->bcontainer; + + if (e->type != MIG_EVENT_PRECOPY_FAILED) { + return 0; + } + + if (container->cpr.vaddr_unmapped) { + /* + * Force a call to vfio_region_remap for each mapped section by + * temporarily registering a listener, and temporarily diverting + * dma_map to vfio_legacy_cpr_dma_map. The latter restores vaddr. + */ + + VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + vioc->dma_map = vfio_legacy_cpr_dma_map; + + container->cpr.remap_listener = (MemoryListener) { + .name = "vfio cpr recover", + .region_add = vfio_region_remap + }; + memory_listener_register(&container->cpr.remap_listener, + bcontainer->space->as); + memory_listener_unregister(&container->cpr.remap_listener); + container->cpr.vaddr_unmapped = false; + vioc->dma_map = vfio_legacy_dma_map; + } + return 0; +} + bool vfio_legacy_cpr_register_container(VFIOContainer *container, Error **errp) { VFIOContainerBase *bcontainer = &container->bcontainer; @@ -145,6 +188,10 @@ bool vfio_legacy_cpr_register_container(VFIOContainer *container, Error **errp) VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); vioc->dma_map = vfio_legacy_cpr_dma_map; } + + migration_add_notifier_mode(&container->cpr.transfer_notifier, + vfio_cpr_fail_notifier, + MIG_MODE_CPR_TRANSFER); return true; } @@ -155,6 +202,50 @@ void vfio_legacy_cpr_unregister_container(VFIOContainer *container) migration_remove_notifier(&bcontainer->cpr_reboot_notifier); migrate_del_blocker(&container->cpr.blocker); vmstate_unregister(NULL, &vfio_container_vmstate, container); + migration_remove_notifier(&container->cpr.transfer_notifier); +} + +/* + * In old QEMU, VFIO_DMA_UNMAP_FLAG_VADDR may fail on some mapping after + * succeeding for others, so the latter have lost their vaddr. Call this + * to restore vaddr for a section with a giommu. + * + * The giommu already exists. Find it and replay it, which calls + * vfio_legacy_cpr_dma_map further down the stack. + */ +void vfio_cpr_giommu_remap(VFIOContainerBase *bcontainer, + MemoryRegionSection *section) +{ + VFIOGuestIOMMU *giommu = NULL; + hwaddr as_offset = section->offset_within_address_space; + hwaddr iommu_offset = as_offset - section->offset_within_region; + + QLIST_FOREACH(giommu, &bcontainer->giommu_list, giommu_next) { + if (giommu->iommu_mr == IOMMU_MEMORY_REGION(section->mr) && + giommu->iommu_offset == iommu_offset) { + break; + } + } + g_assert(giommu); + memory_region_iommu_replay(giommu->iommu_mr, &giommu->n); +} + +/* + * In old QEMU, VFIO_DMA_UNMAP_FLAG_VADDR may fail on some mapping after + * succeeding for others, so the latter have lost their vaddr. Call this + * to restore vaddr for a section with a RamDiscardManager. + * + * The ram discard listener already exists. Call its populate function + * directly, which calls vfio_legacy_cpr_dma_map. + */ +bool vfio_cpr_register_ram_discard_listener(VFIOContainerBase *bcontainer, + MemoryRegionSection *section) +{ + VFIORamDiscardListener *vrdl = + vfio_find_ram_discard_listener(bcontainer, section); + + g_assert(vrdl); + return vrdl->listener.notify_populate(&vrdl->listener, section) == 0; } static bool same_device(int fd1, int fd2) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 7f33476..1563f3a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -315,6 +315,9 @@ int vfio_get_dirty_bitmap(const VFIOContainerBase *bcontainer, uint64_t iova, VFIORamDiscardListener *vfio_find_ram_discard_listener( VFIOContainerBase *bcontainer, MemoryRegionSection *section); +void vfio_container_region_add(VFIOContainerBase *bcontainer, + MemoryRegionSection *section, bool cpr_remap); + /* Returns 0 on success, or a negative errno. */ bool vfio_device_get_name(VFIODevice *vbasedev, Error **errp); void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp); diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index 25ac944..40d4f06 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -8,11 +8,15 @@ #ifndef HW_VFIO_VFIO_CPR_H #define HW_VFIO_VFIO_CPR_H +#include "exec/memory.h" #include "migration/misc.h" typedef struct VFIOContainerCPR { Error *blocker; bool reused; + bool vaddr_unmapped; + NotifierWithReturn transfer_notifier; + MemoryListener remap_listener; } VFIOContainerCPR; typedef struct VFIODeviceCPR { @@ -22,6 +26,7 @@ typedef struct VFIODeviceCPR { struct VFIOContainer; struct VFIOGroup; +struct VFIOContainerBase; int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp); @@ -32,4 +37,10 @@ void vfio_legacy_cpr_unregister_container(struct VFIOContainer *container); bool vfio_cpr_container_match(struct VFIOContainer *container, struct VFIOGroup *group, int *fd); + +void vfio_cpr_giommu_remap(struct VFIOContainerBase *bcontainer, + MemoryRegionSection *section); + +bool vfio_cpr_register_ram_discard_listener( + struct VFIOContainerBase *bcontainer, MemoryRegionSection *section); #endif From patchwork Fri Feb 14 14:13:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974968 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79A54C021A6 for ; Fri, 14 Feb 2025 14:15:06 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT3-0004mJ-To; Fri, 14 Feb 2025 09:14:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT1-0004bo-8L for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:55 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwSz-0004Ta-Je for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:55 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtT3i022092; Fri, 14 Feb 2025 14:14:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=VpyJXhAsaPJ8USvuLDCkdI/slPrwul3k+jzms2oo3CU=; b= V6LNBfVPP0U32i0hSNJP6IzT8oTLEgxOENb5+ujKARayL28JLY7PbIuxFrNnRDA2 DF6R5+DvQLEa0DNHMAL5IftzJ9EVVOxeKvUiwW7G5xK9TaBsdV92HD4QitYdysQT cVEXisO6oI3F39LYHj7yor4arDkPIYmzUnmqpBU98SLvBwbm6RLBoDRXotlQk1yQ 8G/P+sjaceubiz1HF/GsWLp5mDdR0FCC/OL49ZMENRjenROezgyNjG5yNmoIubf6 ZsHMUAVzyyL9FnnEZAqdkHRrrXamYBfFGBRJHqzfyCbOumUVLQuDHGfeApfeEBvc XPwhBCspbSlsUCqQZp7YFg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0s43q6q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:51 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ED0mAE025265; Fri, 14 Feb 2025 14:14:49 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgcn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:49 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPh006920; Fri, 14 Feb 2025 14:14:49 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-16; Fri, 14 Feb 2025 14:14:49 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 15/45] pci: export msix_is_pending Date: Fri, 14 Feb 2025 06:13:57 -0800 Message-Id: <1739542467-226739-16-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: EsNNpvMMLkh2tOcekt-Jb-KYBXydNZ_g X-Proofpoint-ORIG-GUID: EsNNpvMMLkh2tOcekt-Jb-KYBXydNZ_g Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Export msix_is_pending for use by cpr. No functional change. Signed-off-by: Steve Sistare Acked-by: Michael S. Tsirkin --- hw/pci/msix.c | 2 +- include/hw/pci/msix.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/pci/msix.c b/hw/pci/msix.c index 57ec708..c7b40cd 100644 --- a/hw/pci/msix.c +++ b/hw/pci/msix.c @@ -71,7 +71,7 @@ static uint8_t *msix_pending_byte(PCIDevice *dev, int vector) return dev->msix_pba + vector / 8; } -static int msix_is_pending(PCIDevice *dev, int vector) +int msix_is_pending(PCIDevice *dev, unsigned int vector) { return *msix_pending_byte(dev, vector) & msix_pending_mask(vector); } diff --git a/include/hw/pci/msix.h b/include/hw/pci/msix.h index 0e6f257..11ef945 100644 --- a/include/hw/pci/msix.h +++ b/include/hw/pci/msix.h @@ -32,6 +32,7 @@ int msix_present(PCIDevice *dev); bool msix_is_masked(PCIDevice *dev, unsigned vector); void msix_set_pending(PCIDevice *dev, unsigned vector); void msix_clr_pending(PCIDevice *dev, int vector); +int msix_is_pending(PCIDevice *dev, unsigned vector); void msix_vector_use(PCIDevice *dev, unsigned vector); void msix_vector_unuse(PCIDevice *dev, unsigned vector); From patchwork Fri Feb 14 14:13:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A465C021A9 for ; Fri, 14 Feb 2025 14:28:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT5-0004pk-By; Fri, 14 Feb 2025 09:14:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT1-0004gj-R6 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:55 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT0-0004Tl-Bp for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:55 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtRtO011440; Fri, 14 Feb 2025 14:14:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=KyN/R71FgJE3qkLYjR8uFeEQ7zzFxTKeAKoto5Q5KiQ=; b= C/M7paGCxofLnKOenoaj0H9PCZg/Ba7AWUBg/Xra8lUmqusR9BtEOPOppOXu/zql 5tM7ZkE+mfT1JYdDkHdwfs98Aq999Q5wEJ0K5fPWc4d5wdNj7JaWFux7Ax1f+IbS o7jYGYOmXmQsJZA4vOUd5Ehcd1u+FP1bdPrMZZwZPuyouh730vuLhFqP8eX9Rgtt EnohQO1G2rSYgHlERQWkEA//PIbRH0uYVNNytVwU4wBgpr9LfFpyplYKyXkiwRDY UXCTioz+sPS1ThzAHLM6VXyneArQQNOftUuuqaRjpYNXOHMDFchiaHCzBbi+ewLY UQAyQ7+5DxvlbBV27xuAOQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwmj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:51 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECfZOA025148; Fri, 14 Feb 2025 14:14:51 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgdd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:51 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPj006920; Fri, 14 Feb 2025 14:14:50 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-17; Fri, 14 Feb 2025 14:14:50 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 16/45] pci: skip reset during cpr Date: Fri, 14 Feb 2025 06:13:58 -0800 Message-Id: <1739542467-226739-17-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: qkVX94VQ7JJZ5M4gdyaaN1JlzmGDyShi X-Proofpoint-GUID: qkVX94VQ7JJZ5M4gdyaaN1JlzmGDyShi Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Do not reset a vfio-pci device during CPR. Signed-off-by: Steve Sistare --- hw/pci/pci.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/hw/pci/pci.c b/hw/pci/pci.c index 2afa423..2fa8884 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -32,6 +32,8 @@ #include "hw/pci/pci_host.h" #include "hw/qdev-properties.h" #include "hw/qdev-properties-system.h" +#include "migration/cpr.h" +#include "migration/misc.h" #include "migration/qemu-file-types.h" #include "migration/vmstate.h" #include "net/net.h" @@ -459,6 +461,17 @@ static void pci_reset_regions(PCIDevice *dev) static void pci_do_device_reset(PCIDevice *dev) { + /* + * A PCI device that is resuming for cpr is already configured, so do + * not reset it here when we are called from qemu_system_reset prior to + * cpr load, else interrupts may be lost for vfio-pci devices. It is + * safe to skip this reset for all PCI devices, because vmstate load will + * set all fields that would have been set here. + */ + if (cpr_is_incoming()) { + return; + } + pci_device_deassert_intx(dev); assert(dev->irq_state == 0); From patchwork Fri Feb 14 14:13:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3027CC021A4 for ; Fri, 14 Feb 2025 14:28:57 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT5-0004pV-2q; Fri, 14 Feb 2025 09:14:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT3-0004kl-GK for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:57 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT1-0004Tx-Up for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:57 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtTl1001915; Fri, 14 Feb 2025 14:14:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=ndZ8Cn9lMpheRCsaXrX8Fh4pS07p8vIhpbsG5gRn4uc=; b= cDVgnDQG+peFIcKXakDm5uaocPcTm6OQWDZqA+puyvYD35aWEhK2cDYd4FcbSZDT znKSGMZA5Ezg2Q44LLeUUi+auTVhALLxJ2JpMkvx42n1nIPdE+FQIrkVIfTLvqDt bEXrjzwZXnkBaV5fyd24qF6XxkVhW0JQMMfkZIMqorMHRHD672GNbcHFNpvnflfG ciLx6v8zgD4eqCxdNYaZH+U0DVy8W6omVT12fqtrzPP0yFzM8tPj56r0kGTEffaj 41yo5mUxRluhVACqaMqgMra2b0ffNK+aV1F8JDNNvfgbmCxVJn7ZXe45QjCtnpw5 TQELILrV+ZBemP3xkKQf/Q== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyusvp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:52 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECSfbh025328; Fri, 14 Feb 2025 14:14:52 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksge6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:52 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPl006920; Fri, 14 Feb 2025 14:14:51 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-18; Fri, 14 Feb 2025 14:14:51 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 17/45] vfio-pci: skip reset during cpr Date: Fri, 14 Feb 2025 06:13:59 -0800 Message-Id: <1739542467-226739-18-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: LJYNiIca4hX00iG9zTBfqxk8iJHfuvSO X-Proofpoint-GUID: LJYNiIca4hX00iG9zTBfqxk8iJHfuvSO Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Do not reset a vfio-pci device during CPR, and do not complain if the kernel's PCI config space changes for non-emulated bits between the vmstate save and load, which can happen due to ongoing interrupt activity. Signed-off-by: Steve Sistare --- hw/vfio/cpr.c | 31 +++++++++++++++++++++++++++++++ hw/vfio/pci.c | 6 ++++++ include/hw/vfio/vfio-cpr.h | 2 ++ 3 files changed, 39 insertions(+) diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index 6790f8a..8268c0c 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -8,6 +8,8 @@ #include "qemu/osdep.h" #include "hw/vfio/vfio-common.h" #include "hw/vfio/vfio-cpr.h" +#include "hw/vfio/pci.h" +#include "migration/cpr.h" #include "qapi/error.h" #include "system/runstate.h" @@ -37,3 +39,32 @@ void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer) { migration_remove_notifier(&bcontainer->cpr_reboot_notifier); } + +/* + * The kernel may change non-emulated config bits. Exclude them from the + * changed-bits check in get_pci_config_device. + */ +static int vfio_cpr_pci_pre_load(void *opaque) +{ + VFIOPCIDevice *vdev = opaque; + PCIDevice *pdev = &vdev->pdev; + int size = MIN(pci_config_size(pdev), vdev->config_size); + int i; + + for (i = 0; i < size; i++) { + pdev->cmask[i] &= vdev->emulated_config_bits[i]; + } + + return 0; +} + +const VMStateDescription vfio_cpr_pci_vmstate = { + .name = "vfio-cpr-pci", + .version_id = 0, + .minimum_version_id = 0, + .pre_load = vfio_cpr_pci_pre_load, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 89d900e..bd080ea 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3294,6 +3294,11 @@ static void vfio_pci_reset(DeviceState *dev) { VFIOPCIDevice *vdev = VFIO_PCI(dev); + /* Do not reset the device during qemu_system_reset prior to cpr load */ + if (vdev->vbasedev.cpr.reused) { + return; + } + trace_vfio_pci_reset(vdev->vbasedev.name); vfio_pci_pre_reset(vdev); @@ -3427,6 +3432,7 @@ static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) #ifdef CONFIG_IOMMUFD object_class_property_add_str(klass, "fd", NULL, vfio_pci_set_fd); #endif + dc->vmsd = &vfio_cpr_pci_vmstate; dc->desc = "VFIO-based PCI device assignment"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); pdc->realize = vfio_realize; diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index 40d4f06..f5480de 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -43,4 +43,6 @@ void vfio_cpr_giommu_remap(struct VFIOContainerBase *bcontainer, bool vfio_cpr_register_ram_discard_listener( struct VFIOContainerBase *bcontainer, MemoryRegionSection *section); + +extern const VMStateDescription vfio_cpr_pci_vmstate; #endif From patchwork Fri Feb 14 14:14:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09F4FC021A6 for ; Fri, 14 Feb 2025 14:28:40 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwT8-000538-4f; Fri, 14 Feb 2025 09:15:02 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT5-0004rm-7I for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:59 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT3-0004U8-FW for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:14:58 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECu5LQ019631; Fri, 14 Feb 2025 14:14:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=ko8/DK9KUJwwd4C3swGa6X4nnN0vZ6lduituR25pmU8=; b= U79z/icnWhCTC0mt/IiliUJOKu/NKyRUPpVhj/H8pNFvcDz9hM6HtyTsxsUsMcn4 8cLZPXzb/JmL9LTcF6ws8ba36bPKyNTanXtHmbW5OOI6x5AtroWzNHMTGBMfDA1E D7GTDyiQhNWj9o5GQ9oj9ePCGnR/XLebhk0oTiQTaPoTIMc8HpcpQBPxcspEbaqD +41BoW/c58+KOWyhKZt/I2WmysDxqCD/neqKcthaNGB13F2pJGgbsSTAMDz9Wzhn VWLDgLBUjcyPJqHTFhMnOEIdmbkBdB1Ikx2teEayGBVs20jXLfs6WlfR2NiiVrgS Q1jaaa39jqgxf1KTvmgD2A== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0t4bmuu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:55 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECelZ1025288; Fri, 14 Feb 2025 14:14:53 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgeq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:53 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPn006920; Fri, 14 Feb 2025 14:14:53 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-19; Fri, 14 Feb 2025 14:14:53 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 18/45] vfio/pci: vfio_vector_init Date: Fri, 14 Feb 2025 06:14:00 -0800 Message-Id: <1739542467-226739-19-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: 7kznKiFb5N7AsgH2z8Rhi9HQq1yJ0gY0 X-Proofpoint-GUID: 7kznKiFb5N7AsgH2z8Rhi9HQq1yJ0gY0 Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Extract a subroutine vfio_vector_init. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bd080ea..883257a 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -511,6 +511,22 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, kvm_irqchip_commit_routes(kvm_state); } +static void vfio_vector_init(VFIOPCIDevice *vdev, int nr) +{ + VFIOMSIVector *vector = &vdev->msi_vectors[nr]; + PCIDevice *pdev = &vdev->pdev; + + vector->vdev = vdev; + vector->virq = -1; + if (event_notifier_init(&vector->interrupt, 0)) { + error_report("vfio: Error: event_notifier_init failed"); + } + vector->use = true; + if (vdev->interrupt == VFIO_INT_MSIX) { + msix_vector_use(pdev, nr); + } +} + static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, MSIMessage *msg, IOHandler *handler) { @@ -524,13 +540,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, vector = &vdev->msi_vectors[nr]; if (!vector->use) { - vector->vdev = vdev; - vector->virq = -1; - if (event_notifier_init(&vector->interrupt, 0)) { - error_report("vfio: Error: event_notifier_init failed"); - } - vector->use = true; - msix_vector_use(pdev, nr); + vfio_vector_init(vdev, nr); } qemu_set_fd_handler(event_notifier_get_fd(&vector->interrupt), From patchwork Fri Feb 14 14:14:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975022 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0047DC021A6 for ; Fri, 14 Feb 2025 14:25:30 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTR-0005tU-05; Fri, 14 Feb 2025 09:15:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTG-0005SU-Sl for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:10 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTE-0004UO-TJ for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:10 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtqaT022708; Fri, 14 Feb 2025 14:14:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=2mDboa9BzWXJq3YMdxxt/EVvHBYpzTQW6ZaFp5O6CRY=; b= Ojnp9x/phf6rxFX1bL6AgQ4Ldz6+v2IBfoeWsC5BgyUwZdgUsUmNpv/Kjivntn7j 7sXeDsz5zGJZbfiTESxjyfR6GgeNOIJ/b/SIBspX1rJbgBZHc/vr9QuSN+UfGsit QeccnMLHJrabmB2cZYlNC2IdyIWnp2bJrs7+dAJ/aPqjAS5mc3of6WqGrjs8odnY OKu1dX8VfGuS4adOVPWruYyt4TroP75tBxjXMcdzdn20mzsXyQX64cprlanXzGhZ eDvpf/XqDDIOInr5Zaxhv0XMUUZMzKtTXyOs24us2UpVChX5hf3tScDxge7jWNrj fve2IFGhnjcjlw1CiWeqdQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0s43q6u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:56 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECsnOL025169; Fri, 14 Feb 2025 14:14:55 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgff-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:55 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPp006920; Fri, 14 Feb 2025 14:14:54 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-20; Fri, 14 Feb 2025 14:14:54 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 19/45] vfio/pci: vfio_notifier_init Date: Fri, 14 Feb 2025 06:14:01 -0800 Message-Id: <1739542467-226739-20-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: r0pZRckNK6Eu0BYXlfyKfB-75GXLXBR- X-Proofpoint-ORIG-GUID: r0pZRckNK6Eu0BYXlfyKfB-75GXLXBR- Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Move event_notifier_init calls to a helper vfio_notifier_init. This version is trivial, but it will be expanded to support CPR in subsequent patches. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 883257a..688b7d3 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -54,6 +54,16 @@ static void vfio_disable_interrupts(VFIOPCIDevice *vdev); static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled); static void vfio_msi_disable_common(VFIOPCIDevice *vdev); +static bool vfio_notifier_init(EventNotifier *e, const char *name, Error **errp) +{ + int ret = event_notifier_init(e, 0); + + if (ret) { + error_setg_errno(errp, -ret, "vfio_notifier_init %s failed", name); + } + return !ret; +} + /* * Disabling BAR mmaping can be slow, but toggling it around INTx can * also be a huge overhead. We try to get the best of both worlds by @@ -134,8 +144,7 @@ static bool vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) pci_irq_deassert(&vdev->pdev); /* Get an eventfd for resample/unmask */ - if (event_notifier_init(&vdev->intx.unmask, 0)) { - error_setg(errp, "event_notifier_init failed eoi"); + if (!vfio_notifier_init(&vdev->intx.unmask, "intx-unmask", errp)) { goto fail; } @@ -266,7 +275,6 @@ static bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) uint8_t pin = vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1); Error *err = NULL; int32_t fd; - int ret; if (!pin) { @@ -289,9 +297,7 @@ static bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) } #endif - ret = event_notifier_init(&vdev->intx.interrupt, 0); - if (ret) { - error_setg_errno(errp, -ret, "event_notifier_init failed"); + if (!vfio_notifier_init(&vdev->intx.interrupt, "intx-interrupt", errp)) { return false; } fd = event_notifier_get_fd(&vdev->intx.interrupt); @@ -473,11 +479,13 @@ static void vfio_add_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, static void vfio_connect_kvm_msi_virq(VFIOMSIVector *vector) { + const char *name = "kvm_interrupt"; + if (vector->virq < 0) { return; } - if (event_notifier_init(&vector->kvm_interrupt, 0)) { + if (!vfio_notifier_init(&vector->kvm_interrupt, name, NULL)) { goto fail_notifier; } @@ -515,11 +523,12 @@ static void vfio_vector_init(VFIOPCIDevice *vdev, int nr) { VFIOMSIVector *vector = &vdev->msi_vectors[nr]; PCIDevice *pdev = &vdev->pdev; + Error *err = NULL; vector->vdev = vdev; vector->virq = -1; - if (event_notifier_init(&vector->interrupt, 0)) { - error_report("vfio: Error: event_notifier_init failed"); + if (!vfio_notifier_init(&vector->interrupt, "interrupt", &err)) { + error_report_err(err); } vector->use = true; if (vdev->interrupt == VFIO_INT_MSIX) { @@ -746,13 +755,14 @@ retry: for (i = 0; i < vdev->nr_vectors; i++) { VFIOMSIVector *vector = &vdev->msi_vectors[i]; + Error *err = NULL; vector->vdev = vdev; vector->virq = -1; vector->use = true; - if (event_notifier_init(&vector->interrupt, 0)) { - error_report("vfio: Error: event_notifier_init failed"); + if (!vfio_notifier_init(&vector->interrupt, "interrupt", &err)) { + error_report_err(err); } qemu_set_fd_handler(event_notifier_get_fd(&vector->interrupt), @@ -2864,8 +2874,8 @@ static void vfio_register_err_notifier(VFIOPCIDevice *vdev) return; } - if (event_notifier_init(&vdev->err_notifier, 0)) { - error_report("vfio: Unable to init event notifier for error detection"); + if (!vfio_notifier_init(&vdev->err_notifier, "err_notifier", &err)) { + error_report_err(err); vdev->pci_aer = false; return; } @@ -2930,8 +2940,8 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) return; } - if (event_notifier_init(&vdev->req_notifier, 0)) { - error_report("vfio: Unable to init event notifier for device request"); + if (!vfio_notifier_init(&vdev->req_notifier, "req_notifier", &err)) { + error_report_err(err); return; } From patchwork Fri Feb 14 14:14:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A35DBC021A4 for ; Fri, 14 Feb 2025 14:17:45 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTC-0005EK-5f; Fri, 14 Feb 2025 09:15:06 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT8-00056P-9B for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:02 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT6-0004Ue-K6 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:02 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECteQB002102; Fri, 14 Feb 2025 14:14:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=1TLnU8WpA/v27WOQ3L6Dkn5Vvy+ojGBtj2U5fbK8qRo=; b= j2IyHA7t9QqvkcM7lWGS5mDNu8j4Bq7rjlR2DtR9zT/hZvu5xGCLyTeEbj1PekOF 6FbAX9MtXoMdiYgtRZclgDS9C++jXP9g5xu9EVtmAGturyO4xG107Zl+23hjlBwc aOpBHQmVEmkb8pdPGHttn3pG6viFj9F7ozwuosR2H9VjddV/p9jiSzdrPHiu55vj tfCQIevmMG67ZheoVlq+NERdZlLFaW35BnJxYyWPBrgJ8Ef9u36Vw7i+VOQrsHx1 hgyKQLzGDh1g/39gQUs0KVWHOuqScnXJQWNaz6Rl1rvtKeu38NlbAmZecp7NXYHB MKEWz3K7Nmk1zh8FCAZ4cg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyusvt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:56 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECog1I025151; Fri, 14 Feb 2025 14:14:56 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgg8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:56 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPr006920; Fri, 14 Feb 2025 14:14:55 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-21; Fri, 14 Feb 2025 14:14:55 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 20/45] vfio/pci: pass vector to virq functions Date: Fri, 14 Feb 2025 06:14:02 -0800 Message-Id: <1739542467-226739-21-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: jJH0Nzfuc1Eqh9o82nAMlzOLf74cQ3yx X-Proofpoint-GUID: jJH0Nzfuc1Eqh9o82nAMlzOLf74cQ3yx Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Pass the vector number to vfio_connect_kvm_msi_virq and vfio_remove_kvm_msi_virq, so it can be passed to their subroutines in a subsequent patch. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 688b7d3..7b2d185 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -477,7 +477,7 @@ static void vfio_add_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, vector_n, &vdev->pdev); } -static void vfio_connect_kvm_msi_virq(VFIOMSIVector *vector) +static void vfio_connect_kvm_msi_virq(VFIOMSIVector *vector, int nr) { const char *name = "kvm_interrupt"; @@ -503,7 +503,8 @@ fail_notifier: vector->virq = -1; } -static void vfio_remove_kvm_msi_virq(VFIOMSIVector *vector) +static void vfio_remove_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, + int nr) { kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vector->kvm_interrupt, vector->virq); @@ -561,7 +562,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, */ if (vector->virq >= 0) { if (!msg) { - vfio_remove_kvm_msi_virq(vector); + vfio_remove_kvm_msi_virq(vdev, vector, nr); } else { vfio_update_kvm_msi_virq(vector, *msg, pdev); } @@ -573,7 +574,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, vfio_route_change = kvm_irqchip_begin_route_changes(kvm_state); vfio_add_kvm_msi_virq(vdev, vector, nr, true); kvm_irqchip_commit_route_changes(&vfio_route_change); - vfio_connect_kvm_msi_virq(vector); + vfio_connect_kvm_msi_virq(vector, nr); } } } @@ -680,7 +681,7 @@ static void vfio_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev) kvm_irqchip_commit_route_changes(&vfio_route_change); for (i = 0; i < vdev->nr_vectors; i++) { - vfio_connect_kvm_msi_virq(&vdev->msi_vectors[i]); + vfio_connect_kvm_msi_virq(&vdev->msi_vectors[i], i); } } @@ -817,7 +818,7 @@ static void vfio_msi_disable_common(VFIOPCIDevice *vdev) VFIOMSIVector *vector = &vdev->msi_vectors[i]; if (vdev->msi_vectors[i].use) { if (vector->virq >= 0) { - vfio_remove_kvm_msi_virq(vector); + vfio_remove_kvm_msi_virq(vdev, vector, i); } qemu_set_fd_handler(event_notifier_get_fd(&vector->interrupt), NULL, NULL, NULL); From patchwork Fri Feb 14 14:14:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7CD7DC021A4 for ; Fri, 14 Feb 2025 14:17:10 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTP-0005me-Mz; Fri, 14 Feb 2025 09:15:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTK-0005Zh-7L for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:14 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTH-0004Uk-F4 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:13 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtUQv017777; Fri, 14 Feb 2025 14:14:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=PmsytVwoBKbySCvQqfdLb/N4Rogz3pa//tRNboKj3SM=; b= Hsnx6mi8GygqiUDZ5WZEqZ4Gp3urAm5Q+AGwQuf+gHBxTgdXO43L2+nFp8vTpJxe nEa+GCLnrFDq2aPbLTwg4tr3j6rwqkhzWu4vd8ox+C/V92v72S3PeSv8p2X31GQH ZQVAIhu10Nx4PKeK07JSIRW+7PbekhCTRqibWSTm+cDo+qwF2SK0FU+YAKwBIi4k wAPGGohxV4j4//VNscdfHnHPAycLnWNq2djK9V7MGYkCKE5uvORlhNN7c5JHExdH EbObVDD/kxfFF7x+wVszLb6u+Q3s23MycW1ZOdsR1jd/vibDMGREW5Uh007up2ZB ZNVdKHbCfG0DoWVVdIohBg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tgbtw8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:59 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECelZ4025288; Fri, 14 Feb 2025 14:14:57 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgh4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:57 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPt006920; Fri, 14 Feb 2025 14:14:57 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-22; Fri, 14 Feb 2025 14:14:57 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 21/45] vfio/pci: vfio_notifier_init cpr parameters Date: Fri, 14 Feb 2025 06:14:03 -0800 Message-Id: <1739542467-226739-22-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: zsOkuO0ATAr91Zpl0RhGy27sF95gtiIf X-Proofpoint-ORIG-GUID: zsOkuO0ATAr91Zpl0RhGy27sF95gtiIf Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Pass vdev and nr to vfio_notifier_init, for use by CPR in a subsequent patch. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7b2d185..63fb2d9 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -54,7 +54,8 @@ static void vfio_disable_interrupts(VFIOPCIDevice *vdev); static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled); static void vfio_msi_disable_common(VFIOPCIDevice *vdev); -static bool vfio_notifier_init(EventNotifier *e, const char *name, Error **errp) +static bool vfio_notifier_init(VFIOPCIDevice *vdev, EventNotifier *e, + const char *name, int nr, Error **errp) { int ret = event_notifier_init(e, 0); @@ -144,7 +145,7 @@ static bool vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) pci_irq_deassert(&vdev->pdev); /* Get an eventfd for resample/unmask */ - if (!vfio_notifier_init(&vdev->intx.unmask, "intx-unmask", errp)) { + if (!vfio_notifier_init(vdev, &vdev->intx.unmask, "intx-unmask", 0, errp)) { goto fail; } @@ -297,7 +298,8 @@ static bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) } #endif - if (!vfio_notifier_init(&vdev->intx.interrupt, "intx-interrupt", errp)) { + if (!vfio_notifier_init(vdev, &vdev->intx.interrupt, "intx-interrupt", 0, + errp)) { return false; } fd = event_notifier_get_fd(&vdev->intx.interrupt); @@ -485,7 +487,8 @@ static void vfio_connect_kvm_msi_virq(VFIOMSIVector *vector, int nr) return; } - if (!vfio_notifier_init(&vector->kvm_interrupt, name, NULL)) { + if (!vfio_notifier_init(vector->vdev, &vector->kvm_interrupt, name, nr, + NULL)) { goto fail_notifier; } @@ -528,7 +531,7 @@ static void vfio_vector_init(VFIOPCIDevice *vdev, int nr) vector->vdev = vdev; vector->virq = -1; - if (!vfio_notifier_init(&vector->interrupt, "interrupt", &err)) { + if (!vfio_notifier_init(vdev, &vector->interrupt, "interrupt", nr, &err)) { error_report_err(err); } vector->use = true; @@ -762,7 +765,8 @@ retry: vector->virq = -1; vector->use = true; - if (!vfio_notifier_init(&vector->interrupt, "interrupt", &err)) { + if (!vfio_notifier_init(vdev, &vector->interrupt, "interrupt", i, + &err)) { error_report_err(err); } @@ -2875,7 +2879,8 @@ static void vfio_register_err_notifier(VFIOPCIDevice *vdev) return; } - if (!vfio_notifier_init(&vdev->err_notifier, "err_notifier", &err)) { + if (!vfio_notifier_init(vdev, &vdev->err_notifier, "err_notifier", 0, + &err)) { error_report_err(err); vdev->pci_aer = false; return; @@ -2941,7 +2946,8 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) return; } - if (!vfio_notifier_init(&vdev->req_notifier, "req_notifier", &err)) { + if (!vfio_notifier_init(vdev, &vdev->req_notifier, "req_notifier", 0, + &err)) { error_report_err(err); return; } From patchwork Fri Feb 14 14:14:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46CDFC021A7 for ; Fri, 14 Feb 2025 14:24:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTE-0005N6-VZ; Fri, 14 Feb 2025 09:15:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTA-0005EU-JL for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:05 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwT8-0004V6-F2 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:04 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtTtM028522; Fri, 14 Feb 2025 14:15:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=bJnLeOJlO2ZA8TsTSPMcLslRWPR7DDVqPrvhnMk9Gp4=; b= jM84L1iKzkG6pksV3toKmUeEN8FCZbeRIIaewfsGr5EVXhzafryVip4+BC5Uw0Og e8N6PWeW349ezViJdLbINUjoqyYnMKEd6Fn03wZEoFu+mYwg7RqLWWaRskZeOQBV svMwEzMKlxLwZgC3nDWKiSsINdMbzfuFq40tONpHGLgyGNf961csfdGewm72jlTy APavCPuDdaXxeByzyQN2ok1b5qkHhzvLlvOBG9ETmWmv1KYywmAoBHORpDHESI5h NLsa9iCggIMwI99ucL3xmvkerGX6D4mNQnxAT9o+qZx0JhHBMd6SmYYtdv5lZA6l 2CCAt4Y5bjCVrgevMouP4Q== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnnm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:59 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuXhD025256; Fri, 14 Feb 2025 14:14:59 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksghv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:14:59 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPv006920; Fri, 14 Feb 2025 14:14:58 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-23; Fri, 14 Feb 2025 14:14:58 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 22/45] vfio/pci: vfio_notifier_cleanup Date: Fri, 14 Feb 2025 06:14:04 -0800 Message-Id: <1739542467-226739-23-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: DUeAR4gnmpLJRWnrrpCyaY6VwjPqlGi7 X-Proofpoint-ORIG-GUID: DUeAR4gnmpLJRWnrrpCyaY6VwjPqlGi7 Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Move event_notifier_cleanup calls to a helper vfio_notifier_cleanup. This version is trivial, and does not yet use the vdev and nr parameters. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 63fb2d9..fd51b9e 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -65,6 +65,12 @@ static bool vfio_notifier_init(VFIOPCIDevice *vdev, EventNotifier *e, return !ret; } +static void vfio_notifier_cleanup(VFIOPCIDevice *vdev, EventNotifier *e, + const char *name, int nr) +{ + event_notifier_cleanup(e); +} + /* * Disabling BAR mmaping can be slow, but toggling it around INTx can * also be a huge overhead. We try to get the best of both worlds by @@ -177,7 +183,7 @@ fail_vfio: kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vdev->intx.interrupt, vdev->intx.route.irq); fail_irqfd: - event_notifier_cleanup(&vdev->intx.unmask); + vfio_notifier_cleanup(vdev, &vdev->intx.unmask, "intx-unmask", 0); fail: qemu_set_fd_handler(irq_fd, vfio_intx_interrupt, NULL, vdev); vfio_unmask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); @@ -209,7 +215,7 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) } /* We only need to close the eventfd for VFIO to cleanup the kernel side */ - event_notifier_cleanup(&vdev->intx.unmask); + vfio_notifier_cleanup(vdev, &vdev->intx.unmask, "intx-unmask", 0); /* QEMU starts listening for interrupt events. */ qemu_set_fd_handler(event_notifier_get_fd(&vdev->intx.interrupt), @@ -308,7 +314,7 @@ static bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) if (!vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_TRIGGER, fd, errp)) { qemu_set_fd_handler(fd, NULL, NULL, vdev); - event_notifier_cleanup(&vdev->intx.interrupt); + vfio_notifier_cleanup(vdev, &vdev->intx.interrupt, "intx-interrupt", 0); return false; } @@ -335,7 +341,7 @@ static void vfio_intx_disable(VFIOPCIDevice *vdev) fd = event_notifier_get_fd(&vdev->intx.interrupt); qemu_set_fd_handler(fd, NULL, NULL, vdev); - event_notifier_cleanup(&vdev->intx.interrupt); + vfio_notifier_cleanup(vdev, &vdev->intx.interrupt, "intx-interrupt", 0); vdev->interrupt = VFIO_INT_NONE; @@ -500,7 +506,7 @@ static void vfio_connect_kvm_msi_virq(VFIOMSIVector *vector, int nr) return; fail_kvm: - event_notifier_cleanup(&vector->kvm_interrupt); + vfio_notifier_cleanup(vector->vdev, &vector->kvm_interrupt, name, nr); fail_notifier: kvm_irqchip_release_virq(kvm_state, vector->virq); vector->virq = -1; @@ -513,7 +519,7 @@ static void vfio_remove_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, vector->virq); kvm_irqchip_release_virq(kvm_state, vector->virq); vector->virq = -1; - event_notifier_cleanup(&vector->kvm_interrupt); + vfio_notifier_cleanup(vdev, &vector->kvm_interrupt, "kvm_interrupt", nr); } static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, @@ -826,7 +832,7 @@ static void vfio_msi_disable_common(VFIOPCIDevice *vdev) } qemu_set_fd_handler(event_notifier_get_fd(&vector->interrupt), NULL, NULL, NULL); - event_notifier_cleanup(&vector->interrupt); + vfio_notifier_cleanup(vdev, &vector->interrupt, "interrupt", i); } } @@ -2893,7 +2899,7 @@ static void vfio_register_err_notifier(VFIOPCIDevice *vdev) VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); qemu_set_fd_handler(fd, NULL, NULL, vdev); - event_notifier_cleanup(&vdev->err_notifier); + vfio_notifier_cleanup(vdev, &vdev->err_notifier, "err_notifier", 0); vdev->pci_aer = false; } } @@ -2912,7 +2918,7 @@ static void vfio_unregister_err_notifier(VFIOPCIDevice *vdev) } qemu_set_fd_handler(event_notifier_get_fd(&vdev->err_notifier), NULL, NULL, vdev); - event_notifier_cleanup(&vdev->err_notifier); + vfio_notifier_cleanup(vdev, &vdev->err_notifier, "err_notifier", 0); } static void vfio_req_notifier_handler(void *opaque) @@ -2959,7 +2965,7 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); qemu_set_fd_handler(fd, NULL, NULL, vdev); - event_notifier_cleanup(&vdev->req_notifier); + vfio_notifier_cleanup(vdev, &vdev->req_notifier, "req_notifier", 0); } else { vdev->req_enabled = true; } @@ -2979,7 +2985,7 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) } qemu_set_fd_handler(event_notifier_get_fd(&vdev->req_notifier), NULL, NULL, vdev); - event_notifier_cleanup(&vdev->req_notifier); + vfio_notifier_cleanup(vdev, &vdev->req_notifier, "req_notifier", 0); vdev->req_enabled = false; } From patchwork Fri Feb 14 14:14:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C51A2C021A4 for ; Fri, 14 Feb 2025 14:15:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTJ-0005SP-56; Fri, 14 Feb 2025 09:15:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTD-0005J1-4i for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:07 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTA-0004bh-Mq for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:06 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtShS001883; Fri, 14 Feb 2025 14:15:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=/6XhSvv3sFczrEGfq/gelvALvNCL1zXAoPUq7EQ3ALE=; b= LvMVY28nEl88v4juH7obTiLHs0OvCx6s9FlkkJcpQ/UWVKV5YJNfJHd4zLOACRfK RPjN8D/baTzKSGC1Zuf7rbPi+xCF46qSih6421CDYAJ/SzEwnzN/b7TeZiXk0dqu rZWQbwCjIrQVi7nNtixSQfKkNYbRr3IUCPjzOi0I4GlPjQ7PzJL9ZjhC8vfhnXoJ c2sZtkPQjydvLQuIy2/4UoYSn1PFPSP2KWebSE7zSMvxoAs/leTa43gz04ZaVYJ/ 0XlUPDir8vHeiSYE30c0CvCvhE+Mvr/sHojHzm2jFTmyVONS6wcR0oCb9HvSmv9h nhHoFq4Bnf1FZw6Vv5myxg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyusvy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:01 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECZ4j2025433; Fri, 14 Feb 2025 14:15:00 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgjm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:00 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETPx006920; Fri, 14 Feb 2025 14:15:00 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-24; Fri, 14 Feb 2025 14:14:59 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 23/45] vfio/pci: export MSI functions Date: Fri, 14 Feb 2025 06:14:05 -0800 Message-Id: <1739542467-226739-24-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: LaGo8S9kng3rmpptxLp4YFuZvtuVKc_n X-Proofpoint-GUID: LaGo8S9kng3rmpptxLp4YFuZvtuVKc_n Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Export various MSI functions, for use by CPR in subsequent patches. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 21 ++++++++++----------- hw/vfio/pci.h | 12 ++++++++++++ 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index fd51b9e..29a5b3d 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -277,7 +277,7 @@ static void vfio_irqchip_change(Notifier *notify, void *data) vfio_intx_update(vdev, &vdev->intx.route); } -static bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) +bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) { uint8_t pin = vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1); Error *err = NULL; @@ -351,7 +351,7 @@ static void vfio_intx_disable(VFIOPCIDevice *vdev) /* * MSI/X */ -static void vfio_msi_interrupt(void *opaque) +void vfio_msi_interrupt(void *opaque) { VFIOMSIVector *vector = opaque; VFIOPCIDevice *vdev = vector->vdev; @@ -474,8 +474,8 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) return ret; } -static void vfio_add_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, - int vector_n, bool msix) +void vfio_add_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, + int vector_n, bool msix) { if ((msix && vdev->no_kvm_msix) || (!msix && vdev->no_kvm_msi)) { return; @@ -529,7 +529,7 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, kvm_irqchip_commit_routes(kvm_state); } -static void vfio_vector_init(VFIOPCIDevice *vdev, int nr) +void vfio_vector_init(VFIOPCIDevice *vdev, int nr) { VFIOMSIVector *vector = &vdev->msi_vectors[nr]; PCIDevice *pdev = &vdev->pdev; @@ -640,13 +640,12 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, return 0; } -static int vfio_msix_vector_use(PCIDevice *pdev, - unsigned int nr, MSIMessage msg) +int vfio_msix_vector_use(PCIDevice *pdev, unsigned int nr, MSIMessage msg) { return vfio_msix_vector_do_use(pdev, nr, &msg, vfio_msi_interrupt); } -static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) +void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) { VFIOPCIDevice *vdev = VFIO_PCI(pdev); VFIOMSIVector *vector = &vdev->msi_vectors[nr]; @@ -673,14 +672,14 @@ static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) } } -static void vfio_prepare_kvm_msi_virq_batch(VFIOPCIDevice *vdev) +void vfio_prepare_kvm_msi_virq_batch(VFIOPCIDevice *vdev) { assert(!vdev->defer_kvm_irq_routing); vdev->defer_kvm_irq_routing = true; vfio_route_change = kvm_irqchip_begin_route_changes(kvm_state); } -static void vfio_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev) +void vfio_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev) { int i; @@ -2587,7 +2586,7 @@ static Object *vfio_pci_get_object(VFIODevice *vbasedev) return OBJECT(vdev); } -static bool vfio_msix_present(void *opaque, int version_id) +bool vfio_msix_present(void *opaque, int version_id) { PCIDevice *pdev = opaque; diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 43c1666..49faf57 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -197,6 +197,18 @@ static inline bool vfio_is_vga(VFIOPCIDevice *vdev) return class == PCI_CLASS_DISPLAY_VGA; } +/* MSI/MSI-X/INTx */ +void vfio_vector_init(VFIOPCIDevice *vdev, int nr); +void vfio_msi_interrupt(void *opaque); +void vfio_add_kvm_msi_virq(VFIOPCIDevice *vdev, VFIOMSIVector *vector, + int vector_n, bool msix); +int vfio_msix_vector_use(PCIDevice *pdev, unsigned int nr, MSIMessage msg); +void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr); +bool vfio_msix_present(void *opaque, int version_id); +void vfio_prepare_kvm_msi_virq_batch(VFIOPCIDevice *vdev); +void vfio_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev); +bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp); + uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len); void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len); From patchwork Fri Feb 14 14:14:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C72DC02198 for ; Fri, 14 Feb 2025 14:28:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTI-0005S5-RO; Fri, 14 Feb 2025 09:15:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTE-0005Ml-IM for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:08 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTB-0004gU-TV for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:08 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtVob006567; Fri, 14 Feb 2025 14:15:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=a6jTTQQvcIqWlxivx37CfEI0ZQo9AgdOEAsw38ZaQF0=; b= m8nojxMV+9vRQQ0Q+iQAiJay3GB0MtQnI7gn9aCvoXhAfYgEz4G3ZmBveOdQ3Z1K 5sm5/6e0isA/YhDWKQwHQYpw2X3Fz0ULK3RoiAc9QcXnbiOw2fixHRtVk12zOGFn EHcUKkgRARrwQjzvxF379KUsD25YyuI4nbS57JkXt5bYbqYAaY+SzVSuRVbUJgl9 armyfbygRXXWW2qV/5n7gCGKloQnE2/VNwhXcOiNdJ5dwQOi7FyDcLdxHabrWZv6 tuF/hqrznVhFrrlChFOVLONaVt4bAEY+rxNVIAbKYFWqMCuxQbyQEkqKSoyXB4aP 4oEKMA9RbYYlCwWFhYj2fQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qaknw7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:02 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ED5ZWc025287; Fri, 14 Feb 2025 14:15:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgm5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:02 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQ1006920; Fri, 14 Feb 2025 14:15:01 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-25; Fri, 14 Feb 2025 14:15:01 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 24/45] vfio-pci: preserve MSI Date: Fri, 14 Feb 2025 06:14:06 -0800 Message-Id: <1739542467-226739-25-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: hajmOIlDGtoDDLBH8xTQkpvRSHkFM6wv X-Proofpoint-ORIG-GUID: hajmOIlDGtoDDLBH8xTQkpvRSHkFM6wv Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Save the MSI message area as part of vfio-pci vmstate, and preserve the interrupt and notifier eventfd's. migrate_incoming loads the MSI data, then the vfio-pci post_load handler finds the eventfds in CPR state, rebuilds vector data structures, and attaches the interrupts to the new KVM instance. Signed-off-by: Steve Sistare --- hw/vfio/cpr.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 40 ++++++++++++++++++-- include/hw/vfio/vfio-cpr.h | 8 ++++ 3 files changed, 136 insertions(+), 3 deletions(-) diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index 8268c0c..96eb10a 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -9,6 +9,8 @@ #include "hw/vfio/vfio-common.h" #include "hw/vfio/vfio-cpr.h" #include "hw/vfio/pci.h" +#include "hw/pci/msix.h" +#include "hw/pci/msi.h" #include "migration/cpr.h" #include "qapi/error.h" #include "system/runstate.h" @@ -40,6 +42,69 @@ void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer) migration_remove_notifier(&bcontainer->cpr_reboot_notifier); } +#define STRDUP_VECTOR_FD_NAME(vdev, name) \ + g_strdup_printf("%s_%s", (vdev)->vbasedev.name, (name)) + +void vfio_cpr_save_vector_fd(VFIOPCIDevice *vdev, const char *name, int nr, + int fd) +{ + g_autofree char *fdname = STRDUP_VECTOR_FD_NAME(vdev, name); + cpr_resave_fd(fdname, nr, fd); +} + +int vfio_cpr_load_vector_fd(VFIOPCIDevice *vdev, const char *name, int nr) +{ + g_autofree char *fdname = STRDUP_VECTOR_FD_NAME(vdev, name); + return cpr_find_fd(fdname, nr); +} + +void vfio_cpr_delete_vector_fd(VFIOPCIDevice *vdev, const char *name, int nr) +{ + g_autofree char *fdname = STRDUP_VECTOR_FD_NAME(vdev, name); + cpr_delete_fd(fdname, nr); +} + +static void vfio_cpr_claim_vectors(VFIOPCIDevice *vdev, int nr_vectors, + bool msix) +{ + int i, fd; + bool pending = false; + PCIDevice *pdev = &vdev->pdev; + + vdev->nr_vectors = nr_vectors; + vdev->msi_vectors = g_new0(VFIOMSIVector, nr_vectors); + vdev->interrupt = msix ? VFIO_INT_MSIX : VFIO_INT_MSI; + + vfio_prepare_kvm_msi_virq_batch(vdev); + + for (i = 0; i < nr_vectors; i++) { + VFIOMSIVector *vector = &vdev->msi_vectors[i]; + + fd = vfio_cpr_load_vector_fd(vdev, "interrupt", i); + if (fd >= 0) { + vfio_vector_init(vdev, i); + qemu_set_fd_handler(fd, vfio_msi_interrupt, NULL, vector); + } + + if (vfio_cpr_load_vector_fd(vdev, "kvm_interrupt", i) >= 0) { + vfio_add_kvm_msi_virq(vdev, vector, i, msix); + } else { + vdev->msi_vectors[i].virq = -1; + } + + if (msix && msix_is_pending(pdev, i) && msix_is_masked(pdev, i)) { + set_bit(i, vdev->msix->pending); + pending = true; + } + } + + vfio_commit_kvm_msi_virq_batch(vdev); + + if (msix) { + memory_region_set_enabled(&pdev->msix_pba_mmio, pending); + } +} + /* * The kernel may change non-emulated config bits. Exclude them from the * changed-bits check in get_pci_config_device. @@ -58,13 +123,39 @@ static int vfio_cpr_pci_pre_load(void *opaque) return 0; } +static int vfio_cpr_pci_post_load(void *opaque, int version_id) +{ + VFIOPCIDevice *vdev = opaque; + PCIDevice *pdev = &vdev->pdev; + int nr_vectors; + + if (msix_enabled(pdev)) { + msix_set_vector_notifiers(pdev, vfio_msix_vector_use, + vfio_msix_vector_release, NULL); + nr_vectors = vdev->msix->entries; + vfio_cpr_claim_vectors(vdev, nr_vectors, true); + + } else if (msi_enabled(pdev)) { + nr_vectors = msi_nr_vectors_allocated(pdev); + vfio_cpr_claim_vectors(vdev, nr_vectors, false); + + } else if (vfio_pci_read_config(pdev, PCI_INTERRUPT_PIN, 1)) { + g_assert_not_reached(); /* completed in a subsequent patch */ + } + + return 0; +} + const VMStateDescription vfio_cpr_pci_vmstate = { .name = "vfio-cpr-pci", .version_id = 0, .minimum_version_id = 0, .pre_load = vfio_cpr_pci_pre_load, + .post_load = vfio_cpr_pci_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { + VMSTATE_PCI_DEVICE(pdev, VFIOPCIDevice), + VMSTATE_MSIX_TEST(pdev, VFIOPCIDevice, vfio_msix_present), VMSTATE_END_OF_LIST() } }; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 29a5b3d..465ca6b 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -29,6 +29,7 @@ #include "hw/pci/pci_bridge.h" #include "hw/qdev-properties.h" #include "hw/qdev-properties-system.h" +#include "hw/vfio/vfio-cpr.h" #include "migration/vmstate.h" #include "qobject/qdict.h" #include "qemu/error-report.h" @@ -54,13 +55,25 @@ static void vfio_disable_interrupts(VFIOPCIDevice *vdev); static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled); static void vfio_msi_disable_common(VFIOPCIDevice *vdev); +/* Create new or reuse existing eventfd */ static bool vfio_notifier_init(VFIOPCIDevice *vdev, EventNotifier *e, const char *name, int nr, Error **errp) { - int ret = event_notifier_init(e, 0); + int fd = vfio_cpr_load_vector_fd(vdev, name, nr); + int ret = 0; - if (ret) { - error_setg_errno(errp, -ret, "vfio_notifier_init %s failed", name); + if (fd >= 0) { + event_notifier_init_fd(e, fd); + } else { + ret = event_notifier_init(e, 0); + if (ret) { + error_setg_errno(errp, -ret, "vfio_notifier_init %s failed", name); + } else { + fd = event_notifier_get_fd(e); + if (fd >= 0) { + vfio_cpr_save_vector_fd(vdev, name, nr, fd); + } + } } return !ret; } @@ -68,6 +81,7 @@ static bool vfio_notifier_init(VFIOPCIDevice *vdev, EventNotifier *e, static void vfio_notifier_cleanup(VFIOPCIDevice *vdev, EventNotifier *e, const char *name, int nr) { + vfio_cpr_delete_vector_fd(vdev, name, nr); event_notifier_cleanup(e); } @@ -554,6 +568,15 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, int ret; bool resizing = !!(vdev->nr_vectors < nr + 1); + /* + * Ignore the callback from msix_set_vector_notifiers during resume. + * The necessary subset of these actions is called from + * vfio_cpr_claim_vectors during post load. + */ + if (vdev->vbasedev.cpr.reused) { + return 0; + } + trace_vfio_msix_vector_do_use(vdev->vbasedev.name, nr); vector = &vdev->msi_vectors[nr]; @@ -2894,6 +2917,11 @@ static void vfio_register_err_notifier(VFIOPCIDevice *vdev) fd = event_notifier_get_fd(&vdev->err_notifier); qemu_set_fd_handler(fd, vfio_err_notifier_handler, NULL, vdev); + /* Do not alter irq_signaling during vfio_realize for cpr */ + if (vdev->vbasedev.cpr.reused) { + return; + } + if (!vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_ERR_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); @@ -2960,6 +2988,12 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) fd = event_notifier_get_fd(&vdev->req_notifier); qemu_set_fd_handler(fd, vfio_req_notifier_handler, NULL, vdev); + /* Do not alter irq_signaling during vfio_realize for cpr */ + if (vdev->vbasedev.cpr.reused) { + vdev->req_enabled = true; + return; + } + if (!vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_REQ_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index f5480de..a9f2fbe 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -27,6 +27,7 @@ typedef struct VFIODeviceCPR { struct VFIOContainer; struct VFIOGroup; struct VFIOContainerBase; +struct VFIOPCIDevice; int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp); @@ -44,5 +45,12 @@ void vfio_cpr_giommu_remap(struct VFIOContainerBase *bcontainer, bool vfio_cpr_register_ram_discard_listener( struct VFIOContainerBase *bcontainer, MemoryRegionSection *section); +void vfio_cpr_save_vector_fd(struct VFIOPCIDevice *vdev, const char *name, + int nr, int fd); +int vfio_cpr_load_vector_fd(struct VFIOPCIDevice *vdev, const char *name, + int nr); +void vfio_cpr_delete_vector_fd(struct VFIOPCIDevice *vdev, const char *name, + int nr); + extern const VMStateDescription vfio_cpr_pci_vmstate; #endif From patchwork Fri Feb 14 14:14:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ED46DC02198 for ; Fri, 14 Feb 2025 14:23:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTT-00061X-00; Fri, 14 Feb 2025 09:15:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTP-0005s0-Qq for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:19 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTN-0004gn-MO for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:19 -0500 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtUUl022105; Fri, 14 Feb 2025 14:15:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=pslagFP9E8mIaHHpFR0W5yIEnzrG0ZmdbkLwbbFkA/o=; b= RhSsYrmCwjVzSQTaafNzfbYby+c78F7gruF00znXyhU1p3cOt4FFhgncXwR/kTup FGmy6dWCYqycdKJ5pLrNpxHRfPHT1lHPnz7HXB1AvI1jff1bR0I0coP3GbHBqsO5 yxrGU5qxhPU7oXKNviOUqkOqNx04/sDDToNEMeH0qdQGrblkW6Z4jTy2EoCk6bZY WQMoi5r5+GM8yASGBzDhtj7WYtFNtwhnTZFijFnxn+u5AvW5vbT/QDCc1kckHCkQ 15NrhfHZBUiSCb16wwlJKBSlLLSsXaz6A+AIILhHtk41SlmfgqszLCHQ4EFIS6+e LVpbMbZHjwklD4NG4tmFAA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0s43q74-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:04 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECxUWI025134; Fri, 14 Feb 2025 14:15:03 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgnd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:03 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQ3006920; Fri, 14 Feb 2025 14:15:02 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-26; Fri, 14 Feb 2025 14:15:02 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 25/45] vfio-pci: preserve INTx Date: Fri, 14 Feb 2025 06:14:07 -0800 Message-Id: <1739542467-226739-26-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: zrY8DdCpGzgWzpNBGd2vAD5V97oQ69sr X-Proofpoint-ORIG-GUID: zrY8DdCpGzgWzpNBGd2vAD5V97oQ69sr Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Preserve vfio INTx state across cpr-transfer. Preserve VFIOINTx fields as follows: pin : Recover this from the vfio config in kernel space interrupt : Preserve its eventfd descriptor across exec. unmask : Ditto route.irq : This could perhaps be recovered in vfio_pci_post_load by calling pci_device_route_intx_to_irq(pin), whose implementation reads config space for a bridge device such as ich9. However, there is no guarantee that the bridge vmstate is read before vfio vmstate. Rather than fiddling with MigrationPriority for vmstate handlers, explicitly save route.irq in vfio vmstate. pending : save in vfio vmstate. mmap_timeout, mmap_timer : Re-initialize bool kvm_accel : Re-initialize In vfio_realize, defer calling vfio_intx_enable until the vmstate is available, in vfio_pci_post_load. Modify vfio_intx_enable and vfio_intx_kvm_enable to skip vfio initialization, but still perform kvm initialization. Signed-off-by: Steve Sistare --- hw/vfio/cpr.c | 27 ++++++++++++++++++++++++++- hw/vfio/pci.c | 28 +++++++++++++++++++++++++--- 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index 96eb10a..a2400ca 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -140,12 +140,36 @@ static int vfio_cpr_pci_post_load(void *opaque, int version_id) vfio_cpr_claim_vectors(vdev, nr_vectors, false); } else if (vfio_pci_read_config(pdev, PCI_INTERRUPT_PIN, 1)) { - g_assert_not_reached(); /* completed in a subsequent patch */ + Error *err = NULL; + if (!vfio_intx_enable(vdev, &err)) { + error_report_err(err); + return -1; + } } return 0; } +static const VMStateDescription vfio_intx_vmstate = { + .name = "vfio-cpr-intx", + .version_id = 0, + .minimum_version_id = 0, + .fields = (VMStateField[]) { + VMSTATE_BOOL(pending, VFIOINTx), + VMSTATE_UINT32(route.mode, VFIOINTx), + VMSTATE_INT32(route.irq, VFIOINTx), + VMSTATE_END_OF_LIST() + } +}; + +#define VMSTATE_VFIO_INTX(_field, _state) { \ + .name = (stringify(_field)), \ + .size = sizeof(VFIOINTx), \ + .vmsd = &vfio_intx_vmstate, \ + .flags = VMS_STRUCT, \ + .offset = vmstate_offset_value(_state, _field, VFIOINTx), \ +} + const VMStateDescription vfio_cpr_pci_vmstate = { .name = "vfio-cpr-pci", .version_id = 0, @@ -156,6 +180,7 @@ const VMStateDescription vfio_cpr_pci_vmstate = { .fields = (VMStateField[]) { VMSTATE_PCI_DEVICE(pdev, VFIOPCIDevice), VMSTATE_MSIX_TEST(pdev, VFIOPCIDevice, vfio_msix_present), + VMSTATE_VFIO_INTX(intx, VFIOPCIDevice), VMSTATE_END_OF_LIST() } }; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 465ca6b..c5470d0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -158,12 +158,17 @@ static bool vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) return true; } + if (vdev->vbasedev.cpr.reused) { + goto skip_state; + } + /* Get to a known interrupt state */ qemu_set_fd_handler(irq_fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); vdev->intx.pending = false; pci_irq_deassert(&vdev->pdev); +skip_state: /* Get an eventfd for resample/unmask */ if (!vfio_notifier_init(vdev, &vdev->intx.unmask, "intx-unmask", 0, errp)) { goto fail; @@ -177,6 +182,10 @@ static bool vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) goto fail_irqfd; } + if (vdev->vbasedev.cpr.reused) { + goto skip_irq; + } + if (!vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_UNMASK, event_notifier_get_fd(&vdev->intx.unmask), @@ -187,6 +196,7 @@ static bool vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) /* Let'em rip */ vfio_unmask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); +skip_irq: vdev->intx.kvm_accel = true; trace_vfio_intx_enable_kvm(vdev->vbasedev.name); @@ -302,7 +312,13 @@ bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) return true; } - vfio_disable_interrupts(vdev); + /* + * Do not alter interrupt state during vfio_realize and cpr load. The + * reused flag is cleared thereafter. + */ + if (!vdev->vbasedev.cpr.reused) { + vfio_disable_interrupts(vdev); + } vdev->intx.pin = pin - 1; /* Pin A (1) -> irq[0] */ pci_config_set_interrupt_pin(vdev->pdev.config, pin); @@ -325,7 +341,8 @@ bool vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) fd = event_notifier_get_fd(&vdev->intx.interrupt); qemu_set_fd_handler(fd, vfio_intx_interrupt, NULL, vdev); - if (!vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX, 0, + if (!vdev->vbasedev.cpr.reused && + !vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_TRIGGER, fd, errp)) { qemu_set_fd_handler(fd, NULL, NULL, vdev); vfio_notifier_cleanup(vdev, &vdev->intx.interrupt, "intx-interrupt", 0); @@ -3241,7 +3258,12 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vfio_intx_routing_notifier); vdev->irqchip_change_notifier.notify = vfio_irqchip_change; kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); - if (!vfio_intx_enable(vdev, errp)) { + /* + * During CPR, do not call vfio_intx_enable at this time. Instead, + * call it from vfio_pci_post_load after the intx routing data has + * been loaded from vmstate. + */ + if (!vdev->vbasedev.cpr.reused && !vfio_intx_enable(vdev, errp)) { goto out_deregister; } } From patchwork Fri Feb 14 14:14:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A8DA3C02198 for ; Fri, 14 Feb 2025 14:16:28 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTL-0005eL-Nu; Fri, 14 Feb 2025 09:15:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTG-0005Sa-Td for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:10 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTE-0004hB-NN for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:10 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtTNM006492; Fri, 14 Feb 2025 14:15:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=th0gfiJHDWrXElsacnCQOAe/RVbdzlgHDef6GLP3Cjg=; b= d6/YHZaRofeYVsK0aRhGhXhyaBAA6oTwaufSfyv6HNZLqwMm/D3xWfJ1kwMOgyie +xQMQ5SGGXKD1hdI+wBkpZhlmfz8cI6AoJybboVc4M0Ykh9LuyYvA7MxWcdKTUA1 RfHIwN9ZZHW500GyvlSicdjm7zfbnEDTe7uPVMS6a3VGjQzAg+UaZEseGe+NMAOX bPKw8rJHdaaHoBmvwFOKEZPDqh48/ZbGsDxTsgsbcO0QdIt/HtSD45elWqwhiEyC bBpQM6KmPL3QfZxBWWgkxzIvwAziNEVnPtpEDRVm+C1MZiVElmxPpb+eNpYXWAen nFRzSIW40sJzRhzEtpmiyA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qaknwa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:05 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECSkjT025168; Fri, 14 Feb 2025 14:15:05 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgpt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:05 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQ5006920; Fri, 14 Feb 2025 14:15:04 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-27; Fri, 14 Feb 2025 14:15:04 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 26/45] migration: close kvm after cpr Date: Fri, 14 Feb 2025 06:14:08 -0800 Message-Id: <1739542467-226739-27-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: Quko-V2_qx4MsHBpNhh_UJ7V9hM6EYZW X-Proofpoint-ORIG-GUID: Quko-V2_qx4MsHBpNhh_UJ7V9hM6EYZW Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org cpr-transfer breaks vfio network connectivity to and from the guest, and the host system log shows: irq bypass consumer (token 00000000a03c32e5) registration fails: -16 which is EBUSY. This occurs because KVM descriptors are still open in the old QEMU process. Close them. Signed-off-by: Steve Sistare --- accel/kvm/kvm-all.c | 28 ++++++++++++++++++++++++++++ hw/vfio/common.c | 8 ++++++++ include/hw/vfio/vfio-common.h | 1 + include/migration/cpr.h | 2 ++ include/system/kvm.h | 1 + migration/cpr-transfer.c | 18 ++++++++++++++++++ migration/cpr.c | 8 ++++++++ migration/migration.c | 1 + 8 files changed, 67 insertions(+) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index c65b790..cdbe91c 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -507,16 +507,23 @@ static int do_kvm_destroy_vcpu(CPUState *cpu) goto err; } + /* If I am the CPU that created coalesced_mmio_ring, then discard it */ + if (s->coalesced_mmio_ring == (void *)cpu->kvm_run + PAGE_SIZE) { + s->coalesced_mmio_ring = NULL; + } + ret = munmap(cpu->kvm_run, mmap_size); if (ret < 0) { goto err; } + cpu->kvm_run = NULL; if (cpu->kvm_dirty_gfns) { ret = munmap(cpu->kvm_dirty_gfns, s->kvm_dirty_ring_bytes); if (ret < 0) { goto err; } + cpu->kvm_dirty_gfns = NULL; } kvm_park_vcpu(cpu); @@ -595,6 +602,27 @@ err: return ret; } +void kvm_close(void) +{ + CPUState *cpu; + + CPU_FOREACH(cpu) { + cpu_remove_sync(cpu); + close(cpu->kvm_fd); + cpu->kvm_fd = -1; + close(cpu->kvm_vcpu_stats_fd); + cpu->kvm_vcpu_stats_fd = -1; + } + + if (kvm_state && kvm_state->fd != -1) { + close(kvm_state->vmfd); + kvm_state->vmfd = -1; + close(kvm_state->fd); + kvm_state->fd = -1; + } + kvm_state = NULL; +} + /* * dirty pages logging control */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 48663ad..c536698 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1501,6 +1501,14 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) return 0; } +void vfio_kvm_device_close(void) +{ + if (vfio_kvm_device_fd != -1) { + close(vfio_kvm_device_fd); + vfio_kvm_device_fd = -1; + } +} + VFIOAddressSpace *vfio_get_address_space(AddressSpace *as) { VFIOAddressSpace *space; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1563f3a..78e4f12 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -259,6 +259,7 @@ VFIODevice *vfio_get_vfio_device(Object *obj); int vfio_kvm_device_add_fd(int fd, Error **errp); int vfio_kvm_device_del_fd(int fd, Error **errp); +void vfio_kvm_device_close(void); bool vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **errp); void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer); diff --git a/include/migration/cpr.h b/include/migration/cpr.h index 6ad04d4..c5c191d 100644 --- a/include/migration/cpr.h +++ b/include/migration/cpr.h @@ -32,7 +32,9 @@ void cpr_state_close(void); struct QIOChannel *cpr_state_ioc(void); bool cpr_needed_for_reuse(void *opaque); +void cpr_kvm_close(void); +void cpr_transfer_init(void); QEMUFile *cpr_transfer_output(MigrationChannel *channel, Error **errp); QEMUFile *cpr_transfer_input(MigrationChannel *channel, Error **errp); diff --git a/include/system/kvm.h b/include/system/kvm.h index ab17c09..ad5c55e 100644 --- a/include/system/kvm.h +++ b/include/system/kvm.h @@ -194,6 +194,7 @@ bool kvm_has_sync_mmu(void); int kvm_has_vcpu_events(void); int kvm_max_nested_state_length(void); int kvm_has_gsi_routing(void); +void kvm_close(void); /** * kvm_arm_supports_user_irq diff --git a/migration/cpr-transfer.c b/migration/cpr-transfer.c index e1f1403..396558f 100644 --- a/migration/cpr-transfer.c +++ b/migration/cpr-transfer.c @@ -17,6 +17,24 @@ #include "migration/vmstate.h" #include "trace.h" +static int cpr_transfer_notifier(NotifierWithReturn *notifier, + MigrationEvent *e, + Error **errp) +{ + if (e->type == MIG_EVENT_PRECOPY_DONE) { + cpr_kvm_close(); + } + return 0; +} + +void cpr_transfer_init(void) +{ + static NotifierWithReturn notifier; + + migration_add_notifier_mode(¬ifier, cpr_transfer_notifier, + MIG_MODE_CPR_TRANSFER); +} + QEMUFile *cpr_transfer_output(MigrationChannel *channel, Error **errp) { MigrationAddress *addr = channel->addr; diff --git a/migration/cpr.c b/migration/cpr.c index 12c489b..351e12d 100644 --- a/migration/cpr.c +++ b/migration/cpr.c @@ -7,12 +7,14 @@ #include "qemu/osdep.h" #include "qapi/error.h" +#include "hw/vfio/vfio-common.h" #include "migration/cpr.h" #include "migration/misc.h" #include "migration/options.h" #include "migration/qemu-file.h" #include "migration/savevm.h" #include "migration/vmstate.h" +#include "system/kvm.h" #include "system/runstate.h" #include "trace.h" @@ -266,3 +268,9 @@ bool cpr_needed_for_reuse(void *opaque) MigMode mode = migrate_mode(); return mode == MIG_MODE_CPR_TRANSFER; } + +void cpr_kvm_close(void) +{ + kvm_close(); + vfio_kvm_device_close(); +} diff --git a/migration/migration.c b/migration/migration.c index 3969285..bdc5255 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -296,6 +296,7 @@ void migration_object_init(void) ram_mig_init(); dirty_bitmap_mig_init(); + cpr_transfer_init(); /* Initialize cpu throttle timers */ cpu_throttle_init(); From patchwork Fri Feb 14 14:14:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65F33C021A4 for ; Fri, 14 Feb 2025 14:23:27 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTl-0006EX-Mr; Fri, 14 Feb 2025 09:15:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTS-00060J-3b for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:22 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTQ-0004hX-7T for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:21 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtqIr019380; Fri, 14 Feb 2025 14:15:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=fwO6yKzZT3BvW6kLmf13EnVUAuSfTwaAxRHjTF7t7OU=; b= AdmKBmE2GDh31wrtbpvFxnsrlrE46FtkwoZhPQO8Sni/q8LNwwn5XDy95DG62k/E a6LybqYn6ZouFVcT2vONz8islFQBcRmgb/Rb1etPeXmbumWivLbk624V/Qs+JaUN mrJRYnX5/DSiLn1y471wnee4z6lNXv9frppYdL3vfjmF2IGWI9w/r/XVLs80Ykvu +7grKjoIcKPREsQBIjE9CqXKgKTID2WThlosf1iE5pywiC/8yXOoYu06qS71Rm5G j3yk4HHdMNsF4qwwMMm9q1Q6388jP0ZyDjwE+S5mwKJUy9BD3rKNgmV+/K0grKsL QZw8HbnB8XPdqr28ci/tyw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0t4bmv4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:07 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECSnir025242; Fri, 14 Feb 2025 14:15:06 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgqy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:06 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQ7006920; Fri, 14 Feb 2025 14:15:05 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-28; Fri, 14 Feb 2025 14:15:05 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 27/45] migration: cpr_get_fd_param helper Date: Fri, 14 Feb 2025 06:14:09 -0800 Message-Id: <1739542467-226739-28-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: swOtxUWr5BrwHFzqZuKiNheOFV1v5YXh X-Proofpoint-GUID: swOtxUWr5BrwHFzqZuKiNheOFV1v5YXh Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add the helper function cpr_get_fd_param, to use when preserving a file descriptor that is opened externally and passed to QEMU. cpr_get_fd_param returns a descriptor number either from a QEMU command-line parameter, from a getfd command, or from CPR state. When a descriptor is passed to new QEMU via SCM_RIGHTS, its number changes. Hence, during CPR, the command-line parameter is ignored in new QEMU, and over-ridden by the value found in CPR state. Similarly, if the descriptor was originally specified by a getfd command in old QEMU, the fd number is not known outside of QEMU, and it changes when sent to new QEMU via SCM_RIGHTS. Hence the user cannot send getfd to new QEMU, but when the user sends a hotplug command that references the fd, cpr_get_fd_param finds its value in CPR state. Signed-off-by: Steve Sistare --- include/migration/cpr.h | 2 ++ migration/cpr.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/include/migration/cpr.h b/include/migration/cpr.h index c5c191d..23d0af4 100644 --- a/include/migration/cpr.h +++ b/include/migration/cpr.h @@ -33,6 +33,8 @@ struct QIOChannel *cpr_state_ioc(void); bool cpr_needed_for_reuse(void *opaque); void cpr_kvm_close(void); +int cpr_get_fd_param(const char *name, const char *fdname, int index, + bool *reused, Error **errp); void cpr_transfer_init(void); QEMUFile *cpr_transfer_output(MigrationChannel *channel, Error **errp); diff --git a/migration/cpr.c b/migration/cpr.c index 351e12d..c903b24 100644 --- a/migration/cpr.c +++ b/migration/cpr.c @@ -14,6 +14,7 @@ #include "migration/qemu-file.h" #include "migration/savevm.h" #include "migration/vmstate.h" +#include "monitor/monitor.h" #include "system/kvm.h" #include "system/runstate.h" #include "trace.h" @@ -274,3 +275,42 @@ void cpr_kvm_close(void) kvm_close(); vfio_kvm_device_close(); } + +/* + * cpr_get_fd_param: find a descriptor and return its value. + * + * @name: CPR name for the descriptor + * @fdname: An integer-valued string, or a name passed to a getfd command + * @index: CPR index of the descriptor + * @reused: returns true if the fd is found in CPR state, else false. + * @errp: returned error message + * + * If CPR is not being performed, then use @fdname to find the fd. + * If CPR is being performed, then ignore @fdname, and look for @name + * and @index in CPR state. + * + * On success returns the fd value, else returns -1. + */ +int cpr_get_fd_param(const char *name, const char *fdname, int index, + bool *reused, Error **errp) +{ + ERRP_GUARD(); + int fd; + + if (cpr_is_incoming()) { + fd = cpr_find_fd(name, index); + if (fd < 0) { + error_setg(errp, "cannot find saved value for fd %s", fdname); + } + *reused = true; + } else { + fd = monitor_fd_param(monitor_cur(), fdname, errp); + if (fd >= 0) { + cpr_save_fd(name, index, fd); + } else { + error_prepend(errp, "Could not parse object fd %s:", fdname); + } + *reused = false; + } + return fd; +} From patchwork Fri Feb 14 14:14:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9156FC021A6 for ; Fri, 14 Feb 2025 14:16:39 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTP-0005nW-OF; Fri, 14 Feb 2025 09:15:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTL-0005eW-43 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:15 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTI-0004i0-C9 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:14 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtS5G006475; Fri, 14 Feb 2025 14:15:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=vM1B6GeRa4UEqO3Cl/FiOK5ia6+QhVy2/7P8qPOsHjY=; b= E8tRFaiWVzV6WiaA8Hx1xXWYOx10IQh2eTJ1n8TJSS1KWgegdonEd1ZAbxTn2Ygk 0ffHgEWCThqNVGgdofzc4a3TMVLsSotg9BkM3eEzLmADy/9cbekPwz6owhLx/7zy X8W8N9xHNG/BKYHOzSt0oO16v8Yui233P2+5QS4lWByYWnikLyu5Br3vUbL+VB+o DhqcQFLqy/MvI3EByhe5+asa7Ne3Vo8Odg1tGmRQubfnvGtsvz7HlQm5X48oayvi GxlH7BvE5w9cTduTfl94CaRpV/hufPzjA4pak4272r4oLhT5XjVnXWmDlsQNi+35 cAIUN2Khf3YBx7o+c3p1JA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qaknwg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:08 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECGEd5025179; Fri, 14 Feb 2025 14:15:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgs0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:07 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQ9006920; Fri, 14 Feb 2025 14:15:06 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-29; Fri, 14 Feb 2025 14:15:06 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 28/45] vfio: return mr from vfio_get_xlat_addr Date: Fri, 14 Feb 2025 06:14:10 -0800 Message-Id: <1739542467-226739-29-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: MyJd54vITEYBAEgtL61y6arsQLbfe6Jw X-Proofpoint-ORIG-GUID: MyJd54vITEYBAEgtL61y6arsQLbfe6Jw Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Modify memory_get_xlat_addr and vfio_get_xlat_addr to return the memory region that the translated address is found in. This will be needed by CPR in a subsequent patch to map blocks using IOMMU_IOAS_MAP_FILE. Also return the xlat offset, so we can simplify the interface by removing the out parameters that can be trivially derived from mr and xlat. Signed-off-by: Steve Sistare --- hw/vfio/common.c | 21 ++++++++++++++------- hw/virtio/vhost-vdpa.c | 8 ++++++-- include/exec/memory.h | 6 +++--- system/memory.c | 19 ++++--------------- 4 files changed, 27 insertions(+), 27 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index c536698..3b0c520 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -246,14 +246,13 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section) } /* Called with rcu_read_lock held. */ -static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, - ram_addr_t *ram_addr, bool *read_only, - Error **errp) +static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, MemoryRegion **mr_p, + hwaddr *xlat_p, Error **errp) { bool ret, mr_has_discard_manager; - ret = memory_get_xlat_addr(iotlb, vaddr, ram_addr, read_only, - &mr_has_discard_manager, errp); + ret = memory_get_xlat_addr(iotlb, &mr_has_discard_manager, mr_p, xlat_p, + errp); if (ret && mr_has_discard_manager) { /* * Malicious VMs might trigger discarding of IOMMU-mapped memory. The @@ -281,6 +280,8 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) VFIOGuestIOMMU *giommu = container_of(n, VFIOGuestIOMMU, n); VFIOContainerBase *bcontainer = giommu->bcontainer; hwaddr iova = iotlb->iova + giommu->iommu_offset; + MemoryRegion *mr; + hwaddr xlat; void *vaddr; int ret; Error *local_err = NULL; @@ -300,10 +301,13 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, &local_err)) { + if (!vfio_get_xlat_addr(iotlb, &mr, &xlat, &local_err)) { error_report_err(local_err); goto out; } + vaddr = memory_region_get_ram_ptr(mr) + xlat; + read_only = !(iotlb->perm & IOMMU_WO) || mr->readonly; + /* * vaddr is only valid until rcu_read_unlock(). But after * vfio_dma_map has set up the mapping the pages will be @@ -1259,6 +1263,8 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) ram_addr_t translated_addr; Error *local_err = NULL; int ret = -EINVAL; + MemoryRegion *mr; + ram_addr_t xlat; trace_vfio_iommu_map_dirty_notify(iova, iova + iotlb->addr_mask); @@ -1269,10 +1275,11 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } rcu_read_lock(); - if (!vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL, &local_err)) { + if (!vfio_get_xlat_addr(iotlb, &mr, &xlat, &local_err)) { error_report_err(local_err); goto out_unlock; } + translated_addr = memory_region_get_ram_addr(mr) + xlat; ret = vfio_get_dirty_bitmap(bcontainer, iova, iotlb->addr_mask + 1, translated_addr, &local_err); diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 3cdaa12..5dfe51e 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -209,6 +209,8 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) int ret; Int128 llend; Error *local_err = NULL; + MemoryRegion *mr; + hwaddr xlat; if (iotlb->target_as != &address_space_memory) { error_report("Wrong target AS \"%s\", only system memory is allowed", @@ -228,11 +230,13 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!memory_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, NULL, - &local_err)) { + if (!memory_get_xlat_addr(iotlb, NULL, &mr, &xlat, &local_err)) { error_report_err(local_err); return; } + vaddr = memory_region_get_ram_ptr(mr) + xlat; + read_only = !(iotlb->perm & IOMMU_WO) || mr->readonly; + ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova, iotlb->addr_mask + 1, vaddr, read_only); if (ret) { diff --git a/include/exec/memory.h b/include/exec/memory.h index ea5d33a..8590838 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -747,13 +747,13 @@ void ram_discard_manager_unregister_listener(RamDiscardManager *rdm, * @read_only: indicates if writes are allowed * @mr_has_discard_manager: indicates memory is controlled by a * RamDiscardManager + * @mr_p: return the MemoryRegion containing the @iotlb translated addr * @errp: pointer to Error*, to store an error if it happens. * * Return: true on success, else false setting @errp with error. */ -bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, - ram_addr_t *ram_addr, bool *read_only, - bool *mr_has_discard_manager, Error **errp); +bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, bool *mr_has_discard_manager, + MemoryRegion **mr_p, hwaddr *xlat_p, Error **errp); typedef struct CoalescedMemoryRange CoalescedMemoryRange; typedef struct MemoryRegionIoeventfd MemoryRegionIoeventfd; diff --git a/system/memory.c b/system/memory.c index 4c82979..755eafe 100644 --- a/system/memory.c +++ b/system/memory.c @@ -2183,9 +2183,8 @@ void ram_discard_manager_unregister_listener(RamDiscardManager *rdm, } /* Called with rcu_read_lock held. */ -bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, - ram_addr_t *ram_addr, bool *read_only, - bool *mr_has_discard_manager, Error **errp) +bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, bool *mr_has_discard_manager, + MemoryRegion **mr_p, hwaddr *xlat_p, Error **errp) { MemoryRegion *mr; hwaddr xlat; @@ -2238,18 +2237,8 @@ bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, return false; } - if (vaddr) { - *vaddr = memory_region_get_ram_ptr(mr) + xlat; - } - - if (ram_addr) { - *ram_addr = memory_region_get_ram_addr(mr) + xlat; - } - - if (read_only) { - *read_only = !writable || mr->readonly; - } - + *xlat_p = xlat; + *mr_p = mr; return true; } From patchwork Fri Feb 14 14:14:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1338C02198 for ; Fri, 14 Feb 2025 14:17:02 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwTt-0006YS-DY; Fri, 14 Feb 2025 09:15:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTV-0006Bm-Bf for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:38 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTT-0004i9-Ew for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:24 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtfkB011618; Fri, 14 Feb 2025 14:15:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=Cmb7qEyekpXFsKg7j1sj4mD4BeE+GTmnroas7VvhjXE=; b= i0yfRhbqpFfd/1LoYm4DcDFRKsthXIP+92Er33nN3T3E8twci3EuHABdeOzuO0ZN 2o3bP2/F5phfY+RVKLqgtvKMvHBEWt+UQ1xI3ESSrLc7WwfaSXxmiBBfc5dhP95Y WrLlvVhibN0CI5xoPc35r/19a6AuQoiQRzzVOMtGw1adfJoYnWR/ghAGGSABpDXk EtcDxSdNxi3nSW/CyT5CI2KhtdeSgO1D+nI9dMYuh3c0TAu8X6DD+eN79Q7aKzBF 8FShMUJsE+lWq37M6ODUWRt/hvyftTXBsdg4O72GPHr2Qj083qXgoUyXo/WUtChR FaVt9t1HT1B+aBiwdq1etg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwne-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:09 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECIvBg025325; Fri, 14 Feb 2025 14:15:08 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgtg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:08 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQB006920; Fri, 14 Feb 2025 14:15:08 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-30; Fri, 14 Feb 2025 14:15:08 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 29/45] vfio: pass ramblock to vfio_container_dma_map Date: Fri, 14 Feb 2025 06:14:11 -0800 Message-Id: <1739542467-226739-30-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: pXNdohNJ3ENIRuV0Li5pDlkpDCdQGO6R X-Proofpoint-GUID: pXNdohNJ3ENIRuV0Li5pDlkpDCdQGO6R Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Pass ramblock to vfio_container_dma_map for use in a subsequent patch. The ramblock's attributes will be needed to map the block using IOMMU_IOAS_MAP_FILE. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/common.c | 8 +++++--- hw/vfio/container-base.c | 3 ++- include/hw/vfio/vfio-container-base.h | 3 ++- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 3b0c520..350d5aa 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -317,7 +317,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) */ ret = vfio_container_dma_map(bcontainer, iova, iotlb->addr_mask + 1, vaddr, - read_only); + read_only, mr->ram_block); if (ret) { error_report("vfio_container_dma_map(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx", %p) = %d (%s)", @@ -382,7 +382,8 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl, vaddr = memory_region_get_ram_ptr(section->mr) + start; ret = vfio_container_dma_map(bcontainer, iova, next - start, - vaddr, section->readonly); + vaddr, section->readonly, + section->mr->ram_block); if (ret) { /* Rollback */ vfio_ram_discard_notify_discard(rdl, section); @@ -716,7 +717,8 @@ void vfio_container_region_add(VFIOContainerBase *bcontainer, } ret = vfio_container_dma_map(bcontainer, iova, int128_get64(llsize), - vaddr, section->readonly); + vaddr, section->readonly, + section->mr->ram_block); if (ret) { error_setg(&err, "vfio_container_dma_map(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx", %p) = %d (%s)", diff --git a/hw/vfio/container-base.c b/hw/vfio/container-base.c index 749a3fd..302cd4c 100644 --- a/hw/vfio/container-base.c +++ b/hw/vfio/container-base.c @@ -17,7 +17,8 @@ int vfio_container_dma_map(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - void *vaddr, bool readonly) + void *vaddr, bool readonly, + RAMBlock *rb) { VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index 4cff994..d82e256 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -73,7 +73,8 @@ typedef struct VFIORamDiscardListener { int vfio_container_dma_map(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - void *vaddr, bool readonly); + void *vaddr, bool readonly, + RAMBlock *rb); int vfio_container_dma_unmap(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb); From patchwork Fri Feb 14 14:14:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E2F5C3DA4A for ; Fri, 14 Feb 2025 14:21:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUH-0006w5-6l; Fri, 14 Feb 2025 09:16:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTW-0006Bu-9S for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:38 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTU-0004iG-5W for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:25 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtVTI017784; Fri, 14 Feb 2025 14:15:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=9W2Rnyiz7DlZBNIAAKjFZqi0y6Hw0lmu+7DsnzAaMvw=; b= G0f+1rMjeo4iGuG+kXb61HphqiQySInWSVvUB29xzN2az0QxliJUgURyUj2gi5DU S6+FusFzXxHX6ljHX+oXiJ+PmbLc00kN5XIwCa0lLef95zuO5vEXacfxgBQf7Hqv 1fClAjTYB1o8deChL/YHIEv2p/DY06QMguyVbKqDXHb+mQSX4OU71NqxU9dTP/KR ffeLR9mrd5ulHtxbcXDhaZe9jCi8P8Uf8ZnakabnD1ut13EQQztv+lv5BAm7irfj lEKHju3yV81JBh/f/vvSaV4aThSfigH4ulchJKVcsNWIJ6LEEuq89nHgdYPQnPTL 5WJqnpoyGDmyVHgppnG79w== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tgbtwk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:11 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECSfbq025328; Fri, 14 Feb 2025 14:15:10 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgus-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:10 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQD006920; Fri, 14 Feb 2025 14:15:09 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-31; Fri, 14 Feb 2025 14:15:09 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 30/45] backends/iommufd: iommufd_backend_map_file_dma Date: Fri, 14 Feb 2025 06:14:12 -0800 Message-Id: <1739542467-226739-31-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: N3sumdBFR6hBqlquk8lAj4WP51b2x7bJ X-Proofpoint-ORIG-GUID: N3sumdBFR6hBqlquk8lAj4WP51b2x7bJ Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define iommufd_backend_map_file_dma to implement IOMMU_IOAS_MAP_FILE. This will be called as a substitute for iommufd_backend_map_dma, so the error conditions for BARs are copied as-is from that function. Signed-off-by: Steve Sistare --- backends/iommufd.c | 36 ++++++++++++++++++++++++++++++++++++ backends/trace-events | 1 + include/system/iommufd.h | 3 +++ 3 files changed, 40 insertions(+) diff --git a/backends/iommufd.c b/backends/iommufd.c index d57da44..612de78 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -172,6 +172,42 @@ int iommufd_backend_map_dma(IOMMUFDBackend *be, uint32_t ioas_id, hwaddr iova, return ret; } +int iommufd_backend_map_file_dma(IOMMUFDBackend *be, uint32_t ioas_id, + hwaddr iova, ram_addr_t size, + int mfd, unsigned long start, bool readonly) +{ + int ret, fd = be->fd; + struct iommu_ioas_map_file map = { + .size = sizeof(map), + .flags = IOMMU_IOAS_MAP_READABLE | + IOMMU_IOAS_MAP_FIXED_IOVA, + .ioas_id = ioas_id, + .fd = mfd, + .start = start, + .iova = iova, + .length = size, + }; + + if (!readonly) { + map.flags |= IOMMU_IOAS_MAP_WRITEABLE; + } + + ret = ioctl(fd, IOMMU_IOAS_MAP_FILE, &map); + trace_iommufd_backend_map_file_dma(fd, ioas_id, iova, size, mfd, start, + readonly, ret); + if (ret) { + ret = -errno; + + /* TODO: Not support mapping hardware PCI BAR region for now. */ + if (errno == EFAULT) { + warn_report("IOMMU_IOAS_MAP_FILE failed: %m, PCI BAR?"); + } else { + error_report("IOMMU_IOAS_MAP_FILE failed: %m"); + } + } + return ret; +} + int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, hwaddr iova, ram_addr_t size) { diff --git a/backends/trace-events b/backends/trace-events index 40811a3..f478e18 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -11,6 +11,7 @@ iommufd_backend_connect(int fd, bool owned, uint32_t users) "fd=%d owned=%d user iommufd_backend_disconnect(int fd, uint32_t users) "fd=%d users=%d" iommu_backend_set_fd(int fd) "pre-opened /dev/iommu fd=%d" iommufd_backend_map_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, void *vaddr, bool readonly, int ret) " iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" addr=%p readonly=%d (%d)" +iommufd_backend_map_file_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, int fd, unsigned long start, bool readonly, int ret) " iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" fd=%d start=%ld readonly=%d (%d)" iommufd_backend_unmap_dma_non_exist(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, int ret) " Unmap nonexistent mapping: iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" (%d)" iommufd_backend_unmap_dma(int iommufd, uint32_t ioas, uint64_t iova, uint64_t size, int ret) " iommufd=%d ioas=%d iova=0x%"PRIx64" size=0x%"PRIx64" (%d)" iommufd_backend_alloc_ioas(int iommufd, uint32_t ioas) " iommufd=%d ioas=%d" diff --git a/include/system/iommufd.h b/include/system/iommufd.h index cbab75b..ac700b8 100644 --- a/include/system/iommufd.h +++ b/include/system/iommufd.h @@ -43,6 +43,9 @@ void iommufd_backend_disconnect(IOMMUFDBackend *be); bool iommufd_backend_alloc_ioas(IOMMUFDBackend *be, uint32_t *ioas_id, Error **errp); void iommufd_backend_free_id(IOMMUFDBackend *be, uint32_t id); +int iommufd_backend_map_file_dma(IOMMUFDBackend *be, uint32_t ioas_id, + hwaddr iova, ram_addr_t size, int fd, + unsigned long start, bool readonly); int iommufd_backend_map_dma(IOMMUFDBackend *be, uint32_t ioas_id, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly); int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, From patchwork Fri Feb 14 14:14:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49C0DC02198 for ; Fri, 14 Feb 2025 14:16:51 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUL-0006wh-Ve; Fri, 14 Feb 2025 09:16:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTj-0006Ex-1X for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:39 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTV-0004iY-9L for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:30 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtT8T028517; Fri, 14 Feb 2025 14:15:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=GeUQU0ZCwycpRnEhS4qtbct7En6toBtlsSLbV4izNyI=; b= RtDeXdU63mw8MPbvzneNunNGSnZH8lmpT/ctxaAjsk4Jey4LjwYNc2f7All89B/v JObwRd/F7yjp+t7n59/JBmRpeqpqkc+bA1AwL+AvGrTgFAjAxalfuhC1DnmMCQhY M8wb8cS5I6heG8NXA0Z3dpMnHSkaJM/5IoAnX9mKE25WCka2rqkD6vwsH0ohaNhl 3w7Q1usGv2PR+UuR9xe6KDNjacJzOZBWOC0gfdOoZCDlYH0q3chgatdyJit5FZCn mDNT1wWzD/KBvoDED/uiFE+V1ExeATekID3nWS42UyJBC3CVVLdFVWUfwA3gGNep 5GDHSoAzfzxxXVixj1RHJg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnp6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:12 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECO40i025170; Fri, 14 Feb 2025 14:15:11 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgw0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:11 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQF006920; Fri, 14 Feb 2025 14:15:10 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-32; Fri, 14 Feb 2025 14:15:10 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 31/45] backends/iommufd: change process ioctl Date: Fri, 14 Feb 2025 06:14:13 -0800 Message-Id: <1739542467-226739-32-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: YTFOYtDXbQr1CfoyQhmeml3x0J1ShswO X-Proofpoint-ORIG-GUID: YTFOYtDXbQr1CfoyQhmeml3x0J1ShswO Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define the change process ioctl Signed-off-by: Steve Sistare --- backends/iommufd.c | 20 ++++++++++++++++++++ backends/trace-events | 1 + include/system/iommufd.h | 2 ++ 3 files changed, 23 insertions(+) diff --git a/backends/iommufd.c b/backends/iommufd.c index 612de78..cc3dcff 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -73,6 +73,26 @@ static void iommufd_backend_class_init(ObjectClass *oc, void *data) object_class_property_add_str(oc, "fd", NULL, iommufd_backend_set_fd); } +bool iommufd_change_process_capable(IOMMUFDBackend *be) +{ + struct iommu_ioas_change_process args = {.size = sizeof(args)}; + + return !ioctl(be->fd, IOMMU_IOAS_CHANGE_PROCESS, &args); +} + +bool iommufd_change_process(IOMMUFDBackend *be, Error **errp) +{ + struct iommu_ioas_change_process args = {.size = sizeof(args)}; + bool ret = !ioctl(be->fd, IOMMU_IOAS_CHANGE_PROCESS, &args); + + if (!ret) { + error_setg_errno(errp, errno, "IOMMU_IOAS_CHANGE_PROCESS fd %d failed", + be->fd); + } + trace_iommufd_change_process(be->fd, ret); + return ret; +} + bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp) { int fd; diff --git a/backends/trace-events b/backends/trace-events index f478e18..5ccdf90 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -7,6 +7,7 @@ dbus_vmstate_loading(const char *id) "id: %s" dbus_vmstate_saving(const char *id) "id: %s" # iommufd.c +iommufd_change_process(int fd, bool ret) "fd=%d (%d)" iommufd_backend_connect(int fd, bool owned, uint32_t users) "fd=%d owned=%d users=%d" iommufd_backend_disconnect(int fd, uint32_t users) "fd=%d users=%d" iommu_backend_set_fd(int fd) "pre-opened /dev/iommu fd=%d" diff --git a/include/system/iommufd.h b/include/system/iommufd.h index ac700b8..db9ed53 100644 --- a/include/system/iommufd.h +++ b/include/system/iommufd.h @@ -64,6 +64,8 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id, uint64_t iova, ram_addr_t size, uint64_t page_size, uint64_t *data, Error **errp); +bool iommufd_change_process_capable(IOMMUFDBackend *be); +bool iommufd_change_process(IOMMUFDBackend *be, Error **errp); #define TYPE_HOST_IOMMU_DEVICE_IOMMUFD TYPE_HOST_IOMMU_DEVICE "-iommufd" #endif From patchwork Fri Feb 14 14:14:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7C6EC021A4 for ; Fri, 14 Feb 2025 14:29:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUZ-0007PV-7B; Fri, 14 Feb 2025 09:16:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTj-0006Ey-1f for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:39 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTX-0004il-C6 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:31 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtT5A001920; Fri, 14 Feb 2025 14:15:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=Wts3mBRNqK+I2caIjXiPPXIryy+hjiXMI9LaT5gNu9k=; b= UhIcg1HHoAhnFk5BBfYN67UIMWrr8BCXCRuAaUqQ9DkUHGLp8RT7G+D/TWERkGyF A0hbq5PBVZeTEKsIruKwv+xR9I6Xg26XNrolj2S3YXUGH+j8+qCcwnPH1Rq2BTHz Yl6pJFiHLZEeYjfwvk/ZyrQBF/GR3WSb4RawAeiU0wlbsrts2MIogP2oj3GpjaaV 7tIHjFD29ETuntodcEwyBX9oYWZauuiKkqIYHc1enLzAZThGEGv+/2twoJkNoYZB 0224udrLoItyQRvjV/L8yYk9Yk8pE3sAnRI6J9DLSCHT47Ai3gfp/Zlx9hws/3Vi tHfty37h6EIWy31a6u7Jlg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyuswp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:13 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECO40k025170; Fri, 14 Feb 2025 14:15:12 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgx7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:12 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQH006920; Fri, 14 Feb 2025 14:15:12 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-33; Fri, 14 Feb 2025 14:15:12 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 32/45] physmem: qemu_ram_get_fd_offset Date: Fri, 14 Feb 2025 06:14:14 -0800 Message-Id: <1739542467-226739-33-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: hU3oG7veGmF700wue36VZAvuTpjpqnn3 X-Proofpoint-GUID: hU3oG7veGmF700wue36VZAvuTpjpqnn3 Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define qemu_ram_get_fd_offset, so CPR can map a memory region using IOMMU_IOAS_MAP_FILE in a subsequent patch. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu --- include/exec/cpu-common.h | 1 + system/physmem.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h index b1d76d6..0cab252 100644 --- a/include/exec/cpu-common.h +++ b/include/exec/cpu-common.h @@ -95,6 +95,7 @@ void qemu_ram_unset_idstr(RAMBlock *block); const char *qemu_ram_get_idstr(RAMBlock *rb); void *qemu_ram_get_host_addr(RAMBlock *rb); ram_addr_t qemu_ram_get_offset(RAMBlock *rb); +ram_addr_t qemu_ram_get_fd_offset(RAMBlock *rb); ram_addr_t qemu_ram_get_used_length(RAMBlock *rb); ram_addr_t qemu_ram_get_max_length(RAMBlock *rb); bool qemu_ram_is_shared(RAMBlock *rb); diff --git a/system/physmem.c b/system/physmem.c index 0bcfc6c..c41a80b 100644 --- a/system/physmem.c +++ b/system/physmem.c @@ -1569,6 +1569,11 @@ ram_addr_t qemu_ram_get_offset(RAMBlock *rb) return rb->offset; } +ram_addr_t qemu_ram_get_fd_offset(RAMBlock *rb) +{ + return rb->fd_offset; +} + ram_addr_t qemu_ram_get_used_length(RAMBlock *rb) { return rb->used_length; From patchwork Fri Feb 14 14:14:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B6F29C021A6 for ; Fri, 14 Feb 2025 14:20:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUd-0007tZ-F5; Fri, 14 Feb 2025 09:16:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTj-0006Ew-1m for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:39 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTZ-0004iq-If for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:33 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtUrd001527; Fri, 14 Feb 2025 14:15:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=a1/vOxpkzs6n371MsFbqbfBiYksjx83GHsEJPM8R1x0=; b= nAQ5HDDFlqVierj+TDWVzv9VIsLkyZ78Dqg2mJngvga/sHgKdD9ELYBxTClMlVSD sxIqn0bXHxQ4zQrSRxT1QLP2uxbOyES55hUA2yKJVCLNU6J7JPTNJnelNWVX2m9I 74Bbs74tANoWybvsVqaiYoWmqQA5chuocNwX+7xtZHqxLvZwRCIeFG0AnrcnT3sb 5Tsg15ewohE5bvPn8WaTEL3B1h1ygjkH1uJC8EDwC5VNop5eRVD/jblfekpizATW 18HDHO3qxiovVVxTyzD/pImfBO1sNO1hI+FABdprMANO13hlVLNW2eAwfS/W+oSp LxUtUGKagrgW++zLwcdV0g== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0q2krva-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:15 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EDXYq1025140; Fri, 14 Feb 2025 14:15:14 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksgy7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:14 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQJ006920; Fri, 14 Feb 2025 14:15:13 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-34; Fri, 14 Feb 2025 14:15:13 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 33/45] vfio/iommufd: use IOMMU_IOAS_MAP_FILE Date: Fri, 14 Feb 2025 06:14:15 -0800 Message-Id: <1739542467-226739-34-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: O9ZYPYQv__iRanDTV0K4_r6kcNxqdV-k X-Proofpoint-ORIG-GUID: O9ZYPYQv__iRanDTV0K4_r6kcNxqdV-k Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Use IOMMU_IOAS_MAP_FILE when the mapped region is backed by a file. Such a mapping can be preserved without modification during CPR, because it depends on the file's address space, which does not change, rather than on the process's address space, which does change. Signed-off-by: Steve Sistare --- hw/vfio/container-base.c | 9 +++++++++ hw/vfio/iommufd.c | 13 +++++++++++++ include/hw/vfio/vfio-container-base.h | 3 +++ 3 files changed, 25 insertions(+) diff --git a/hw/vfio/container-base.c b/hw/vfio/container-base.c index 302cd4c..fbaf04a 100644 --- a/hw/vfio/container-base.c +++ b/hw/vfio/container-base.c @@ -21,7 +21,16 @@ int vfio_container_dma_map(VFIOContainerBase *bcontainer, RAMBlock *rb) { VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + int mfd = rb ? qemu_ram_get_fd(rb) : -1; + if (mfd >= 0 && vioc->dma_map_file) { + unsigned long start = vaddr - qemu_ram_get_host_addr(rb); + unsigned long offset = qemu_ram_get_fd_offset(rb); + + vioc->dma_map_file(bcontainer, iova, size, mfd, start + offset, + readonly); + return 0; + } g_assert(vioc->dma_map); return vioc->dma_map(bcontainer, iova, size, vaddr, readonly); } diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index df61edf..ea40da5 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -38,6 +38,18 @@ static int iommufd_cdev_map(const VFIOContainerBase *bcontainer, hwaddr iova, iova, size, vaddr, readonly); } +static int iommufd_cdev_map_file(const VFIOContainerBase *bcontainer, + hwaddr iova, ram_addr_t size, + int fd, unsigned long start, bool readonly) +{ + const VFIOIOMMUFDContainer *container = + container_of(bcontainer, VFIOIOMMUFDContainer, bcontainer); + + return iommufd_backend_map_file_dma(container->be, + container->ioas_id, + iova, size, fd, start, readonly); +} + static int iommufd_cdev_unmap(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb) @@ -794,6 +806,7 @@ static void vfio_iommu_iommufd_class_init(ObjectClass *klass, void *data) vioc->hiod_typename = TYPE_HOST_IOMMU_DEVICE_IOMMUFD_VFIO; vioc->dma_map = iommufd_cdev_map; + vioc->dma_map_file = iommufd_cdev_map_file; vioc->dma_unmap = iommufd_cdev_unmap; vioc->attach_device = iommufd_cdev_attach; vioc->detach_device = iommufd_cdev_detach; diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index d82e256..4daa5f8 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -115,6 +115,9 @@ struct VFIOIOMMUClass { int (*dma_map)(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly); + int (*dma_map_file)(const VFIOContainerBase *bcontainer, + hwaddr iova, ram_addr_t size, + int fd, unsigned long start, bool readonly); int (*dma_unmap)(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb); From patchwork Fri Feb 14 14:14:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 67B2DC02198 for ; Fri, 14 Feb 2025 14:20:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUp-0000Zp-V0; Fri, 14 Feb 2025 09:16:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006P4-3U for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTa-0004j2-9G for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:36 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtRB2011445; Fri, 14 Feb 2025 14:15:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=zurSjh+ycUk+qDKlD1hF4rXDMsLxZM0vRjOuN/brrwY=; b= fk/a4MZZGuMbCrSxeosBSjyAMdsF/l2zc7m/FBWNuwdoGLQ3W7lA9SPfNQwzEM1m Fv0cy9boUQPapFDi0eF9bYgIDaSdFT1X6R1TwIEf2HpHrlUaJHvcDLSAJbH/Zymv kDO/K8QRxwRlnonVb0GRW1OLXg0OMI9lk5mLfYodpZSCifAxCv76bTjSqwOyAoZb Gp5mh7ZFValnr3C8FrFH2dYQQ4wA1guIq3dZUS1wl0Ol0NogGnnlNllhw6Hepoyu hh/vXWEsBoIX7MVFvyVTJycqY/+aA1+pO+to7zXnx8b90Sc/X/AuPUwyGFGSREOg fywTinP9eoBivzNrVOJMBg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwnp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:16 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECZ4jA025433; Fri, 14 Feb 2025 14:15:15 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh08-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:15 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQL006920; Fri, 14 Feb 2025 14:15:14 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-35; Fri, 14 Feb 2025 14:15:14 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 34/45] vfio/iommufd: export iommufd_cdev_get_info_iova_range Date: Fri, 14 Feb 2025 06:14:16 -0800 Message-Id: <1739542467-226739-35-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: kT9CuHhFyKTZH1ZSyTtmrRJNEp-9WHlI X-Proofpoint-GUID: kT9CuHhFyKTZH1ZSyTtmrRJNEp-9WHlI Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Export iommufd_cdev_get_info_iova_range, for use by CPR in a subsequent patch to reconstruct the userland device state. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/iommufd.c | 4 ++-- include/hw/vfio/vfio-common.h | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index ea40da5..a6e24a7 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -429,8 +429,8 @@ static int iommufd_cdev_ram_block_discard_disable(bool state) return ram_block_uncoordinated_discard_disable(state); } -static bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, - uint32_t ioas_id, Error **errp) +bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, + uint32_t ioas_id, Error **errp) { VFIOContainerBase *bcontainer = &container->bcontainer; g_autofree struct iommu_ioas_iova_ranges *info = NULL; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 78e4f12..9ca40d0 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -264,6 +264,9 @@ void vfio_kvm_device_close(void); bool vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **errp); void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer); +bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, + uint32_t ioas_id, Error **errp); + extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; typedef QLIST_HEAD(VFIODeviceList, VFIODevice) VFIODeviceList; From patchwork Fri Feb 14 14:14:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13974985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23385C02198 for ; Fri, 14 Feb 2025 14:16:38 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUa-0007gV-Cb; Fri, 14 Feb 2025 09:16:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTj-0006F8-II for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:40 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTa-0004iy-5K for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:33 -0500 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtSpa006439; Fri, 14 Feb 2025 14:15:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=jivM27DVJP4ghdtnYsoXrniOA5k7eFfXLtJN3lmwVyg=; b= UkDLvw3VaPX4/9N5766hVYUhRp/X2eEasQtSfVj2nDV4HOmv2ncusCF7uWyOfXXq wlMFiARWfuEOjFp/W0hTg7wUjZrYHKXJGh/xG+IVVxmsS8IdXMMQXtZZrpQxM49a Zv3jGR4oB1T4sja2qwMoWOHDMMgO5MXTqQVZYC0HiMyqnTYyWAuzViNvoZbhpCfg Ic35FPPsD++IDeFdnvdVq+CZiB6Wv/1tIH0zjjeNLs9LHnsqbRSSRgnVD+0PVlTZ 71CESpKTrMN9N96pCClrKDb/WrDajbc4ZcXn4F4s160O/S+dL5pAHAa4OFSMMg7l 416socZP3PIJ9E4dhcyw6Q== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qaknwv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:17 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ED5ZWp025287; Fri, 14 Feb 2025 14:15:16 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh1j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:16 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQN006920; Fri, 14 Feb 2025 14:15:16 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-36; Fri, 14 Feb 2025 14:15:16 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 35/45] vfio/iommufd: define hwpt constructors Date: Fri, 14 Feb 2025 06:14:17 -0800 Message-Id: <1739542467-226739-36-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: wAB1pfZegL2um3YpEQbuh-fDj_0WoFOd X-Proofpoint-ORIG-GUID: wAB1pfZegL2um3YpEQbuh-fDj_0WoFOd Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Extract hwpt creation code from iommufd_cdev_autodomains_get into the helpers iommufd_cdev_set_hwpt and iommufd_cdev_make_hwpt. These will be used by CPR in a subsequent patch. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/iommufd.c | 52 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 34 insertions(+), 18 deletions(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index a6e24a7..7c0cdd7 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -287,6 +287,34 @@ static bool iommufd_cdev_detach_ioas_hwpt(VFIODevice *vbasedev, Error **errp) return true; } +static void iommufd_cdev_set_hwpt(VFIODevice *vbasedev, VFIOIOASHwpt *hwpt) +{ + vbasedev->hwpt = hwpt; + vbasedev->iommu_dirty_tracking = iommufd_hwpt_dirty_tracking(hwpt); + QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); +} + +static VFIOIOASHwpt *iommufd_cdev_make_hwpt(VFIODevice *vbasedev, + VFIOIOMMUFDContainer *container, + uint32_t hwpt_id) +{ + VFIOIOASHwpt *hwpt = g_malloc0(sizeof(*hwpt)); + uint32_t flags = 0; + + if (vbasedev->hiod->caps.hw_caps & IOMMU_HW_CAP_DIRTY_TRACKING) { + flags = IOMMU_HWPT_ALLOC_DIRTY_TRACKING; + } + + hwpt->hwpt_id = hwpt_id; + hwpt->hwpt_flags = flags; + QLIST_INIT(&hwpt->device_list); + + QLIST_INSERT_HEAD(&container->hwpt_list, hwpt, next); + container->bcontainer.dirty_pages_supported |= + vbasedev->iommu_dirty_tracking; + return hwpt; +} + static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev, VFIOIOMMUFDContainer *container, Error **errp) @@ -316,13 +344,10 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev, return false; } else { - vbasedev->hwpt = hwpt; - QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); - vbasedev->iommu_dirty_tracking = iommufd_hwpt_dirty_tracking(hwpt); + iommufd_cdev_set_hwpt(vbasedev, hwpt); return true; } } - /* * This is quite early and VFIO Migration state isn't yet fully * initialized, thus rely only on IOMMU hardware capabilities as to @@ -341,24 +366,15 @@ static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev, return false; } - hwpt = g_malloc0(sizeof(*hwpt)); - hwpt->hwpt_id = hwpt_id; - hwpt->hwpt_flags = flags; - QLIST_INIT(&hwpt->device_list); - - ret = iommufd_cdev_attach_ioas_hwpt(vbasedev, hwpt->hwpt_id, errp); + ret = iommufd_cdev_attach_ioas_hwpt(vbasedev, hwpt_id, errp); if (ret) { - iommufd_backend_free_id(container->be, hwpt->hwpt_id); - g_free(hwpt); + iommufd_backend_free_id(container->be, hwpt_id); return false; } - vbasedev->hwpt = hwpt; - vbasedev->iommu_dirty_tracking = iommufd_hwpt_dirty_tracking(hwpt); - QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); - QLIST_INSERT_HEAD(&container->hwpt_list, hwpt, next); - container->bcontainer.dirty_pages_supported |= - vbasedev->iommu_dirty_tracking; + hwpt = iommufd_cdev_make_hwpt(vbasedev, container, hwpt_id); + iommufd_cdev_set_hwpt(vbasedev, hwpt); + if (container->bcontainer.dirty_pages_supported && !vbasedev->iommu_dirty_tracking) { warn_report("IOMMU instance for device %s doesn't support dirty tracking", From patchwork Fri Feb 14 14:14:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA851C02198 for ; Fri, 14 Feb 2025 14:27:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUt-0000mj-6a; Fri, 14 Feb 2025 09:16:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006P8-Hd for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTc-0004jI-Dh for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:38 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtRtR011440; Fri, 14 Feb 2025 14:15:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=+P4swyqEuKwf82Z7npEcCzUwXba/Uv9ke6CHl/B23lU=; b= OzmzLs34kEN9XB67C9q/0f4Bne8WYk/dnxRJJZltSD+dKWaHm/zIjoQx0jQq265c NVjFlxgEOJQWY3Jwu5wgTnx27rfb7JX5rDqVbtgEGI/zPfCxHFcrfwn4hIpwfyBl GCjvqG9uPXPOzWewkn/8AZNq27MzZnvNquO2LOEQxCR273KcQqLO5/DPsdLLezb/ sBtYFQAJx1UaTPd41/VdO5dt6+bONEbrTVmFhcpf/S3Uh0miIkQ/YWuLo4SU0DL1 hrnyEZ81JVhrk5W408p9twWtGrUqXBEtKfWQYRLT3/c+Abt0dUMFmi4xVyApIrM7 CIV9Ugt4gYWSH+JJDcw7aA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwns-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:19 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECIvBo025325; Fri, 14 Feb 2025 14:15:18 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh2y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:18 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQP006920; Fri, 14 Feb 2025 14:15:17 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-37; Fri, 14 Feb 2025 14:15:17 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 36/45] vfio/iommufd: invariant device name Date: Fri, 14 Feb 2025 06:14:18 -0800 Message-Id: <1739542467-226739-37-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: eBzRyyiEyc6R0yDlcg1rsRrW81wLNpPB X-Proofpoint-GUID: eBzRyyiEyc6R0yDlcg1rsRrW81wLNpPB Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org cpr-transfer will use the device name as a key to find the value of the device descriptor in new QEMU. However, if the descriptor number is specified by a command-line fd parameter, then vfio_device_get_name creates a name that includes the fd number. This causes a chicken-and-egg problem: new QEMU must know the fd number to construct a name to find the fd number. To fix, create an invariant name based on the id command-line parameter. If id is not defined, add a CPR blocker. Signed-off-by: Steve Sistare --- hw/vfio/cpr.c | 21 +++++++++++++++++++++ hw/vfio/helpers.c | 10 ++++------ hw/vfio/iommufd.c | 2 ++ include/hw/vfio/vfio-cpr.h | 4 ++++ 4 files changed, 31 insertions(+), 6 deletions(-) diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index a2400ca..e3ea2bf 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -11,6 +11,7 @@ #include "hw/vfio/pci.h" #include "hw/pci/msix.h" #include "hw/pci/msi.h" +#include "migration/blocker.h" #include "migration/cpr.h" #include "qapi/error.h" #include "system/runstate.h" @@ -184,3 +185,23 @@ const VMStateDescription vfio_cpr_pci_vmstate = { VMSTATE_END_OF_LIST() } }; + +bool vfio_cpr_set_device_name(VFIODevice *vbasedev, Error **errp) +{ + if (vbasedev->dev->id) { + vbasedev->name = g_strdup(vbasedev->dev->id); + return true; + } else { + /* + * Assign a name so any function printing it will not break, but the + * fd number changes across processes, so this cannot be used as an + * invariant name for CPR. + */ + vbasedev->name = g_strdup_printf("VFIO_FD%d", vbasedev->fd); + error_setg(&vbasedev->cpr.id_blocker, + "vfio device with fd=%d needs an id property", + vbasedev->fd); + return migrate_add_blocker_modes(&vbasedev->cpr.id_blocker, errp, + MIG_MODE_CPR_TRANSFER, -1) == 0; + } +} diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 4b255d4..4ff794c 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -29,6 +29,7 @@ #include "qapi/error.h" #include "qemu/error-report.h" #include "qemu/units.h" +#include "migration/cpr.h" #include "monitor/monitor.h" /* @@ -637,6 +638,7 @@ bool vfio_device_get_name(VFIODevice *vbasedev, Error **errp) { ERRP_GUARD(); struct stat st; + bool ret = true; if (vbasedev->fd < 0) { if (stat(vbasedev->sysfsdev, &st) < 0) { @@ -653,16 +655,12 @@ bool vfio_device_get_name(VFIODevice *vbasedev, Error **errp) error_setg(errp, "Use FD passing only with iommufd backend"); return false; } - /* - * Give a name with fd so any function printing out vbasedev->name - * will not break. - */ if (!vbasedev->name) { - vbasedev->name = g_strdup_printf("VFIO_FD%d", vbasedev->fd); + ret = vfio_cpr_set_device_name(vbasedev, errp); } } - return true; + return ret; } void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 7c0cdd7..2de2811 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -24,6 +24,7 @@ #include "system/reset.h" #include "qemu/cutils.h" #include "qemu/chardev_open.h" +#include "migration/blocker.h" #include "pci.h" #include "exec/ram_addr.h" @@ -661,6 +662,7 @@ static void iommufd_cdev_detach(VFIODevice *vbasedev) iommufd_cdev_container_destroy(container); vfio_put_address_space(space); + migrate_del_blocker(&vbasedev->cpr.id_blocker); iommufd_cdev_unbind_and_disconnect(vbasedev); close(vbasedev->fd); } diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index a9f2fbe..8a30d30 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -22,12 +22,14 @@ typedef struct VFIOContainerCPR { typedef struct VFIODeviceCPR { bool reused; Error *mdev_blocker; + Error *id_blocker; } VFIODeviceCPR; struct VFIOContainer; struct VFIOGroup; struct VFIOContainerBase; struct VFIOPCIDevice; +struct VFIODevice; int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp); @@ -53,4 +55,6 @@ void vfio_cpr_delete_vector_fd(struct VFIOPCIDevice *vdev, const char *name, int nr); extern const VMStateDescription vfio_cpr_pci_vmstate; + +bool vfio_cpr_set_device_name(struct VFIODevice *vbasedev, Error **errp); #endif From patchwork Fri Feb 14 14:14:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EBB3DC02198 for ; Fri, 14 Feb 2025 14:21:57 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUv-0000uv-9m; Fri, 14 Feb 2025 09:16:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006P9-I3 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTd-0004jW-Ii for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:43 -0500 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuA16019644; Fri, 14 Feb 2025 14:15:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=b2G/sU4A3dYc8xXBcjXxdR20Ew4Qo4gFlTYVkFtXgBA=; b= oICfJTax26do+pJhQ57fwC9g+Hgnml95gksk1Ac61StBTAcXl+UHUqAtO680vB8B F/McVZnY1r6DTlFTkAWwq3KcdHBW02gNtdXuIfmhsBiH0Yut2eux2isL3aQot7i8 uglhBIZHCHXwQjJks9Ghj2wGLOuTO85JfcbnHZq1odaIUx+AdlceNsFfbvW0EmDs Kn10jM2FQlMRsY10Hh5E2cGp+Ee7lysIqkN0YP1gQlwihLrstG7rhwZ4aItQiWri p/wEoj+XJfjHsMo2aaid2JmJ+uyJv6YXxoX9ykAqztaMAX/l6pqvNCbgkQzYxNuc 7n2RnK5C0qBT436QSd0WOA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0t4bmvf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:21 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECihHR025381; Fri, 14 Feb 2025 14:15:20 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh4g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:19 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQR006920; Fri, 14 Feb 2025 14:15:19 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-38; Fri, 14 Feb 2025 14:15:19 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 37/45] vfio/iommufd: fix cpr register Date: Fri, 14 Feb 2025 06:14:19 -0800 Message-Id: <1739542467-226739-38-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: eZMYzM_SbgIS2PXqHOP9zou5OrCquI0Y X-Proofpoint-GUID: eZMYzM_SbgIS2PXqHOP9zou5OrCquI0Y Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org iommufd_cdev_attach should not call vfio_cpr_register_container if an existing container is found. Fix that by registering earlier in the code flow, which requires an additional call to unregister during error recovery. Note it is safe to call unregister even if register has not been called. Signed-off-by: Steve Sistare --- hw/vfio/iommufd.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 2de2811..87c3bc2c 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -596,6 +596,10 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, bcontainer->initialized = true; + if (!vfio_cpr_register_container(bcontainer, errp)) { + goto err_listener_register; + } + found_container: ret = ioctl(devfd, VFIO_DEVICE_GET_INFO, &dev_info); if (ret) { @@ -603,10 +607,6 @@ found_container: goto err_listener_register; } - if (!vfio_cpr_register_container(bcontainer, errp)) { - goto err_listener_register; - } - /* * TODO: examine RAM_BLOCK_DISCARD stuff, should we do group level * for discarding incompatibility check as well? @@ -629,6 +629,7 @@ found_container: return true; err_listener_register: + vfio_cpr_unregister_container(bcontainer); iommufd_cdev_ram_block_discard_disable(false); err_discard_disable: iommufd_cdev_detach_container(vbasedev, container); From patchwork Fri Feb 14 14:14:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C1D4EC021A6 for ; Fri, 14 Feb 2025 14:23:27 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwV2-0001U9-7w; Fri, 14 Feb 2025 09:17:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006PD-HY for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTg-0004jl-68 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:40 -0500 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECuZE9002760; Fri, 14 Feb 2025 14:15:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=i068UHPtSe3Srh6HjpHZBZMHQ6vg8Ckh26gSYkmd/uQ=; b= n4+mp8GTBeozGtmYfRMwJvVDdEpdBhopaRsooevpSdRwqUvyebzIAAgSdmWMs60Q E/ltuLQd11nJJ7fyB0pdiaslVFQ4TBGZPuoR++K7QsgWF0gLS8rdVH+WDhcyDiCf fMXqusnwhrRdH+vn5uj1bwEyDv8Tva74R5H7+J0PHYgqTgE0NX4tKL6Tga3Ww2Da 4Ob39CgBq1qwkwmhmygYl3SV9eonGj31gCDAl8hHQsU2R/xaUYMdE5bQwL906Aan 5zQtnuYMZRtWeRH8RTzeLF/A8j1+1n5WIsgTzSUG3Ct+KzFd8pVhi+5NfCddpW/2 BkkjEuTB3togy92umuuCmQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0q2krvj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:22 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EC5VhD025186; Fri, 14 Feb 2025 14:15:21 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh60-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:21 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQT006920; Fri, 14 Feb 2025 14:15:20 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-39; Fri, 14 Feb 2025 14:15:20 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 38/45] vfio/iommufd: register container for cpr Date: Fri, 14 Feb 2025 06:14:20 -0800 Message-Id: <1739542467-226739-39-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: MWIoH5jvwtNehtuc9S65sro3hK7aBoDG X-Proofpoint-ORIG-GUID: MWIoH5jvwtNehtuc9S65sro3hK7aBoDG Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Register a vfio iommufd container and device for CPR, replacing the generic CPR register call with a more specific iommufd register call. Add a blocker if the kernel does not support IOMMU_IOAS_CHANGE_PROCESS. This is mostly boiler plate. The fields to to saved and restored are added in subsequent patches. Signed-off-by: Steve Sistare --- hw/vfio/cpr-iommufd.c | 97 +++++++++++++++++++++++++++++++++++++++++++ hw/vfio/iommufd.c | 8 ++-- hw/vfio/meson.build | 1 + include/hw/vfio/vfio-common.h | 1 + include/hw/vfio/vfio-cpr.h | 8 ++++ 5 files changed, 112 insertions(+), 3 deletions(-) create mode 100644 hw/vfio/cpr-iommufd.c diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c new file mode 100644 index 0000000..9fd0c82 --- /dev/null +++ b/hw/vfio/cpr-iommufd.c @@ -0,0 +1,97 @@ +/* + * Copyright (c) 2024-2025 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio-cpr.h" +#include "migration/blocker.h" +#include "migration/cpr.h" +#include "migration/migration.h" +#include "migration/vmstate.h" +#include "system/iommufd.h" + +static bool vfio_cpr_supported(VFIOIOMMUFDContainer *container, Error **errp) +{ + if (!iommufd_change_process_capable(container->be)) { + error_setg(errp, + "VFIO container does not support IOMMU_IOAS_CHANGE_PROCESS"); + return false; + } + return true; +} + +static const VMStateDescription vfio_container_vmstate = { + .name = "vfio-iommufd-container", + .version_id = 0, + .minimum_version_id = 0, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; + +static const VMStateDescription iommufd_cpr_vmstate = { + .name = "iommufd", + .version_id = 0, + .minimum_version_id = 0, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; + +bool vfio_iommufd_cpr_register_container(VFIOIOMMUFDContainer *container, + Error **errp) +{ + VFIOContainerBase *bcontainer = &container->bcontainer; + Error **cpr_blocker = &container->cpr_blocker; + + migration_add_notifier_mode(&bcontainer->cpr_reboot_notifier, + vfio_cpr_reboot_notifier, + MIG_MODE_CPR_REBOOT); + + if (!vfio_cpr_supported(container, cpr_blocker)) { + return migrate_add_blocker_modes(cpr_blocker, errp, + MIG_MODE_CPR_TRANSFER, -1) == 0; + } + + vmstate_register(NULL, -1, &vfio_container_vmstate, container); + vmstate_register(NULL, -1, &iommufd_cpr_vmstate, container->be); + + return true; +} + +void vfio_iommufd_cpr_unregister_container(VFIOIOMMUFDContainer *container) +{ + VFIOContainerBase *bcontainer = &container->bcontainer; + + vmstate_unregister(NULL, &iommufd_cpr_vmstate, container->be); + vmstate_unregister(NULL, &vfio_container_vmstate, container); + migrate_del_blocker(&container->cpr_blocker); + migration_remove_notifier(&bcontainer->cpr_reboot_notifier); +} + +static const VMStateDescription vfio_device_vmstate = { + .name = "vfio-iommufd-device", + .version_id = 0, + .minimum_version_id = 0, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; + +void vfio_iommufd_cpr_register_device(VFIODevice *vbasedev) +{ + vmstate_register(NULL, -1, &vfio_device_vmstate, vbasedev); +} + +void vfio_iommufd_cpr_unregister_device(VFIODevice *vbasedev) +{ + vmstate_unregister(NULL, &vfio_device_vmstate, vbasedev); +} diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 87c3bc2c..ddb3d23 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -596,7 +596,7 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, bcontainer->initialized = true; - if (!vfio_cpr_register_container(bcontainer, errp)) { + if (!vfio_iommufd_cpr_register_container(container, errp)) { goto err_listener_register; } @@ -623,13 +623,14 @@ found_container: vbasedev->bcontainer = bcontainer; QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); + vfio_iommufd_cpr_register_device(vbasedev); trace_iommufd_cdev_device_info(vbasedev->name, devfd, vbasedev->num_irqs, vbasedev->num_regions, vbasedev->flags); return true; err_listener_register: - vfio_cpr_unregister_container(bcontainer); + vfio_iommufd_cpr_unregister_container(container); iommufd_cdev_ram_block_discard_disable(false); err_discard_disable: iommufd_cdev_detach_container(vbasedev, container); @@ -658,12 +659,13 @@ static void iommufd_cdev_detach(VFIODevice *vbasedev) iommufd_cdev_ram_block_discard_disable(false); } - vfio_cpr_unregister_container(bcontainer); + vfio_iommufd_cpr_unregister_container(container); iommufd_cdev_detach_container(vbasedev, container); iommufd_cdev_container_destroy(container); vfio_put_address_space(space); migrate_del_blocker(&vbasedev->cpr.id_blocker); + vfio_iommufd_cpr_unregister_device(vbasedev); iommufd_cdev_unbind_and_disconnect(vbasedev); close(vbasedev->fd); } diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index 5487815..998adb5 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -13,6 +13,7 @@ vfio_ss.add(when: 'CONFIG_IOMMUFD', if_true: files( vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'cpr.c', 'cpr-legacy.c', + 'cpr-iommufd.c', 'display.c', 'pci-quirks.c', 'pci.c', diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 9ca40d0..6701393 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -110,6 +110,7 @@ typedef struct VFIOIOASHwpt { typedef struct VFIOIOMMUFDContainer { VFIOContainerBase bcontainer; IOMMUFDBackend *be; + Error *cpr_blocker; uint32_t ioas_id; QLIST_HEAD(, VFIOIOASHwpt) hwpt_list; } VFIOIOMMUFDContainer; diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index 8a30d30..fa4b928 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -30,6 +30,7 @@ struct VFIOGroup; struct VFIOContainerBase; struct VFIOPCIDevice; struct VFIODevice; +struct VFIOIOMMUFDContainer; int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp); @@ -38,6 +39,13 @@ bool vfio_legacy_cpr_register_container(struct VFIOContainer *container, Error **errp); void vfio_legacy_cpr_unregister_container(struct VFIOContainer *container); +bool vfio_iommufd_cpr_register_container(struct VFIOIOMMUFDContainer *container, + Error **errp); +void vfio_iommufd_cpr_unregister_container( + struct VFIOIOMMUFDContainer *container); +void vfio_iommufd_cpr_register_device(struct VFIODevice *vbasedev); +void vfio_iommufd_cpr_unregister_device(struct VFIODevice *vbasedev); + bool vfio_cpr_container_match(struct VFIOContainer *container, struct VFIOGroup *group, int *fd); From patchwork Fri Feb 14 14:14:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92C25C021A6 for ; Fri, 14 Feb 2025 14:27:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUR-0007Ay-1R; Fri, 14 Feb 2025 09:16:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006PA-Ha for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTg-0004k3-N1 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:40 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtSua028500; Fri, 14 Feb 2025 14:15:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=zRjjkYuqMg/7hCHaD2R+SCcAvam7aTyIE9+kZERZ6ic=; b= oKg0bPXidzPs9kWeGjZf2aEfJ+jfZJMt8gfuyGIlWq3pyHmMA2efvJRoZpBAERx4 VNT/9q1q77X+oTjHutjOPTXCb6wstnLArW2nj9RjPZJKU37qfEAgoPoCl/3Mc3d2 c2820N17fIKKw0jbuL+k8R6hXhunNOeHEI2M4j5ImiT3gutRpybhDFhD3ECR1fci mmar4O+oP+l8iNh8FCSkyIp5dCy5PZdgCa37yvfRi0JdvZhym6ev78KzvwdW9aqZ wtQDr9bE2XYejVZTKhZITwupGKcAzcUPpqqrIZR1Oakmc87NxDlaZPn/UEmq0lO1 Wgym074g58YJkTTL//mwRg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnpp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:23 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECx1SZ025131; Fri, 14 Feb 2025 14:15:22 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh75-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:22 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQV006920; Fri, 14 Feb 2025 14:15:22 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-40; Fri, 14 Feb 2025 14:15:21 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 39/45] vfio/iommufd: preserve descriptors Date: Fri, 14 Feb 2025 06:14:21 -0800 Message-Id: <1739542467-226739-40-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: h_MZDOij5Q-DSaP4rvA5q5C6XwYnof69 X-Proofpoint-ORIG-GUID: h_MZDOij5Q-DSaP4rvA5q5C6XwYnof69 Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Save the iommu and vfio device fd in CPR state when it is created. After CPR, the fd number is found in CPR state and reused. Remember the reused status for subsequent patches. The reused status is cleared when vmstate load finishes. Signed-off-by: Steve Sistare --- backends/iommufd.c | 19 ++++++++++--------- hw/vfio/cpr-iommufd.c | 15 +++++++++++++++ hw/vfio/helpers.c | 10 ++-------- hw/vfio/iommufd.c | 10 +++++++++- include/system/iommufd.h | 1 + 5 files changed, 37 insertions(+), 18 deletions(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index cc3dcff..da90b21 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -16,12 +16,18 @@ #include "qemu/module.h" #include "qom/object_interfaces.h" #include "qemu/error-report.h" +#include "migration/cpr.h" #include "monitor/monitor.h" #include "trace.h" #include "hw/vfio/vfio-common.h" #include #include +static const char *iommufd_fd_name(IOMMUFDBackend *be) +{ + return object_get_canonical_path_component(OBJECT(be)); +} + static void iommufd_backend_init(Object *obj) { IOMMUFDBackend *be = IOMMUFD_BACKEND(obj); @@ -47,9 +53,8 @@ static void iommufd_backend_set_fd(Object *obj, const char *str, Error **errp) IOMMUFDBackend *be = IOMMUFD_BACKEND(obj); int fd = -1; - fd = monitor_fd_param(monitor_cur(), str, errp); + fd = cpr_get_fd_param(iommufd_fd_name(be), str, 0, &be->cpr_reused, errp); if (fd == -1) { - error_prepend(errp, "Could not parse remote object fd %s:", str); return; } be->fd = fd; @@ -95,14 +100,9 @@ bool iommufd_change_process(IOMMUFDBackend *be, Error **errp) bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp) { - int fd; - if (be->owned && !be->users) { - fd = qemu_open("/dev/iommu", O_RDWR, errp); - if (fd < 0) { - return false; - } - be->fd = fd; + be->fd = cpr_open_fd("/dev/iommu", O_RDWR, iommufd_fd_name(be), 0, + &be->cpr_reused, errp); } be->users++; @@ -121,6 +121,7 @@ void iommufd_backend_disconnect(IOMMUFDBackend *be) be->fd = -1; } out: + cpr_delete_fd(iommufd_fd_name(be), 0); trace_iommufd_backend_disconnect(be->fd, be->users); } diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index 9fd0c82..8c2fa3a 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -25,10 +25,25 @@ static bool vfio_cpr_supported(VFIOIOMMUFDContainer *container, Error **errp) return true; } +static int vfio_container_post_load(void *opaque, int version_id) +{ + VFIOIOMMUFDContainer *container = opaque; + VFIOContainerBase *bcontainer = &container->bcontainer; + VFIODevice *vbasedev; + + QLIST_FOREACH(vbasedev, &bcontainer->device_list, container_next) { + vbasedev->cpr.reused = false; + } + container->be->cpr_reused = false; + + return 0; +} + static const VMStateDescription vfio_container_vmstate = { .name = "vfio-iommufd-container", .version_id = 0, .minimum_version_id = 0, + .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { VMSTATE_END_OF_LIST() diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 4ff794c..679d33b 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -665,14 +665,8 @@ bool vfio_device_get_name(VFIODevice *vbasedev, Error **errp) void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp) { - ERRP_GUARD(); - int fd = monitor_fd_param(monitor_cur(), str, errp); - - if (fd < 0) { - error_prepend(errp, "Could not parse remote object fd %s:", str); - return; - } - vbasedev->fd = fd; + vbasedev->fd = cpr_get_fd_param(vbasedev->dev->id, str, 0, + &vbasedev->cpr.reused, errp); } void vfio_device_init(VFIODevice *vbasedev, int type, VFIODeviceOps *ops, diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index ddb3d23..6c44303 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -25,6 +25,7 @@ #include "qemu/cutils.h" #include "qemu/chardev_open.h" #include "migration/blocker.h" +#include "migration/cpr.h" #include "pci.h" #include "exec/ram_addr.h" @@ -502,13 +503,18 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, VFIO_IOMMU_CLASS(object_class_by_name(TYPE_VFIO_IOMMU_IOMMUFD)); if (vbasedev->fd < 0) { - devfd = iommufd_cdev_getfd(vbasedev->sysfsdev, errp); + devfd = cpr_find_fd(vbasedev->name, 0); + vbasedev->cpr.reused = (devfd >= 0); + if (!vbasedev->cpr.reused) { + devfd = iommufd_cdev_getfd(vbasedev->sysfsdev, errp); + } if (devfd < 0) { return false; } vbasedev->fd = devfd; } else { devfd = vbasedev->fd; + /* reused was set in iommufd_backend_set_fd */ } if (!iommufd_cdev_connect_and_bind(vbasedev, errp)) { @@ -624,6 +630,7 @@ found_container: QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); vfio_iommufd_cpr_register_device(vbasedev); + cpr_resave_fd(vbasedev->name, 0, devfd); trace_iommufd_cdev_device_info(vbasedev->name, devfd, vbasedev->num_irqs, vbasedev->num_regions, vbasedev->flags); @@ -666,6 +673,7 @@ static void iommufd_cdev_detach(VFIODevice *vbasedev) migrate_del_blocker(&vbasedev->cpr.id_blocker); vfio_iommufd_cpr_unregister_device(vbasedev); + cpr_delete_fd(vbasedev->name, 0); iommufd_cdev_unbind_and_disconnect(vbasedev); close(vbasedev->fd); } diff --git a/include/system/iommufd.h b/include/system/iommufd.h index db9ed53..5c17abd 100644 --- a/include/system/iommufd.h +++ b/include/system/iommufd.h @@ -32,6 +32,7 @@ struct IOMMUFDBackend { /*< protected >*/ int fd; /* /dev/iommu file descriptor */ bool owned; /* is the /dev/iommu opened internally */ + bool cpr_reused; /* fd is reused after CPR */ uint32_t users; /*< public >*/ From patchwork Fri Feb 14 14:14:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975028 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 80B44C02198 for ; Fri, 14 Feb 2025 14:25:47 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwV8-00025e-Iy; Fri, 14 Feb 2025 09:17:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006PC-Hd for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTh-0004kJ-UB for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:41 -0500 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtVTK017784; Fri, 14 Feb 2025 14:15:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=dKRcX6DHqijWWZrMD4CJ2+AikL+wSCrqQlK6hbgIOdk=; b= WgZPnIsg1qiP6LoYYv6LbdCAkdZcZ29+8aNzrjAS8n4AUgdH56cQQQ8dMC3MNpm9 nMJfeged8ZsTc5x3O7m3X6Zo342ipEfclyBgVl/kxDMvRw4PRuMOhY+m5qy1uJFX /2L7fk/mKjRC1kQzfldiR7QNMre3FnUDKD/dIzonlv75N7N0cVZ18uWoOdLxgXSu +Ma0aMbKhIrZ6m452iC/vZYnCAW23KFMN4o4sy+ZImEIU0x8K+GeaaFR4gKpTRz2 Ko8xSytamhSNJaJX0COGQ7+eTfJ8TGpaYxe+FGsgyt1GScZYBzqu5x4g/8OmDbSO ++PUVE1KMOHNh3HtDVLmQg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tgbtwy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:25 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECSfc2025328; Fri, 14 Feb 2025 14:15:23 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh82-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:23 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQX006920; Fri, 14 Feb 2025 14:15:23 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-41; Fri, 14 Feb 2025 14:15:23 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 40/45] vfio/iommufd: reconstruct device Date: Fri, 14 Feb 2025 06:14:22 -0800 Message-Id: <1739542467-226739-41-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: BCWYvWp9vdbwTwDQvTzHQAp2VaxarUgx X-Proofpoint-ORIG-GUID: BCWYvWp9vdbwTwDQvTzHQAp2VaxarUgx Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Reconstruct userland device state after CPR. During vfio_realize, skip all ioctls that configure the device, as it was already configured in old QEMU. Save the ioas_id in vmstate, and skip its allocation in vfio_realize. Because we skip ioctl's, it is not needed at realize time. However, we do need the range info, so defer the call to iommufd_cdev_get_info_iova_range to a post_load handler, at which time the ioas_id is known. This reconstruction is not complete. hwpt_id and devid need special treatment, handled in subsequent patches. Signed-off-by: Steve Sistare --- hw/vfio/cpr-iommufd.c | 8 ++++++++ hw/vfio/iommufd.c | 23 +++++++++++++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index 8c2fa3a..8453d76 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -30,6 +30,13 @@ static int vfio_container_post_load(void *opaque, int version_id) VFIOIOMMUFDContainer *container = opaque; VFIOContainerBase *bcontainer = &container->bcontainer; VFIODevice *vbasedev; + Error *err = NULL; + uint32_t ioas_id = container->ioas_id; + + if (!iommufd_cdev_get_info_iova_range(container, ioas_id, &err)) { + error_report_err(err); + return -1; + } QLIST_FOREACH(vbasedev, &bcontainer->device_list, container_next) { vbasedev->cpr.reused = false; @@ -46,6 +53,7 @@ static const VMStateDescription vfio_container_vmstate = { .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { + VMSTATE_UINT32(ioas_id, VFIOIOMMUFDContainer), VMSTATE_END_OF_LIST() } }; diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 6c44303..3fc530d 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -99,6 +99,10 @@ static bool iommufd_cdev_connect_and_bind(VFIODevice *vbasedev, Error **errp) goto err_kvm_device_add; } + if (vbasedev->cpr.reused) { + goto skip_bind; + } + /* Bind device to iommufd */ bind.iommufd = iommufd->fd; if (ioctl(vbasedev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind)) { @@ -110,6 +114,8 @@ static bool iommufd_cdev_connect_and_bind(VFIODevice *vbasedev, Error **errp) vbasedev->devid = bind.out_devid; trace_iommufd_cdev_connect_and_bind(bind.iommufd, vbasedev->name, vbasedev->fd, vbasedev->devid); + +skip_bind: return true; err_bind: iommufd_cdev_kvm_device_del(vbasedev); @@ -541,7 +547,8 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, vbasedev->iommufd != container->be) { continue; } - if (!iommufd_cdev_attach_container(vbasedev, container, &err)) { + if (!vbasedev->cpr.reused && + !iommufd_cdev_attach_container(vbasedev, container, &err)) { const char *msg = error_get_pretty(err); trace_iommufd_cdev_fail_attach_existing_container(msg); @@ -558,6 +565,11 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, } } + if (vbasedev->cpr.reused) { + ioas_id = -1; /* ioas_id will be received from vmstate */ + goto skip_ioas_alloc; + } + /* Need to allocate a new dedicated container */ if (!iommufd_backend_alloc_ioas(vbasedev->iommufd, &ioas_id, errp)) { goto err_alloc_ioas; @@ -565,6 +577,7 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, trace_iommufd_cdev_alloc_ioas(vbasedev->iommufd->fd, ioas_id); +skip_ioas_alloc: container = VFIO_IOMMU_IOMMUFD(object_new(TYPE_VFIO_IOMMU_IOMMUFD)); container->be = vbasedev->iommufd; container->ioas_id = ioas_id; @@ -573,7 +586,8 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, bcontainer = &container->bcontainer; vfio_address_space_insert(space, bcontainer); - if (!iommufd_cdev_attach_container(vbasedev, container, errp)) { + if (!vbasedev->cpr.reused && + !iommufd_cdev_attach_container(vbasedev, container, errp)) { goto err_attach_container; } @@ -583,6 +597,10 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, goto err_discard_disable; } + if (vbasedev->cpr.reused) { + goto skip_info; + } + if (!iommufd_cdev_get_info_iova_range(container, ioas_id, &err)) { error_append_hint(&err, "Fallback to default 64bit IOVA range and 4K page size\n"); @@ -591,6 +609,7 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, bcontainer->pgsizes = qemu_real_host_page_size(); } +skip_info: bcontainer->listener = vfio_memory_listener; memory_listener_register(&bcontainer->listener, bcontainer->space->as); From patchwork Fri Feb 14 14:14:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8919BC02198 for ; Fri, 14 Feb 2025 14:22:19 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUd-0007vx-Pj; Fri, 14 Feb 2025 09:16:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTp-0006PB-IL for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTi-0004kT-Nc for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:42 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtRB5011445; Fri, 14 Feb 2025 14:15:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=+Yztyi6Ja1nZDKOCyh8rDA3zvhN8dFHXhj/9UC3pi/Q=; b= nTt2gPQ8oXe9KajZLRO1B3yvxeY3m1i4cT0jJT54ASwZc1grGdU3T/Thc5v5Qwro s4qQujPUNqYAosdOt8qU+JCCqyOmJ+L63zVChmuKAm54l62qmp1sFqHGzxxJBXM8 6XGNkfO7z4dQYcHN+7ip/jnSA0N+443EZuoIoeXeKchUAnXEb+sNbiiC6X+qK4zj PUIFVdy1rboe6lVfPX+ohXJMVJ5Lw0aGnyCFF1BINJ5jKzsCN1++PEy3X7elsn4l zPf+IocvkIvHWihkG/z5it996+63vllOFuhRZnNbcxwwLEeRcDwuxMTZ8ArrwWMI 0s4ImPOGwLcBrCGiZm+9SQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwp2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:25 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51EC5VhG025186; Fri, 14 Feb 2025 14:15:25 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh8n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:25 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQZ006920; Fri, 14 Feb 2025 14:15:24 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-42; Fri, 14 Feb 2025 14:15:24 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 41/45] vfio/iommufd: reconstruct hw_caps Date: Fri, 14 Feb 2025 06:14:23 -0800 Message-Id: <1739542467-226739-42-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: 3nyD_Tz5LaZCMiKI3UF1WWLibIgJotX9 X-Proofpoint-GUID: 3nyD_Tz5LaZCMiKI3UF1WWLibIgJotX9 Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org hw_caps is normally derived during realize, at vfio_device_hiod_realize -> hiod_iommufd_vfio_realize -> iommufd_backend_get_device_info. However, this depends on the devid, which is not preserved during CPR. Save devid in vmstate. Defer the vfio_device_hiod_realize call to post_load time, after devid has been recovered from vmstate. Signed-off-by: Steve Sistare --- hw/vfio/cpr-iommufd.c | 14 ++++++++++++++ hw/vfio/iommufd.c | 3 ++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index 8453d76..a1ac517 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -99,12 +99,26 @@ void vfio_iommufd_cpr_unregister_container(VFIOIOMMUFDContainer *container) migration_remove_notifier(&bcontainer->cpr_reboot_notifier); } +static int vfio_device_post_load(void *opaque, int version_id) +{ + VFIODevice *vbasedev = opaque; + Error *err = NULL; + + if (!vfio_device_hiod_realize(vbasedev, &err)) { + error_report_err(err); + return false; + } + return true; +} + static const VMStateDescription vfio_device_vmstate = { .name = "vfio-iommufd-device", .version_id = 0, .minimum_version_id = 0, + .post_load = vfio_device_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { + VMSTATE_INT32(devid, VFIODevice), VMSTATE_END_OF_LIST() } }; diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 3fc530d..693ed19 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -536,7 +536,8 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, * FD to be connected and having a devid to be able to successfully call * iommufd_backend_get_device_info(). */ - if (!vfio_device_hiod_realize(vbasedev, errp)) { + if (!vbasedev->cpr.reused && + !vfio_device_hiod_realize(vbasedev, errp)) { goto err_alloc_ioas; } From patchwork Fri Feb 14 14:14:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975019 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 868FBC021AA for ; Fri, 14 Feb 2025 14:24:33 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUe-0007zk-E8; Fri, 14 Feb 2025 09:16:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTr-0006UR-Ku for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTk-0004kZ-Le for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:46 -0500 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtfkF011618; Fri, 14 Feb 2025 14:15:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=JiVpgb+Q1o93jKYbFztC7nGqDo9Po9K1XF/ulP7GSAI=; b= CVM+qVCk/o1VUHQnmvJCHfcFFFhgh3RYRKKI8JWTNtTUCmcaWBhmaVmPfYj9EdOg GKJowPDVnVc4rGctzH2kpz5exiUavXbRyy2Y82b1cTwuKsU+lYG0EYHhmuUStpPU Vh56512bN+ZNB6ZpbazsnQhy9FUxRX7A24Kxs4Tpbu3jLy4WEpcNvrk/C0yl7hip Y3m/I0SJr0Zis6gCyTbq2juX0bSF8yo+X0PsgH/PpwTYWOFmZwq8zE/Y5fb+Qm5X iWPpoj7qx2EA+PBk5BM+hL3e8Y/CnjnonV55Wl8wXp9/ezPcECqqvULSoRiFXbMu gU5yp286d1+WekC2uyuMNw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0sqbwp3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:26 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECSkjj025168; Fri, 14 Feb 2025 14:15:26 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqksh9j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:26 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQb006920; Fri, 14 Feb 2025 14:15:25 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-43; Fri, 14 Feb 2025 14:15:25 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 42/45] vfio/iommufd: reconstruct hwpt Date: Fri, 14 Feb 2025 06:14:24 -0800 Message-Id: <1739542467-226739-43-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: hym0yEbttVOxLou658T3fB4gzoH4Ncjm X-Proofpoint-GUID: hym0yEbttVOxLou658T3fB4gzoH4Ncjm Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Save the hwpt_id in vmstate. In realize, skip its allocation from iommufd_cdev_attach -> iommufd_cdev_attach_container -> iommufd_cdev_autodomains_get. Rebuild userland structures to hold hwpt_id by calling iommufd_cdev_rebuild_hwpt at post load time. This depends on hw_caps, which was restored by the post_load call to vfio_device_hiod_realize. Signed-off-by: Steve Sistare --- hw/vfio/cpr-iommufd.c | 7 +++++++ hw/vfio/iommufd.c | 19 +++++++++++++++++++ hw/vfio/trace-events | 1 + include/hw/vfio/vfio-common.h | 2 ++ include/hw/vfio/vfio-cpr.h | 1 + 5 files changed, 30 insertions(+) diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index a1ac517..4b78ebf 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -108,6 +108,12 @@ static int vfio_device_post_load(void *opaque, int version_id) error_report_err(err); return false; } + if (!vbasedev->mdev) { + VFIOIOMMUFDContainer *container = container_of(vbasedev->bcontainer, + VFIOIOMMUFDContainer, + bcontainer); + iommufd_cdev_rebuild_hwpt(vbasedev, container); + } return true; } @@ -119,6 +125,7 @@ static const VMStateDescription vfio_device_vmstate = { .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { VMSTATE_INT32(devid, VFIODevice), + VMSTATE_UINT32(cpr.hwpt_id, VFIODevice), VMSTATE_END_OF_LIST() } }; diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 693ed19..cf92c96 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -298,6 +298,7 @@ static bool iommufd_cdev_detach_ioas_hwpt(VFIODevice *vbasedev, Error **errp) static void iommufd_cdev_set_hwpt(VFIODevice *vbasedev, VFIOIOASHwpt *hwpt) { vbasedev->hwpt = hwpt; + vbasedev->cpr.hwpt_id = hwpt->hwpt_id; vbasedev->iommu_dirty_tracking = iommufd_hwpt_dirty_tracking(hwpt); QLIST_INSERT_HEAD(&hwpt->device_list, vbasedev, hwpt_next); } @@ -323,6 +324,24 @@ static VFIOIOASHwpt *iommufd_cdev_make_hwpt(VFIODevice *vbasedev, return hwpt; } +void iommufd_cdev_rebuild_hwpt(VFIODevice *vbasedev, + VFIOIOMMUFDContainer *container) +{ + VFIOIOASHwpt *hwpt; + int hwpt_id = vbasedev->cpr.hwpt_id; + + trace_iommufd_cdev_rebuild_hwpt(container->be->fd, hwpt_id); + + QLIST_FOREACH(hwpt, &container->hwpt_list, next) { + if (hwpt->hwpt_id == hwpt_id) { + iommufd_cdev_set_hwpt(vbasedev, hwpt); + return; + } + } + hwpt = iommufd_cdev_make_hwpt(vbasedev, container, hwpt_id); + iommufd_cdev_set_hwpt(vbasedev, hwpt); +} + static bool iommufd_cdev_autodomains_get(VFIODevice *vbasedev, VFIOIOMMUFDContainer *container, Error **errp) diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index cab1cf1..25ff04c 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -176,6 +176,7 @@ iommufd_cdev_connect_and_bind(int iommufd, const char *name, int devfd, int devi iommufd_cdev_getfd(const char *dev, int devfd) " %s (fd=%d)" iommufd_cdev_attach_ioas_hwpt(int iommufd, const char *name, int devfd, int id) " [iommufd=%d] Successfully attached device %s (%d) to id=%d" iommufd_cdev_detach_ioas_hwpt(int iommufd, const char *name) " [iommufd=%d] Successfully detached %s" +iommufd_cdev_rebuild_hwpt(int iommufd, int hwpt_id) " [iommufd=%d] hwpt %d" iommufd_cdev_fail_attach_existing_container(const char *msg) " %s" iommufd_cdev_alloc_ioas(int iommufd, int ioas_id) " [iommufd=%d] new IOMMUFD container with ioasid=%d" iommufd_cdev_device_info(char *name, int devfd, int num_irqs, int num_regions, int flags) " %s (%d) num_irqs=%d num_regions=%d flags=%d" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 6701393..00831b7 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -265,6 +265,8 @@ void vfio_kvm_device_close(void); bool vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **errp); void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer); +void iommufd_cdev_rebuild_hwpt(VFIODevice *vbasedev, + VFIOIOMMUFDContainer *container); bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, uint32_t ioas_id, Error **errp); diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index fa4b928..d195ce6 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -23,6 +23,7 @@ typedef struct VFIODeviceCPR { bool reused; Error *mdev_blocker; Error *id_blocker; + uint32_t hwpt_id; } VFIODeviceCPR; struct VFIOContainer; From patchwork Fri Feb 14 14:14:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 887A3C02198 for ; Fri, 14 Feb 2025 14:29:17 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUZ-0007e3-GP; Fri, 14 Feb 2025 09:16:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTq-0006UD-QM for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTo-0004kk-W0 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:46 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECteQH002102; Fri, 14 Feb 2025 14:15:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=r90gFQ21+yS0UTC/7EKonJwYjp5Qncpmusv/+zV/X8U=; b= J6eSLpIl/m0gtI+JzepFY821BnCiw263wwZ1gA5MbX6tud9F22ynOQVQXJ3HieOW NgxP5ipkGn3SPitE+kuwv3xw88u1a7BO4EcaWiBOvSUSSH3/YX2y4zde8r5uEquC LgbkiPmKKMyZbOYqRaCP4TmnHB7R+bDsDyy9fwGmwWAJVczQT/2pPF3BXjYtH0G1 pKHp9XtHp2lNu3CRES4l2H9ypQX1xjp8xET3K5D951VXXvWFDplOlcsuT64bE97c nPjr+84Ioy26cpKXYynhtaWYoX04hS9UWN1RGTS/9HdwmRRHizHq7XiZbUeTpFAi XDwfg21OUG2I22MY80Z/Eg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyusx7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:28 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ED8Y7f025427; Fri, 14 Feb 2025 14:15:27 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqkshap-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:27 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQd006920; Fri, 14 Feb 2025 14:15:27 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-44; Fri, 14 Feb 2025 14:15:27 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 43/45] vfio/iommufd: change process Date: Fri, 14 Feb 2025 06:14:25 -0800 Message-Id: <1739542467-226739-44-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: fIQyY0rxt566jGcpq5Gc6POoDXJg4lnY X-Proofpoint-GUID: fIQyY0rxt566jGcpq5Gc6POoDXJg4lnY Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Finish CPR by change the owning process of the iommufd device in post load. Signed-off-by: Steve Sistare --- hw/vfio/cpr-iommufd.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index 4b78ebf..92b101d 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -46,10 +46,27 @@ static int vfio_container_post_load(void *opaque, int version_id) return 0; } +static int vfio_container_pre_save(void *opaque) +{ + VFIOIOMMUFDContainer *container = opaque; + Error *err = NULL; + + /* + * The process has not changed yet, but proactively call the ioctl, + * and it will fail if any DMA mappings are not supported. + */ + if (!iommufd_change_process(container->be, &err)) { + error_report_err(err); + return -1; + } + return 0; +} + static const VMStateDescription vfio_container_vmstate = { .name = "vfio-iommufd-container", .version_id = 0, .minimum_version_id = 0, + .pre_save = vfio_container_pre_save, .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { @@ -58,10 +75,23 @@ static const VMStateDescription vfio_container_vmstate = { } }; +static int iommufd_cpr_post_load(void *opaque, int version_id) +{ + IOMMUFDBackend *be = opaque; + Error *err = NULL; + + if (!iommufd_change_process(be, &err)) { + error_report_err(err); + return -1; + } + return 0; +} + static const VMStateDescription iommufd_cpr_vmstate = { .name = "iommufd", .version_id = 0, .minimum_version_id = 0, + .post_load = iommufd_cpr_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { VMSTATE_END_OF_LIST() From patchwork Fri Feb 14 14:14:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65820C021A4 for ; Fri, 14 Feb 2025 14:21:04 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUh-0008Nh-KA; Fri, 14 Feb 2025 09:16:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTs-0006Vy-7E for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:48 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTq-0004ks-An for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:47 -0500 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtShd001883; Fri, 14 Feb 2025 14:15:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=FnKrXU1jKMTtQwUQPGYZ/6nNzRzuGuz84bj4pjqq8bE=; b= JVkOHXAb86/8pJz6Wsw3G/QPDGdnit56USrxf6V09WsBWt9pCYhHnC8nEVjQ+EMj 3mul0ejgA67rtBSIvl7GUH3UQpTPf+IeRwCzQlzs0OPeEEaE1wz3fKZ3Wi1EVarx hPk7N3mg0G3V4i40VMhA72JX8vErTxZ6pwAR5QfH3++I3ByKag8jz7+AkBqZpnVz 2/I3D6BuiUGcQeY+KqIIKkjMdFpl6yhZEWjZe2MuL0e7OiIjqaMKj7PgAlVmTaCm r1Kk9d290wt9DnG8+XSqnuD6+Lkuzlm/s2NPbJCa29p1K2QSZ/rpF5xfTC7i7oCp 8UUG4GS36vXrD8A3eVyDhA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0qyusxa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:29 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECfZOX025148; Fri, 14 Feb 2025 14:15:29 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqkshc1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:29 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQf006920; Fri, 14 Feb 2025 14:15:28 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-45; Fri, 14 Feb 2025 14:15:28 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 44/45] iommufd: preserve DMA mappings Date: Fri, 14 Feb 2025 06:14:26 -0800 Message-Id: <1739542467-226739-45-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-ORIG-GUID: _92L9w1fwTF1OGNEKE3SYZbz6Z__w70v X-Proofpoint-GUID: _92L9w1fwTF1OGNEKE3SYZbz6Z__w70v Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org During cpr-transfer load in new QEMU, the vfio_memory_listener causes spurious calls to map and unmap DMA regions, as devices are created and the address space is built. This memory was already already mapped by the device in old QEMU, so suppress the map and unmap callbacks during CPR -- eg, if the reused flag is set. The reused flag is cleared in the post_load handler. Signed-off-by: Steve Sistare --- backends/iommufd.c | 8 ++++++++ hw/vfio/cpr-iommufd.c | 1 + 2 files changed, 9 insertions(+) diff --git a/backends/iommufd.c b/backends/iommufd.c index da90b21..dfcfd6b 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -209,6 +209,10 @@ int iommufd_backend_map_file_dma(IOMMUFDBackend *be, uint32_t ioas_id, .length = size, }; + if (be->cpr_reused) { + return 0; + } + if (!readonly) { map.flags |= IOMMU_IOAS_MAP_WRITEABLE; } @@ -240,6 +244,10 @@ int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, .length = size, }; + if (be->cpr_reused) { + return 0; + } + ret = ioctl(fd, IOMMU_IOAS_UNMAP, &unmap); /* * IOMMUFD takes mapping as some kind of object, unmapping diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index 92b101d..286597a 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -66,6 +66,7 @@ static const VMStateDescription vfio_container_vmstate = { .name = "vfio-iommufd-container", .version_id = 0, .minimum_version_id = 0, + .priority = MIG_PRI_LOW, /* Must happen after devices and groups */ .pre_save = vfio_container_pre_save, .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, From patchwork Fri Feb 14 14:14:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13975015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3B01EC02198 for ; Fri, 14 Feb 2025 14:24:31 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tiwUm-0000Uh-UV; Fri, 14 Feb 2025 09:16:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTu-0006hP-Lk for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:51 -0500 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tiwTs-0004l0-PD for qemu-devel@nongnu.org; Fri, 14 Feb 2025 09:15:50 -0500 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51ECtbTH028581; Fri, 14 Feb 2025 14:15:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :date:from:in-reply-to:message-id:references:subject:to; s= corp-2023-11-20; bh=M6n/2Nrdcr6vRHr8XUEc17Yekxu0kEsI+9Zq6TKtWq0=; b= JG7CUaXki6ZzNCFwIu9kluH3hvQ1Dc4Ok6lBzkVt5B7Gt34PYOVs3Ml9SHmGjg6U 4re9OxMQ5iFcuN3H+JZBjb/I+THaPsUOgWKhyoO6tOF2X7MUINL4lQBX8NdYyc9f kFnsM/wh8v27zhvKszhSccrjw0OhnmbHFqzxZx1PIRPbt63VRs96klBIVkrzRGlI 6Whn0teJ80zF8zc7hCGzVAe8TE9n00x4RNQLxKTlEihkqQemEYiffNwef7ec+kcu jsY8lTvD660Pj06XOv/nM8gWLNh2vWOMg4r//T3LvsQSmOqqUErq1tv1tKd6EZT9 8LBmdIRZeCo680SeftawJQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 44p0tnbnq2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:31 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 51ED0mAi025265; Fri, 14 Feb 2025 14:15:30 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 44nwqkshd1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 14 Feb 2025 14:15:30 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 51EEETQh006920; Fri, 14 Feb 2025 14:15:30 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 44nwqksg2h-46; Fri, 14 Feb 2025 14:15:29 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Alex Williamson , Cedric Le Goater , Yi Liu , Eric Auger , Zhenzhong Duan , "Michael S. Tsirkin" , Marcel Apfelbaum , Peter Xu , Fabiano Rosas , Steve Sistare Subject: [PATCH V2 45/45] vfio/container: delete old cpr register Date: Fri, 14 Feb 2025 06:14:27 -0800 Message-Id: <1739542467-226739-46-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> References: <1739542467-226739-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-14_06,2025-02-13_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 mlxscore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2501170000 definitions=main-2502140104 X-Proofpoint-GUID: CJmXIRXS5C3Ek-7Fo7vn5CkIMpFItB2r X-Proofpoint-ORIG-GUID: CJmXIRXS5C3Ek-7Fo7vn5CkIMpFItB2r Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org vfio_cpr_[un]register_container is no longer used since they were subsumed by container type-specific registration. Delete them. Signed-off-by: Steve Sistare --- hw/vfio/cpr.c | 14 -------------- include/hw/vfio/vfio-common.h | 3 --- 2 files changed, 17 deletions(-) diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index e3ea2bf..5387b31 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -6,7 +6,6 @@ */ #include "qemu/osdep.h" -#include "hw/vfio/vfio-common.h" #include "hw/vfio/vfio-cpr.h" #include "hw/vfio/pci.h" #include "hw/pci/msix.h" @@ -30,19 +29,6 @@ int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, return 0; } -bool vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **errp) -{ - migration_add_notifier_mode(&bcontainer->cpr_reboot_notifier, - vfio_cpr_reboot_notifier, - MIG_MODE_CPR_REBOOT); - return true; -} - -void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer) -{ - migration_remove_notifier(&bcontainer->cpr_reboot_notifier); -} - #define STRDUP_VECTOR_FD_NAME(vdev, name) \ g_strdup_printf("%s_%s", (vdev)->vbasedev.name, (name)) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 00831b7..d8c6510 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -262,9 +262,6 @@ int vfio_kvm_device_add_fd(int fd, Error **errp); int vfio_kvm_device_del_fd(int fd, Error **errp); void vfio_kvm_device_close(void); -bool vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **errp); -void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer); - void iommufd_cdev_rebuild_hwpt(VFIODevice *vbasedev, VFIOIOMMUFDContainer *container); bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container,