From patchwork Tue Feb 18 02:15:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Longfang Liu X-Patchwork-Id: 13978839 Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CCA61E51D; Tue, 18 Feb 2025 02:16:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=45.249.212.191 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739844995; cv=none; b=lutn0JewvHg5bLdGDpGcg4m4aw0lJlnv2mCxDKXmfusqXnjFHDFwmDZFZFkGPBDxTp1VtD7E6rqp0yQFdCT7q3YRcBpKU89RSgpcB7zFKnDYuEZ3Ya4785O7f0CBPpX5cWiVBFP5iehqafA/mPANTpB47Ss4SI+hTwGEutHeaso= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739844995; c=relaxed/simple; bh=4pyof3JJU+48W9PMkgXpSeUzFR7S8Ks3YmrdQHCQvYs=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=NVU57UO0V6yF9RUlq99mH1f17S3edJit9JX8+Y/vAMyVk4W4612f47PwF6eoWzPMlEDXuSwA87JowdF42XGTj6CCVWEQ6qN0syuumvhoaq67CKbLnUhtSVKpkzHP7L5edKHKd9tnBPHGsIO38strGmJtbrQEBTILI/NFenDZARU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; arc=none smtp.client-ip=45.249.212.191 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4Yxjlk4nG7zMqc9; Tue, 18 Feb 2025 10:14:58 +0800 (CST) Received: from kwepemg500006.china.huawei.com (unknown [7.202.181.43]) by mail.maildlp.com (Postfix) with ESMTPS id D1FCC1A016C; Tue, 18 Feb 2025 10:16:27 +0800 (CST) Received: from huawei.com (10.50.165.33) by kwepemg500006.china.huawei.com (7.202.181.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Tue, 18 Feb 2025 10:16:27 +0800 From: Longfang Liu To: , , , CC: , , , Subject: [PATCH 3/3] migration: adapt to new migration configuration Date: Tue, 18 Feb 2025 10:15:07 +0800 Message-ID: <20250218021507.40740-4-liulongfang@huawei.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20250218021507.40740-1-liulongfang@huawei.com> References: <20250218021507.40740-1-liulongfang@huawei.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemg500006.china.huawei.com (7.202.181.43) On the new hardware platform, the migration region is moved from VF to PF. the driver also needs to be modified accordingly to adapt to the new hardware device. Use PF's io base directly on the new hardware platform. and no mmap operation is required. If it is on an old platform, the driver needs to be compatible with the old solution. Signed-off-by: Longfang Liu --- .../vfio/pci/hisilicon/hisi_acc_vfio_pci.c | 165 ++++++++++++------ .../vfio/pci/hisilicon/hisi_acc_vfio_pci.h | 7 + 2 files changed, 119 insertions(+), 53 deletions(-) diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c index 599905dbb707..cf5a807c2199 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c @@ -125,6 +125,72 @@ static int qm_get_cqc(struct hisi_qm *qm, u64 *addr) return 0; } +static int qm_get_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, + struct acc_vf_data *vf_data) +{ + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; + struct device *dev = &qm->pdev->dev; + u32 eqc_addr, aeqc_addr; + int ret; + + if (qm->ver == QM_HW_V3) { + eqc_addr = QM_EQC_DW0; + aeqc_addr = QM_AEQC_DW0; + } else { + eqc_addr = QM_EQC_PF_DW0; + aeqc_addr = QM_AEQC_PF_DW0; + } + + /* QM_EQC_DW has 7 regs */ + ret = qm_read_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); + if (ret) { + dev_err(dev, "failed to read QM_EQC_DW\n"); + return ret; + } + + /* QM_AEQC_DW has 7 regs */ + ret = qm_read_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); + if (ret) { + dev_err(dev, "failed to read QM_AEQC_DW\n"); + return ret; + } + + return 0; +} + +static int qm_set_xqc_regs(struct hisi_acc_vf_core_device *hisi_acc_vdev, + struct acc_vf_data *vf_data) +{ + struct hisi_qm *qm = &hisi_acc_vdev->vf_qm; + struct device *dev = &qm->pdev->dev; + u32 eqc_addr, aeqc_addr; + int ret; + + if (qm->ver == QM_HW_V3) { + eqc_addr = QM_EQC_DW0; + aeqc_addr = QM_AEQC_DW0; + } else { + eqc_addr = QM_EQC_PF_DW0; + aeqc_addr = QM_AEQC_PF_DW0; + } + + /* QM_EQC_DW has 7 regs */ + ret = qm_write_regs(qm, eqc_addr, vf_data->qm_eqc_dw, 7); + if (ret) { + dev_err(dev, "failed to write QM_EQC_DW\n"); + return ret; + } + + /* QM_AEQC_DW has 7 regs */ + ret = qm_write_regs(qm, aeqc_addr, vf_data->qm_aeqc_dw, 7); + if (ret) { + dev_err(dev, "failed to write QM_AEQC_DW\n"); + return ret; + } + + return 0; +} + static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) { struct device *dev = &qm->pdev->dev; @@ -167,20 +233,6 @@ static int qm_get_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) return ret; } - /* QM_EQC_DW has 7 regs */ - ret = qm_read_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); - if (ret) { - dev_err(dev, "failed to read QM_EQC_DW\n"); - return ret; - } - - /* QM_AEQC_DW has 7 regs */ - ret = qm_read_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7); - if (ret) { - dev_err(dev, "failed to read QM_AEQC_DW\n"); - return ret; - } - return 0; } @@ -238,20 +290,6 @@ static int qm_set_regs(struct hisi_qm *qm, struct acc_vf_data *vf_data) return ret; } - /* QM_EQC_DW has 7 regs */ - ret = qm_write_regs(qm, QM_EQC_DW0, vf_data->qm_eqc_dw, 7); - if (ret) { - dev_err(dev, "failed to write QM_EQC_DW\n"); - return ret; - } - - /* QM_AEQC_DW has 7 regs */ - ret = qm_write_regs(qm, QM_AEQC_DW0, vf_data->qm_aeqc_dw, 7); - if (ret) { - dev_err(dev, "failed to write QM_AEQC_DW\n"); - return ret; - } - return 0; } @@ -470,6 +508,10 @@ static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev, return ret; } + ret = qm_set_xqc_regs(hisi_acc_vdev, vf_data); + if (ret) + return ret; + ret = hisi_qm_mb(qm, QM_MB_CMD_SQC_BT, qm->sqc_dma, 0, 0); if (ret) { dev_err(dev, "set sqc failed\n"); @@ -544,6 +586,10 @@ static int vf_qm_state_save(struct hisi_acc_vf_core_device *hisi_acc_vdev, return ret; } + ret = qm_get_xqc_regs(hisi_acc_vdev, vf_data); + if (ret) + return -EINVAL; + ret = vf_qm_read_data(vf_qm, vf_data); if (ret) return -EINVAL; @@ -1131,34 +1177,46 @@ static int hisi_acc_vf_qm_init(struct hisi_acc_vf_core_device *hisi_acc_vdev) { struct vfio_pci_core_device *vdev = &hisi_acc_vdev->core_device; struct hisi_qm *vf_qm = &hisi_acc_vdev->vf_qm; + struct hisi_qm *pf_qm = hisi_acc_vdev->pf_qm; struct pci_dev *vf_dev = vdev->pdev; - /* - * ACC VF dev BAR2 region consists of both functional register space - * and migration control register space. For migration to work, we - * need access to both. Hence, we map the entire BAR2 region here. - * But unnecessarily exposing the migration BAR region to the Guest - * has the potential to prevent/corrupt the Guest migration. Hence, - * we restrict access to the migration control space from - * Guest(Please see mmap/ioctl/read/write override functions). - * - * Please note that it is OK to expose the entire VF BAR if migration - * is not supported or required as this cannot affect the ACC PF - * configurations. - * - * Also the HiSilicon ACC VF devices supported by this driver on - * HiSilicon hardware platforms are integrated end point devices - * and the platform lacks the capability to perform any PCIe P2P - * between these devices. - */ + if (pf_qm->ver == QM_HW_V3) { + /* + * ACC VF dev BAR2 region consists of both functional register space + * and migration control register space. For migration to work, we + * need access to both. Hence, we map the entire BAR2 region here. + * But unnecessarily exposing the migration BAR region to the Guest + * has the potential to prevent/corrupt the Guest migration. Hence, + * we restrict access to the migration control space from + * Guest(Please see mmap/ioctl/read/write override functions). + * + * Please note that it is OK to expose the entire VF BAR if migration + * is not supported or required as this cannot affect the ACC PF + * configurations. + * + * Also the HiSilicon ACC VF devices supported by this driver on + * HiSilicon hardware platforms are integrated end point devices + * and the platform lacks the capability to perform any PCIe P2P + * between these devices. + */ - vf_qm->io_base = - ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX), - pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX)); - if (!vf_qm->io_base) - return -EIO; + vf_qm->io_base = + ioremap(pci_resource_start(vf_dev, VFIO_PCI_BAR2_REGION_INDEX), + pci_resource_len(vf_dev, VFIO_PCI_BAR2_REGION_INDEX)); + if (!vf_qm->io_base) + return -EIO; - vf_qm->fun_type = QM_HW_VF; + vf_qm->fun_type = QM_HW_VF; + vf_qm->ver = pf_qm->ver; + } else { + /* + * In the new HW platform, the migration function register space is in BAR2 of PF, + * and each VF occupies 8KB address space. + */ + vf_qm->io_base = pf_qm->io_base + QM_MIG_REGION_OFFSET + + hisi_acc_vdev->vf_id * QM_MIG_REGION_SIZE; + vf_qm->fun_type = QM_HW_PF; + } vf_qm->pdev = vf_dev; mutex_init(&vf_qm->mailbox_lock); @@ -1488,7 +1546,8 @@ static void hisi_acc_vfio_pci_close_device(struct vfio_device *core_vdev) mutex_lock(&hisi_acc_vdev->open_mutex); hisi_acc_vdev->dev_opened = false; - iounmap(vf_qm->io_base); + if (vf_qm->ver == QM_HW_V3) + iounmap(vf_qm->io_base); mutex_unlock(&hisi_acc_vdev->open_mutex); vfio_pci_core_close_device(core_vdev); } diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h index 245d7537b2bc..b01eb54525d3 100644 --- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h +++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.h @@ -50,6 +50,13 @@ #define QM_EQC_DW0 0X8000 #define QM_AEQC_DW0 0X8020 +#define QM_MIG_REGION_OFFSET 0x180000 +#define QM_MIG_REGION_SIZE 0x2000 + +#define QM_SUB_VERSION_ID 0x100210 +#define QM_EQC_PF_DW0 0x1c00 +#define QM_AEQC_PF_DW0 0x1c20 + struct acc_vf_data { #define QM_MATCH_SIZE offsetofend(struct acc_vf_data, qm_rsv_state) /* QM match information */