From patchwork Mon Mar 14 19:49:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 890FDC43217 for ; Mon, 14 Mar 2022 19:50:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244625AbiCNTvr (ORCPT ); Mon, 14 Mar 2022 15:51:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244615AbiCNTvp (ORCPT ); Mon, 14 Mar 2022 15:51:45 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C72A9EB0 for ; Mon, 14 Mar 2022 12:50:11 -0700 (PDT) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVeL002679; Mon, 14 Mar 2022 19:49:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=FzASfKyj0eR9/Ww/nhIftsHsBp/NIss97mXU5M92LDw=; b=joVBjaO0OhgoMzYmHx7klwyPHkVYLYQRJT031PoSnyZm2KKE2/sM0Q79msojBOD4GgTK AwUJC3L/1lF5ZRokM3ZNgEXDiDexad1huaAbyv7ombF6KrJfluL80xEYndRstcnmjyl4 Os8eY5327bFltQKslyyK8c9DwaXmInCvL4m0RztoTMYeXemlAcEjWe/1SGyzNB7gcnix MhVyaspjG4rdlNq14s828v7N7VgJ8rTFuEf+Rni74HW1JM1C+ceJqaIz2lDH5SYhD0/T Xt/nBp4ociZFt5HF7TlLIfZWR+VIVS+YASb8wuJVhX59MINXPFi8c51puOlsKEzowK8c NQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6ah929x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:35 +0000 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJmVr7006314; Mon, 14 Mar 2022 19:49:35 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6ah929s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:35 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlApl009552; Mon, 14 Mar 2022 19:49:34 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma01dal.us.ibm.com with ESMTP id 3erk59ccam-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:34 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJnWCs32964884 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:49:32 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 23134112067; Mon, 14 Mar 2022 19:49:32 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6234A11206E; Mon, 14 Mar 2022 19:49:28 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:49:28 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 01/11] Update linux headers Date: Mon, 14 Mar 2022 15:49:10 -0400 Message-Id: <20220314194920.58888-2-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: nzbBv5zspxk00AnLZbUGpXsMx5DpNqm_ X-Proofpoint-ORIG-GUID: 0AqaTJHA0Eksz6x5jQChjXPBz_OKvOwI X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 suspectscore=0 bulkscore=0 mlxlogscore=992 phishscore=0 spamscore=0 malwarescore=0 impostorscore=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is a placeholder that pulls in 5.17-rc7 + unmerged kernel changes required by this item. A proper header sync can be done once the associated kernel code merges. Signed-off-by: Matthew Rosato --- linux-headers/asm-s390/kvm.h | 1 + linux-headers/asm-x86/kvm.h | 3 ++ linux-headers/linux/kvm.h | 51 +++++++++++++++++++++++++++++++-- linux-headers/linux/vfio.h | 6 ++++ linux-headers/linux/vfio_zdev.h | 6 ++++ 5 files changed, 64 insertions(+), 3 deletions(-) diff --git a/linux-headers/asm-s390/kvm.h b/linux-headers/asm-s390/kvm.h index f053b8304a..d8259ff9a1 100644 --- a/linux-headers/asm-s390/kvm.h +++ b/linux-headers/asm-s390/kvm.h @@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine { #define KVM_S390_VM_CPU_FEAT_PFMFI 11 #define KVM_S390_VM_CPU_FEAT_SIGPIF 12 #define KVM_S390_VM_CPU_FEAT_KSS 13 +#define KVM_S390_VM_CPU_FEAT_ZPCI_INTERP 14 struct kvm_s390_vm_cpu_feat { __u64 feat[16]; }; diff --git a/linux-headers/asm-x86/kvm.h b/linux-headers/asm-x86/kvm.h index 2da3316bb5..bf6e96011d 100644 --- a/linux-headers/asm-x86/kvm.h +++ b/linux-headers/asm-x86/kvm.h @@ -452,6 +452,9 @@ struct kvm_sync_regs { #define KVM_STATE_VMX_PREEMPTION_TIMER_DEADLINE 0x00000001 +/* attributes for system fd (group 0) */ +#define KVM_X86_XCOMP_GUEST_SUPP 0 + struct kvm_vmx_nested_state_data { __u8 vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE]; __u8 shadow_vmcs12[KVM_STATE_NESTED_VMX_VMCS_SIZE]; diff --git a/linux-headers/linux/kvm.h b/linux-headers/linux/kvm.h index 00af3bc333..8f82e6ff20 100644 --- a/linux-headers/linux/kvm.h +++ b/linux-headers/linux/kvm.h @@ -1133,6 +1133,9 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206 #define KVM_CAP_VM_GPA_BITS 207 #define KVM_CAP_XSAVE2 208 +#define KVM_CAP_SYS_ATTRIBUTES 209 +#define KVM_CAP_PPC_AIL_MODE_3 210 +#define KVM_CAP_S390_ZPCI_OP 211 #ifdef KVM_CAP_IRQ_ROUTING @@ -1623,9 +1626,6 @@ struct kvm_enc_region { #define KVM_S390_NORMAL_RESET _IO(KVMIO, 0xc3) #define KVM_S390_CLEAR_RESET _IO(KVMIO, 0xc4) -/* Available with KVM_CAP_XSAVE2 */ -#define KVM_GET_XSAVE2 _IOR(KVMIO, 0xcf, struct kvm_xsave) - struct kvm_s390_pv_sec_parm { __u64 origin; __u64 length; @@ -2047,4 +2047,49 @@ struct kvm_stats_desc { #define KVM_GET_STATS_FD _IO(KVMIO, 0xce) +/* Available with KVM_CAP_XSAVE2 */ +#define KVM_GET_XSAVE2 _IOR(KVMIO, 0xcf, struct kvm_xsave) + +/* Available with KVM_CAP_S390_ZPCI_OP */ +#define KVM_S390_ZPCI_OP _IOW(KVMIO, 0xd0, struct kvm_s390_zpci_op) + +struct kvm_s390_zpci_op { + /* in */ + __u32 fh; /* target device */ + __u8 op; /* operation to perform */ + __u8 pad[3]; + union { + /* for KVM_S390_ZPCIOP_REG_INT */ + struct { + __u64 ibv; /* Guest addr of interrupt bit vector */ + __u64 sb; /* Guest addr of summary bit */ + __u32 flags; + __u32 noi; /* Number of interrupts */ + __u8 isc; /* Guest interrupt subclass */ + __u8 sbo; /* Offset of guest summary bit vector */ + __u16 pad; + } reg_int; + /* for KVM_S390_ZPCIOP_REG_IOAT */ + struct { + __u64 iota; /* I/O Translation settings */ + } reg_ioat; + __u8 reserved[64]; + } u; + /* out */ + __u32 newfh; /* updated device handle */ +}; + +/* types for kvm_s390_zpci_op->op */ +#define KVM_S390_ZPCIOP_INIT 0 +#define KVM_S390_ZPCIOP_END 1 +#define KVM_S390_ZPCIOP_START_INTERP 2 +#define KVM_S390_ZPCIOP_STOP_INTERP 3 +#define KVM_S390_ZPCIOP_REG_INT 4 +#define KVM_S390_ZPCIOP_DEREG_INT 5 +#define KVM_S390_ZPCIOP_REG_IOAT 6 +#define KVM_S390_ZPCIOP_DEREG_IOAT 7 + +/* flags for kvm_s390_zpci_op->u.reg_int.flags */ +#define KVM_S390_ZPCIOP_REGINT_HOST (1 << 0) + #endif /* __LINUX_KVM_H */ diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index e680594f27..38d43e1205 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -52,6 +52,12 @@ /* Supports the vaddr flag for DMA map and unmap */ #define VFIO_UPDATE_VADDR 10 +/* + * The KVM_IOMMU type implies that the hypervisor will control the mappings + * rather than userspace + */ +#define VFIO_KVM_IOMMU 11 + /* * The IOCTL interface is designed for extensibility by embedding the * structure length (argsz) and flags into structures passed between diff --git a/linux-headers/linux/vfio_zdev.h b/linux-headers/linux/vfio_zdev.h index b4309397b6..29351687e9 100644 --- a/linux-headers/linux/vfio_zdev.h +++ b/linux-headers/linux/vfio_zdev.h @@ -29,6 +29,9 @@ struct vfio_device_info_cap_zpci_base { __u16 fmb_length; /* Measurement Block Length (in bytes) */ __u8 pft; /* PCI Function Type */ __u8 gid; /* PCI function group ID */ + /* End of version 1 */ + __u32 fh; /* PCI function handle */ + /* End of version 2 */ }; /** @@ -47,6 +50,9 @@ struct vfio_device_info_cap_zpci_group { __u16 noi; /* Maximum number of MSIs */ __u16 maxstbl; /* Maximum Store Block Length */ __u8 version; /* Supported PCI Version */ + /* End of version 1 */ + __u8 dtsm; /* Supported IOAT Designations */ + /* End of version 2 */ }; /** From patchwork Mon Mar 14 19:49:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24ADBC433EF for ; Mon, 14 Mar 2022 19:50:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244612AbiCNTvu (ORCPT ); Mon, 14 Mar 2022 15:51:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244594AbiCNTvr (ORCPT ); Mon, 14 Mar 2022 15:51:47 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4625A3ED17 for ; Mon, 14 Mar 2022 12:50:20 -0700 (PDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlZEC024364; Mon, 14 Mar 2022 19:49:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=a9WjLZ7JkjFEqlZdkUhc6FVsxBFAWxOYRQ8qrBaiaPU=; b=pPnk190qlviYVya89kRbnV+Sgl4eurLRft5N023eGrRW1LevUnCtNqfqHZBPkwwnfAhc bMZEHpiofaa7Eo8hIB9GKFxftMWrKpRYEO3lTmoTC286DqRoeaCm7xl048EtMkzBdFE8 lOyjnCkJuESoOPPUASvTAkERwv61psBGypDsS62Y+t5LWmgjhgEqkp+0QJ7v71hXx7dZ H0ooousd+/j0riMiBW+cb+iY10lLJ35llkIX5uiAo0nj+pR8NHCP/jPy+61FehQuJGQE ONb90ETEymqlJ9aBdQijmGQJGbbVHYd1FGlaqhi1hgvqaGBz1AXAKC+RunYSu+CNgHuS zQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6ab91ns-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:41 +0000 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJmj3Y030942; Mon, 14 Mar 2022 19:49:41 GMT Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6ab91nf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:40 +0000 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJl6CF005336; Mon, 14 Mar 2022 19:49:39 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma01wdc.us.ibm.com with ESMTP id 3erk5989gj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:39 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJnc7o13631814 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:49:38 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EC11F112071; Mon, 14 Mar 2022 19:49:37 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ACFA311206D; Mon, 14 Mar 2022 19:49:32 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:49:32 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 02/11] vfio: handle KVM-owned IOMMU requests Date: Mon, 14 Mar 2022 15:49:11 -0400 Message-Id: <20220314194920.58888-3-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ARIVp70G1NptRZdBDwvv-2ycrmDWUY4M X-Proofpoint-ORIG-GUID: EkWs5pdD0m8fPFsYOqS_lgphhENrkBTX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 mlxscore=0 clxscore=1015 phishscore=0 impostorscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org s390x PCI devices need to use a special KVM-managed IOMMU domain as part of zPCI interpretation. To facilitate this, let a vfio device indicate that it wishes to use a KVM-managed IOMMU so that it can be reflected by the group and, ultimately, trigger a KVM-managed argument for the VFIO_SET_IOMMU ioctl. This patch sets up the framework to allow a device to trigger the VFIO_SET_IOMMU with the new KVM-owend type. A subsequent patch will add exploitation by s390x PCI. Signed-off-by: Matthew Rosato --- hw/vfio/ap.c | 2 +- hw/vfio/ccw.c | 2 +- hw/vfio/common.c | 26 +++++++++++++++++++++----- hw/vfio/pci.c | 3 ++- hw/vfio/pci.h | 1 + hw/vfio/platform.c | 2 +- include/hw/vfio/vfio-common.h | 4 +++- 7 files changed, 30 insertions(+), 10 deletions(-) diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index e0dd561e85..22c402771a 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -81,7 +81,7 @@ static VFIOGroup *vfio_ap_get_group(VFIOAPDevice *vapdev, Error **errp) g_free(group_path); - return vfio_get_group(groupid, &address_space_memory, errp); + return vfio_get_group(groupid, &address_space_memory, false, errp); } static void vfio_ap_realize(DeviceState *dev, Error **errp) diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 0354737666..08b0af5897 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -650,7 +650,7 @@ static VFIOGroup *vfio_ccw_get_group(S390CCWDevice *cdev, Error **errp) return NULL; } - return vfio_get_group(groupid, &address_space_memory, errp); + return vfio_get_group(groupid, &address_space_memory, false, errp); } static void vfio_ccw_realize(DeviceState *dev, Error **errp) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 080046e3f5..227880bf84 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1873,7 +1873,7 @@ static int vfio_get_iommu_type(VFIOContainer *container, return -EINVAL; } -static int vfio_init_container(VFIOContainer *container, int group_fd, +static int vfio_init_container(VFIOContainer *container, VFIOGroup *group, Error **errp) { int iommu_type, ret; @@ -1883,12 +1883,20 @@ static int vfio_init_container(VFIOContainer *container, int group_fd, return iommu_type; } - ret = ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); + ret = ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd); if (ret) { error_setg_errno(errp, errno, "Failed to set group container"); return -errno; } + /* + * In the case where KVM will manage the IOMMU, we must instruct the host + * IOMMU to use the appropriate domain ops + */ + if (group->kvm_managed_iommu) { + iommu_type = VFIO_KVM_IOMMU; + } + while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { if (iommu_type == VFIO_SPAPR_TCE_v2_IOMMU) { /* @@ -2062,7 +2070,7 @@ static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, QLIST_INIT(&container->hostwin_list); QLIST_INIT(&container->vrdl_list); - ret = vfio_init_container(container, group->fd, errp); + ret = vfio_init_container(container, group, errp); if (ret) { goto free_container_exit; } @@ -2265,7 +2273,8 @@ static void vfio_disconnect_container(VFIOGroup *group) } } -VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) +VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, bool kvm_managed_iommu, + Error **errp) { VFIOGroup *group; char path[32]; @@ -2273,7 +2282,13 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) QLIST_FOREACH(group, &vfio_group_list, next) { if (group->groupid == groupid) { - /* Found it. Now is it already in the right context? */ + /* Found it. Ensure using same IOMMU type */ + if (group->kvm_managed_iommu != kvm_managed_iommu) { + error_setg(errp, "group %d using conflicting iommu ops", + group->groupid); + return NULL; + } + /* Is it already in the right context? */ if (group->container->space->as == as) { return group; } else { @@ -2307,6 +2322,7 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) } group->groupid = groupid; + group->kvm_managed_iommu = kvm_managed_iommu; QLIST_INIT(&group->device_list); if (vfio_connect_container(group, as, errp)) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7b45353ce2..80f7e2880a 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2855,7 +2855,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) trace_vfio_realize(vdev->vbasedev.name, groupid); - group = vfio_get_group(groupid, pci_device_iommu_address_space(pdev), errp); + group = vfio_get_group(groupid, pci_device_iommu_address_space(pdev), + vdev->kvm_managed_iommu, errp); if (!group) { goto error; } diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 64777516d1..f74524384c 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -171,6 +171,7 @@ struct VFIOPCIDevice { bool no_kvm_ioeventfd; bool no_vfio_ioeventfd; bool enable_ramfb; + bool kvm_managed_iommu; VFIODisplay *dpy; Notifier irqchip_change_notifier; }; diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index f8f08a0f36..08793401dd 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -577,7 +577,7 @@ static int vfio_base_device_init(VFIODevice *vbasedev, Error **errp) trace_vfio_platform_base_device_init(vbasedev->name, groupid); - group = vfio_get_group(groupid, &address_space_memory, errp); + group = vfio_get_group(groupid, &address_space_memory, false, errp); if (!group) { return -ENOENT; } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8af11b0a76..37aa6ca162 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -162,6 +162,7 @@ typedef struct VFIOGroup { QLIST_ENTRY(VFIOGroup) next; QLIST_ENTRY(VFIOGroup) container_next; bool ram_block_discard_allowed; + bool kvm_managed_iommu; } VFIOGroup; typedef struct VFIODMABuf { @@ -208,7 +209,8 @@ void vfio_region_unmap(VFIORegion *region); void vfio_region_exit(VFIORegion *region); void vfio_region_finalize(VFIORegion *region); void vfio_reset_handler(void *opaque); -VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); +VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, bool kvm_managed_iommu, + Error **errp); void vfio_put_group(VFIOGroup *group); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); From patchwork Mon Mar 14 19:49:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18FA7C433F5 for ; Mon, 14 Mar 2022 19:50:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244483AbiCNTvz (ORCPT ); Mon, 14 Mar 2022 15:51:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244644AbiCNTvt (ORCPT ); Mon, 14 Mar 2022 15:51:49 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFBDF3ED1C for ; Mon, 14 Mar 2022 12:50:22 -0700 (PDT) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVq2006488; Mon, 14 Mar 2022 19:49:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=iPN8RCxQTjFAr0jkqNej31njqirlstNIh+iKWnvKFBA=; b=iENyZDSlUuztSf1E14z2JHEW2umfIBonou6c8ws2q5oCo6LoDjBKjPZl9wb9mKwoegNU APb0EE5kRWM5cDJpbiqEw8qmZheyVJCkQB7FLU12vtsMa8k11NctXkt4CAFiWZjeF4I+ HrMtcdtB/FxVrh7YivOvpKA9XJ7e4fU+dLNDGYAgMUlRMugAqyJi1V7f75X7B7wL7QyP KwZLaoyKnTpG8aT1fSUsedZYW2Kt4X51b0eI5JtilqzHIulzPpxAyNMGfO2eJGNAjY8v 3ixres3Vg0WYYI3ld22JuNlH4VOQDf4/PyD5nL1y+kMl+5N3SH7sq3/WdrLDt55DzoiF 0g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6d6rjuv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:46 +0000 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJnj1p016181; Mon, 14 Mar 2022 19:49:45 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6d6rjuq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:45 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlD7h010251; Mon, 14 Mar 2022 19:49:44 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma05wdc.us.ibm.com with ESMTP id 3erk59rany-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:44 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJnhvi7012720 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:49:43 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8A19E11206D; Mon, 14 Mar 2022 19:49:43 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8BA8D11206F; Mon, 14 Mar 2022 19:49:38 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:49:38 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 03/11] target/s390x: add zpci-interp to cpu models Date: Mon, 14 Mar 2022 15:49:12 -0400 Message-Id: <20220314194920.58888-4-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Dw422ubruaSGqvAy1RUTh2zUKYHPsJ0d X-Proofpoint-ORIG-GUID: PBk6Hw3gJrQTdWiw0ZvqQ80oL58oEB8x X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 impostorscore=0 clxscore=1015 suspectscore=0 spamscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The zpci-interp feature is used to specify whether zPCI interpretation is to be used for this guest. Signed-off-by: Matthew Rosato --- hw/s390x/s390-virtio-ccw.c | 1 + target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 ++ target/s390x/kvm/kvm.c | 1 + 4 files changed, 5 insertions(+) diff --git a/hw/s390x/s390-virtio-ccw.c b/hw/s390x/s390-virtio-ccw.c index 90480e7cf9..b190234308 100644 --- a/hw/s390x/s390-virtio-ccw.c +++ b/hw/s390x/s390-virtio-ccw.c @@ -805,6 +805,7 @@ static void ccw_machine_6_2_instance_options(MachineState *machine) static const S390FeatInit qemu_cpu_feat = { S390_FEAT_LIST_QEMU_V6_2 }; ccw_machine_7_0_instance_options(machine); + s390_cpudef_featoff_greater(14, 1, S390_FEAT_ZPCI_INTERP); s390_set_qemu_cpu_model(0x3906, 14, 2, qemu_cpu_feat); } diff --git a/target/s390x/cpu_features_def.h.inc b/target/s390x/cpu_features_def.h.inc index e86662bb3b..4ade3182aa 100644 --- a/target/s390x/cpu_features_def.h.inc +++ b/target/s390x/cpu_features_def.h.inc @@ -146,6 +146,7 @@ DEF_FEAT(SIE_CEI, "cei", SCLP_CPU, 43, "SIE: Conditional-external-interception f DEF_FEAT(DAT_ENH_2, "dateh2", MISC, 0, "DAT-enhancement facility 2") DEF_FEAT(CMM, "cmm", MISC, 0, "Collaborative-memory-management facility") DEF_FEAT(AP, "ap", MISC, 0, "AP instructions installed") +DEF_FEAT(ZPCI_INTERP, "zpci-interp", MISC, 0, "zPCI interpretation") /* Features exposed via the PLO instruction. */ DEF_FEAT(PLO_CL, "plo-cl", PLO, 0, "PLO Compare and load (32 bit in general registers)") diff --git a/target/s390x/gen-features.c b/target/s390x/gen-features.c index 22846121c4..9db6bd545e 100644 --- a/target/s390x/gen-features.c +++ b/target/s390x/gen-features.c @@ -554,6 +554,7 @@ static uint16_t full_GEN14_GA1[] = { S390_FEAT_HPMA2, S390_FEAT_SIE_KSS, S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF, + S390_FEAT_ZPCI_INTERP, }; #define full_GEN14_GA2 EmptyFeat @@ -650,6 +651,7 @@ static uint16_t default_GEN14_GA1[] = { S390_FEAT_GROUP_MSA_EXT_8, S390_FEAT_MULTIPLE_EPOCH, S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF, + S390_FEAT_ZPCI_INTERP, }; #define default_GEN14_GA2 EmptyFeat diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c index 6acf14d5ec..0357bfda89 100644 --- a/target/s390x/kvm/kvm.c +++ b/target/s390x/kvm/kvm.c @@ -2294,6 +2294,7 @@ static int kvm_to_feat[][2] = { { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI}, { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF}, { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS}, + { KVM_S390_VM_CPU_FEAT_ZPCI_INTERP, S390_FEAT_ZPCI_INTERP }, }; static int query_cpu_feat(S390FeatBitmap features) From patchwork Mon Mar 14 19:49:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3DC0C433EF for ; Mon, 14 Mar 2022 19:50:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244626AbiCNTwE (ORCPT ); Mon, 14 Mar 2022 15:52:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244629AbiCNTv5 (ORCPT ); Mon, 14 Mar 2022 15:51:57 -0400 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C695B3F88F for ; Mon, 14 Mar 2022 12:50:29 -0700 (PDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVio008396; Mon, 14 Mar 2022 19:49:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=2QhRnd+yb5RzMPnv2Mjxmht6IHoKFSJpiHswRupauyk=; b=N/tak9QmFe8XIIaAv3sJwmvynp/RA3XrG25+8dVaQOl9TJsTzoSncy1IyegRUzOVUoHk X+9v/3RUwl4cQCgzqPTuiV2QonpQ+6wAWxPXXJjGooSOMPJ2DtJzlr6dGV/RtnGDOYo6 Y8K6XIxwuPQC7L2Xfu8ZfU0CLtgBf+Bpbx3fZk1Hzhm7db8zlA633ANFiclgNxBalqae 6rNNomPYZuqzQd0OASN1QM0N5w/VIjRSbcjHpbTRWKVvJl78XVFUAJInZaYafbTplvra IuupxMDoQw+JWzj5tWBCZwmfGDb2z2AD8dJhBYMpQpR3UN4mOz/By4vlcNNcwwp+uVqA zA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6cs0qhd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:52 +0000 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJloeU009950; Mon, 14 Mar 2022 19:49:51 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6cs0qh5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:51 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlBhC028329; Mon, 14 Mar 2022 19:49:50 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma04dal.us.ibm.com with ESMTP id 3erk59ccb6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:50 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJnn1234996570 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:49:49 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 07BCC11206F; Mon, 14 Mar 2022 19:49:49 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F3C0B11206E; Mon, 14 Mar 2022 19:49:43 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:49:43 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 04/11] s390x/pci: add routine to get host function handle from CLP info Date: Mon, 14 Mar 2022 15:49:13 -0400 Message-Id: <20220314194920.58888-5-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: C2_KmmaOcN-5O_P5VWQVUoGOgSdjcICK X-Proofpoint-ORIG-GUID: -LXRH9RuA5oNej7bqnQKC8flsaVDAKNg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 priorityscore=1501 mlxscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 phishscore=0 spamscore=0 adultscore=0 impostorscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to interface with the underlying host zPCI device, we need to know it's function handle. Add a routine to grab this from the vfio CLP capabilities chain. Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-vfio.c | 83 ++++++++++++++++++++++++++------ include/hw/s390x/s390-pci-vfio.h | 6 +++ 2 files changed, 73 insertions(+), 16 deletions(-) diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index 6f80a47e29..4bf0a7e22d 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -124,6 +124,27 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, pbdev->zpci_fn.pft = 0; } +static bool get_host_fh(S390PCIBusDevice *pbdev, struct vfio_device_info *info, + uint32_t *fh) +{ + struct vfio_info_cap_header *hdr; + struct vfio_device_info_cap_zpci_base *cap; + VFIOPCIDevice *vpci = container_of(pbdev->pdev, VFIOPCIDevice, pdev); + + hdr = vfio_get_device_info_cap(info, VFIO_DEVICE_INFO_CAP_ZPCI_BASE); + + /* Can only get the host fh with version 2 or greater */ + if (hdr == NULL || hdr->version < 2) { + trace_s390_pci_clp_cap(vpci->vbasedev.name, + VFIO_DEVICE_INFO_CAP_ZPCI_BASE); + return false; + } + cap = (void *) hdr; + + *fh = cap->fh; + return true; +} + static void s390_pci_read_group(S390PCIBusDevice *pbdev, struct vfio_device_info *info) { @@ -217,25 +238,13 @@ static void s390_pci_read_pfip(S390PCIBusDevice *pbdev, memcpy(pbdev->zpci_fn.pfip, cap->pfip, CLP_PFIP_NR_SEGMENTS); } -/* - * This function will issue the VFIO_DEVICE_GET_INFO ioctl and look for - * capabilities that contain information about CLP features provided by the - * underlying host. - * On entry, defaults have already been placed into the guest CLP response - * buffers. On exit, defaults will have been overwritten for any CLP features - * found in the capability chain; defaults will remain for any CLP features not - * found in the chain. - */ -void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) +static struct vfio_device_info *get_device_info(S390PCIBusDevice *pbdev, + uint32_t argsz) { - g_autofree struct vfio_device_info *info = NULL; + struct vfio_device_info *info = g_malloc0(argsz); VFIOPCIDevice *vfio_pci; - uint32_t argsz; int fd; - argsz = sizeof(*info); - info = g_malloc0(argsz); - vfio_pci = container_of(pbdev->pdev, VFIOPCIDevice, pdev); fd = vfio_pci->vbasedev.fd; @@ -250,7 +259,8 @@ retry: if (ioctl(fd, VFIO_DEVICE_GET_INFO, info)) { trace_s390_pci_clp_dev_info(vfio_pci->vbasedev.name); - return; + free(info); + return NULL; } if (info->argsz > argsz) { @@ -259,6 +269,47 @@ retry: goto retry; } + return info; +} + +/* + * Get the host function handle from the vfio CLP capabilities chain. Returns + * true if a fh value was placed into the provided buffer. Returns false + * if a fh could not be obtained (ioctl failed or capabilitiy version does + * not include the fh) + */ +bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh) +{ + g_autofree struct vfio_device_info *info = NULL; + + assert(fh); + + info = get_device_info(pbdev, sizeof(*info)); + if (!info) { + return false; + } + + return get_host_fh(pbdev, info, fh); +} + +/* + * This function will issue the VFIO_DEVICE_GET_INFO ioctl and look for + * capabilities that contain information about CLP features provided by the + * underlying host. + * On entry, defaults have already been placed into the guest CLP response + * buffers. On exit, defaults will have been overwritten for any CLP features + * found in the capability chain; defaults will remain for any CLP features not + * found in the chain. + */ +void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) +{ + g_autofree struct vfio_device_info *info = NULL; + + info = get_device_info(pbdev, sizeof(*info)); + if (!info) { + return; + } + /* * Find the CLP features provided and fill in the guest CLP responses. * Always call s390_pci_read_base first as information from this could diff --git a/include/hw/s390x/s390-pci-vfio.h b/include/hw/s390x/s390-pci-vfio.h index ff708aef50..0c2e4b5175 100644 --- a/include/hw/s390x/s390-pci-vfio.h +++ b/include/hw/s390x/s390-pci-vfio.h @@ -20,6 +20,7 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail); S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, S390PCIBusDevice *pbdev); void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); +bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); #else static inline bool s390_pci_update_dma_avail(int fd, unsigned int *avail) @@ -33,6 +34,11 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, } static inline void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt) { } +static inline bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, + unsigned int *fh) +{ + return false; +} static inline void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) { } #endif From patchwork Mon Mar 14 19:49:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3D7BC433F5 for ; Mon, 14 Mar 2022 19:51:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244702AbiCNTwT (ORCPT ); Mon, 14 Mar 2022 15:52:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244565AbiCNTwQ (ORCPT ); Mon, 14 Mar 2022 15:52:16 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31CC326FD for ; Mon, 14 Mar 2022 12:50:40 -0700 (PDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlXQ3005110; Mon, 14 Mar 2022 19:49:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=YbmNfifklOCLP3pb4sII09nI2ZUduYgn2qgGTlE0BOk=; b=WrC96QxIZxwGKLeWaKPboNp0HmVc8N3vf+kM7mXDd+T2s88kibJOD/TyZeVOmIwxKYZ5 UY8z/8sW514qICvfMtxdsZYB8PYoadTycFmE/F/t0W2W6/mfFSDerBZC0QcGdtc8XVfz OR3X3j2DtvLZeS/kKxnrATqdm6oPjwsFLkaeUX6dE6Tr+ZrxsFOq6RTl3ykUzK8UfauS vRSKHLvK1rSdUB7oq4PGyloJFHw5sqaGJC65L2LlPK6kBAgeaQsvY0AR2tEz0FxyUU7E wRATDcsV9RNlcTqDkOPXbGcAlILxIRUt5naol/BoXqmaTp+Qf+xRKKeuERqM+d/B0+eu qg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6af0wqe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:58 +0000 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJms1S012077; Mon, 14 Mar 2022 19:49:57 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6af0wq4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:57 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJl7dP004519; Mon, 14 Mar 2022 19:49:56 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma04wdc.us.ibm.com with ESMTP id 3erk59g7wt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:49:56 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJnsGu48562546 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:49:54 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 82A6F112071; Mon, 14 Mar 2022 19:49:54 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 76984112062; Mon, 14 Mar 2022 19:49:49 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:49:49 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 05/11] s390x/pci: enable for load/store intepretation Date: Mon, 14 Mar 2022 15:49:14 -0400 Message-Id: <20220314194920.58888-6-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: bm26c9dmadsCSsp3xngyQ0QTQn0e15Ew X-Proofpoint-ORIG-GUID: J1HdvGHYpYmTmnKikICy-KJBO9kXTFQv X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 phishscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 mlxscore=0 impostorscore=0 priorityscore=1501 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use the associated kvm ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- we use an initial value from vfio CLP and maintain an updated fh value from kvm ioctl response info. By default, unless interpret=off is specified, interpretation support will always be assumed and exploited if the necessary ioctl and features are available on the host kernel. When these are unavailable, we will silently revert to the interception model; this allows existing guest configurations to work unmodified on hosts with and without zPCI interpretation support, allowing QEMU to choose the best support model available. Signed-off-by: Matthew Rosato --- hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 67 ++++++++++++++++++- hw/s390x/s390-pci-inst.c | 54 ++++++++++++++- hw/s390x/s390-pci-kvm.c | 112 ++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 46 +++++++++++++ target/s390x/kvm/kvm.c | 7 ++ target/s390x/kvm/kvm_s390x.h | 1 + 8 files changed, 287 insertions(+), 2 deletions(-) create mode 100644 hw/s390x/s390-pci-kvm.c create mode 100644 include/hw/s390x/s390-pci-kvm.h diff --git a/hw/s390x/meson.build b/hw/s390x/meson.build index 28484256ec..6e6e47fcda 100644 --- a/hw/s390x/meson.build +++ b/hw/s390x/meson.build @@ -23,6 +23,7 @@ s390x_ss.add(when: 'CONFIG_KVM', if_true: files( 's390-skeys-kvm.c', 's390-stattrib-kvm.c', 'pv.c', + 's390-pci-kvm.c', )) s390x_ss.add(when: 'CONFIG_TCG', if_true: files( 'tod-tcg.c', diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 4b2bdd94b3..7ce7bda26d 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -16,6 +16,7 @@ #include "qapi/visitor.h" #include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-kvm.h" #include "hw/s390x/s390-pci-vfio.h" #include "hw/pci/pci_bus.h" #include "hw/qdev-properties.h" @@ -971,12 +972,45 @@ static void s390_pci_update_subordinate(PCIDevice *dev, uint32_t nr) } } +static int s390_pci_interp_plug(S390pciState *s, S390PCIBusDevice *pbdev) +{ + uint32_t idx; + int rc; + + rc = s390_pci_kvm_plug(pbdev); + if (rc) { + return rc; + } + + /* Next, see if the idx is already in-use */ + idx = pbdev->fh & FH_MASK_INDEX; + if (pbdev->idx != idx) { + if (s390_pci_find_dev_by_idx(s, idx)) { + return -EINVAL; + } + /* + * Update the idx entry with the passed through idx + * If the relinquished idx is lower than next_idx, use it + * to replace next_idx + */ + g_hash_table_remove(s->zpci_table, &pbdev->idx); + if (idx < s->next_idx) { + s->next_idx = idx; + } + pbdev->idx = idx; + g_hash_table_insert(s->zpci_table, &pbdev->idx, pbdev); + } + + return 0; +} + static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { S390pciState *s = S390_PCI_HOST_BRIDGE(hotplug_dev); PCIDevice *pdev = NULL; S390PCIBusDevice *pbdev = NULL; + int rc; if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) { PCIBridge *pb = PCI_BRIDGE(dev); @@ -1022,12 +1056,35 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, set_pbdev_info(pbdev); if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) { - pbdev->fh |= FH_SHM_VFIO; + /* + * By default, interpretation is always requested; if the available + * facilities indicate it is not available, fallback to the + * interception model. + */ + if (pbdev->interp) { + if (s390_pci_kvm_zpciop_allowed()) { + rc = s390_pci_interp_plug(s, pbdev); + if (rc) { + error_setg(errp, "Plug failed for zPCI device in " + "interpretation mode: %d", rc); + return; + } + } else { + DPRINTF("zPCI interpretation facilities missing.\n"); + pbdev->interp = false; + } + } pbdev->iommu->dma_limit = s390_pci_start_dma_count(s, pbdev); /* Fill in CLP information passed via the vfio region */ s390_pci_get_clp_info(pbdev); + if (!pbdev->interp) { + /* Do vfio passthrough but intercept for I/O */ + pbdev->fh |= FH_SHM_VFIO; + } } else { pbdev->fh |= FH_SHM_EMUL; + /* Always intercept emulated devices */ + pbdev->interp = false; } if (s390_pci_msix_init(pbdev)) { @@ -1078,6 +1135,8 @@ static void s390_pcihost_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->pdev = NULL; pbdev->state = ZPCI_FS_RESERVED; } else if (object_dynamic_cast(OBJECT(dev), TYPE_S390_PCI_DEVICE)) { + int rc; + pbdev = S390_PCI_DEVICE(dev); pbdev->fid = 0; QTAILQ_REMOVE(&s->zpci_devs, pbdev, link); @@ -1085,6 +1144,11 @@ static void s390_pcihost_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, if (pbdev->iommu->dma_limit) { s390_pci_end_dma_count(s, pbdev->iommu->dma_limit); } + rc = s390_pci_kvm_unplug(pbdev); + if (rc) { + error_setg(errp, "Unplug failed for zPCI device in interpretation " + "mode rc=%d", rc); + } qdev_unrealize(dev); } } @@ -1360,6 +1424,7 @@ static Property s390_pci_device_properties[] = { DEFINE_PROP_UINT16("uid", S390PCIBusDevice, uid, UID_UNDEFINED), DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid), DEFINE_PROP_STRING("target", S390PCIBusDevice, target), + DEFINE_PROP_BOOL("interpret", S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 6d400d4147..92ea7b73e4 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -18,6 +18,8 @@ #include "sysemu/hw_accel.h" #include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST @@ -156,6 +158,37 @@ out: return rc; } +static int clp_enable_interp(S390PCIBusDevice *pbdev) +{ + int rc; + + rc = s390_pci_kvm_interp_enable(pbdev); + if (rc) { + DPRINTF("Failed to enable interpretation\n"); + return rc; + } + + if (!(pbdev->fh & FH_MASK_ENABLE)) { + DPRINTF("Passthrough handle is not enabled\n"); + return -EINVAL; + } + + return 0; +} + +static int clp_disable_interp(S390PCIBusDevice *pbdev) +{ + int rc; + + rc = s390_pci_kvm_interp_disable(pbdev); + if (rc) { + DPRINTF("Failed to disable interpretation\n"); + return rc; + } + + return 0; +} + int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) { ClpReqHdr *reqh; @@ -246,7 +279,19 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } - pbdev->fh |= FH_MASK_ENABLE; + /* + * If interpretation is specified, attempt to enable this now and + * update with the host fh + */ + if (pbdev->interp) { + if (clp_enable_interp(pbdev)) { + stw_p(&ressetpci->hdr.rsp, CLP_RC_SETPCIFN_ERR); + goto out; + } + } else { + pbdev->fh |= FH_MASK_ENABLE; + } + pbdev->state = ZPCI_FS_ENABLED; stl_p(&ressetpci->fh, pbdev->fh); stw_p(&ressetpci->hdr.rsp, CLP_RC_OK); @@ -257,6 +302,13 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } device_legacy_reset(DEVICE(pbdev)); + if (pbdev->interp) { + if (clp_disable_interp(pbdev)) { + stw_p(&ressetpci->hdr.rsp, CLP_RC_SETPCIFN_ERR); + goto out; + } + } + /* Mask off the enabled bit for interpreted devices too */ pbdev->fh &= ~FH_MASK_ENABLE; pbdev->state = ZPCI_FS_DISABLED; stl_p(&ressetpci->fh, pbdev->fh); diff --git a/hw/s390x/s390-pci-kvm.c b/hw/s390x/s390-pci-kvm.c new file mode 100644 index 0000000000..755ea0618a --- /dev/null +++ b/hw/s390x/s390-pci-kvm.c @@ -0,0 +1,112 @@ +/* + * s390 zPCI KVM interfaces + * + * Copyright 2022 IBM Corp. + * Author(s): Matthew Rosato + * + * This work is licensed under the terms of the GNU GPL, version 2 or (at + * your option) any later version. See the COPYING file in the top-level + * directory. + */ + +#include "qemu/osdep.h" + +#include + +#include "kvm/kvm_s390x.h" +#include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-vfio.h" + +bool s390_pci_kvm_zpciop_allowed(void) +{ + return s390_has_feat(S390_FEAT_ZPCI_INTERP) && kvm_s390_get_zpci_op(); +} + +int s390_pci_kvm_plug(S390PCIBusDevice *pbdev) +{ + int rc; + + struct kvm_s390_zpci_op args = { + .op = KVM_S390_ZPCIOP_INIT + }; + + if (!s390_pci_get_host_fh(pbdev, &args.fh)) { + return -EINVAL; + } + + rc = kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (!rc) { + /* + * The host device is already in an enabled state, but we always present + * the initial device state to the guest as disabled (ZPCI_FS_DISABLED). + * Therefore, mask off the enable bit from the passthrough handle until + * the guest issues a CLP SET PCI FN later to enable the device. + */ + pbdev->fh = (args.newfh & ~FH_MASK_ENABLE); + } + + return rc; +} + +int s390_pci_kvm_unplug(S390PCIBusDevice *pbdev) +{ + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh | FH_MASK_ENABLE, + .op = KVM_S390_ZPCIOP_END + }; + + return kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); +} + +int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev) +{ + uint32_t fh; + int rc; + + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh | FH_MASK_ENABLE, + .op = KVM_S390_ZPCIOP_START_INTERP + }; + + retry: + rc = kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + + if (rc == -ENODEV) { + /* + * If the function wasn't found, re-sync the function handle with vfio + * and if a change is detected, retry the operation with the new fh. + * This can happen while the device is disabled to the guest due to + * vfio-triggered events (e.g. vfio hot reset for ISM during plug) + */ + if (!s390_pci_get_host_fh(pbdev, &fh)) { + return -EINVAL; + } + if (fh != args.fh) { + args.fh = fh; + goto retry; + } + } + if (!rc) { + pbdev->fh = args.newfh; + } + + return rc; +} + +int s390_pci_kvm_interp_disable(S390PCIBusDevice *pbdev) +{ + int rc; + + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op = KVM_S390_ZPCIOP_STOP_INTERP + }; + + rc = kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); + if (!rc) { + pbdev->fh = args.newfh; + } + + return rc; +} diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h index da3cde2bb4..a9843dfe97 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -350,6 +350,7 @@ struct S390PCIBusDevice { IndAddr *indicator; bool pci_unplug_request_processed; bool unplug_requested; + bool interp; QTAILQ_ENTRY(S390PCIBusDevice) link; }; diff --git a/include/hw/s390x/s390-pci-kvm.h b/include/hw/s390x/s390-pci-kvm.h new file mode 100644 index 0000000000..6b2528cf82 --- /dev/null +++ b/include/hw/s390x/s390-pci-kvm.h @@ -0,0 +1,46 @@ +/* + * s390 PCI KVM interfaces + * + * Copyright 2022 IBM Corp. + * Author(s): Matthew Rosato + * + * This work is licensed under the terms of the GNU GPL, version 2 or (at + * your option) any later version. See the COPYING file in the top-level + * directory. + */ + +#ifndef HW_S390_PCI_KVM_H +#define HW_S390_PCI_KVM_H + +#include "hw/s390x/s390-pci-bus.h" + +#ifdef CONFIG_KVM +bool s390_pci_kvm_zpciop_allowed(void); +int s390_pci_kvm_plug(S390PCIBusDevice *pbdev); +int s390_pci_kvm_unplug(S390PCIBusDevice *pbdev); +int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev); +int s390_pci_kvm_interp_disable(S390PCIBusDevice *pbdev); +#else +static inline bool s390_pci_kvm_zpciop_allowed(void) +{ + return false; +} +static inline int s390_pci_kvm_plug(S390PCIBusDevice *pbdev) +{ + return -EINVAL; +} +static inline int s390_pci_kvm_unplug(S390PCIBusDevice *pbdev) +{ + return -EINVAL; +} +static inline int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev) +{ + return -EINVAL; +} +static inline int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev) +{ + return -EINVAL; +} +#endif + +#endif diff --git a/target/s390x/kvm/kvm.c b/target/s390x/kvm/kvm.c index 0357bfda89..288fbd1d75 100644 --- a/target/s390x/kvm/kvm.c +++ b/target/s390x/kvm/kvm.c @@ -157,6 +157,7 @@ static int cap_ri; static int cap_hpage_1m; static int cap_vcpu_resets; static int cap_protected; +static int cap_zpci_op; static int active_cmma; @@ -358,6 +359,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_s390_irq = kvm_check_extension(s, KVM_CAP_S390_INJECT_IRQ); cap_vcpu_resets = kvm_check_extension(s, KVM_CAP_S390_VCPU_RESETS); cap_protected = kvm_check_extension(s, KVM_CAP_S390_PROTECTED); + cap_zpci_op = kvm_check_extension(s, KVM_CAP_S390_ZPCI_OP); kvm_vm_enable_cap(s, KVM_CAP_S390_USER_SIGP, 0); kvm_vm_enable_cap(s, KVM_CAP_S390_VECTOR_REGISTERS, 0); @@ -2567,3 +2569,8 @@ bool kvm_arch_cpu_check_are_resettable(void) { return true; } + +int kvm_s390_get_zpci_op(void) +{ + return cap_zpci_op; +} diff --git a/target/s390x/kvm/kvm_s390x.h b/target/s390x/kvm/kvm_s390x.h index 05a5e1e6f4..aaae8570de 100644 --- a/target/s390x/kvm/kvm_s390x.h +++ b/target/s390x/kvm/kvm_s390x.h @@ -27,6 +27,7 @@ void kvm_s390_vcpu_interrupt_pre_save(S390CPU *cpu); int kvm_s390_vcpu_interrupt_post_load(S390CPU *cpu); int kvm_s390_get_hpage_1m(void); int kvm_s390_get_ri(void); +int kvm_s390_get_zpci_op(void); int kvm_s390_get_clock(uint8_t *tod_high, uint64_t *tod_clock); int kvm_s390_get_clock_ext(uint8_t *tod_high, uint64_t *tod_clock); int kvm_s390_set_clock(uint8_t tod_high, uint64_t tod_clock); From patchwork Mon Mar 14 19:49:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF66C433FE for ; Mon, 14 Mar 2022 19:51:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244751AbiCNTwe (ORCPT ); Mon, 14 Mar 2022 15:52:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244678AbiCNTwQ (ORCPT ); Mon, 14 Mar 2022 15:52:16 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BFC73EABA for ; Mon, 14 Mar 2022 12:50:43 -0700 (PDT) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVm4006438; Mon, 14 Mar 2022 19:50:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=2A3XmtF8yBiCyYbhFlRyNB3L4makRDlvj61mHEDhGuM=; b=Dqxvf19to0inb+tkUcZ46kEvQsQYKtTpeX/VOGp6lKvDQXcW2W/a11ir376dHbWVh8Kw JpcWgnsFKE6EIbUsQR0pTjYGZxW1/EDhD8j5qOWpR6TxDGlAk/oad2wDe9bijopsY7rT HbiEm6HLg8i/ZSTTG2ZI0D9z0MC+dHx0eHvfe+HpXf7VdH/Moe1SeSUeB9X0QnXChMrH +kauU283DMS7/1uq6ajwKwrX8pKWhBBMnl7zBfTiDdfffDvO0AyfIuEA3F8386wl/Ycz twQJWLBhxrUAjlkZHPj5CWhGKMoegBAK33MCchCwfbs2HMI1MpDO31/ZaIxCVyLec+cw cg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6d6rjyq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:03 +0000 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJlpfX007669; Mon, 14 Mar 2022 19:50:02 GMT Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6d6rjyg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:02 +0000 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJl8Bm005348; Mon, 14 Mar 2022 19:50:02 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma01wdc.us.ibm.com with ESMTP id 3erk5989j6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:01 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJo0lP16974268 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:50:00 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70963112062; Mon, 14 Mar 2022 19:50:00 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E8543112071; Mon, 14 Mar 2022 19:49:54 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:49:54 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 06/11] s390x/pci: don't fence interpreted devices without MSI-X Date: Mon, 14 Mar 2022 15:49:15 -0400 Message-Id: <20220314194920.58888-7-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Pyd90NFSpvzFBZZSF6gW_LESCgrUMJSe X-Proofpoint-ORIG-GUID: UHz3beWOtkHxJg1U9iUOzPDvCLeOIRZC X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 impostorscore=0 clxscore=1015 suspectscore=0 spamscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Lack of MSI-X support is not an issue for interpreted passthrough devices, so let's let these in. This will allow, for example, ISM devices to be passed through -- but only when interpretation is available and being used. Reviewed-by: Thomas Huth Reviewed-by: Pierre Morel Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-bus.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 7ce7bda26d..acc91af64c 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -1087,7 +1087,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->interp = false; } - if (s390_pci_msix_init(pbdev)) { + if (s390_pci_msix_init(pbdev) && !pbdev->interp) { error_setg(errp, "MSI-X support is mandatory " "in the S390 architecture"); return; From patchwork Mon Mar 14 19:49:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A65DC433EF for ; Mon, 14 Mar 2022 19:51:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244699AbiCNTwc (ORCPT ); Mon, 14 Mar 2022 15:52:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244690AbiCNTwR (ORCPT ); Mon, 14 Mar 2022 15:52:17 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2B303ED0D for ; Mon, 14 Mar 2022 12:50:46 -0700 (PDT) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVuE009185; Mon, 14 Mar 2022 19:50:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=0Vfelxcx5BY4MlQ6QOML3av4XFGN5L+cn8E61h1TWiA=; b=hCAbJMAcXbXwHCG38lXAQJCIGuZUwEkpK8nq13ksllfLaPw/c8ihdcBfigkdSWMSxNRP coH+ES2kwZjJB5WKO/uNR+NRLmJuUHOSn2BjE2YmNudMvvmwr2Bf8XI6bZ2qIdnPUkDT MdjTbjJ0kWaBJA62kIUnPPRzNqf6sf0SsQ0tuHRnK936fNdc/e4Qn9+8NABFXXlxulh9 NtTjgIiUS7BqulG3pMCg8B6KT571p1GD0iUG51iiR6/J+FaAvOKTVVf0m7ZeDMz8m3po SV4LOG4xjK7XxzBcL5HOLXS/RJuYxUE1dRQdXvltaPnK3Ngshp8eubpn+h1YatnPWN6a Gw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6mer3vm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:10 +0000 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJngGD017826; Mon, 14 Mar 2022 19:50:09 GMT Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6mer3v6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:09 +0000 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJm5XL002536; Mon, 14 Mar 2022 19:50:08 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma03wdc.us.ibm.com with ESMTP id 3erk58r9bk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:08 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJo6Vx19071364 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:50:06 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AEF6F112065; Mon, 14 Mar 2022 19:50:06 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EDC3F112067; Mon, 14 Mar 2022 19:50:00 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:50:00 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 07/11] s390x/pci: enable adapter event notification for interpreted devices Date: Mon, 14 Mar 2022 15:49:16 -0400 Message-Id: <20220314194920.58888-8-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: C7HMTVEBmGd6nQ6KciOj82Ow9JJUvHl5 X-Proofpoint-ORIG-GUID: vl4XuRqee3pIEutEK4d8mmaGWi0xrAt9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 impostorscore=0 adultscore=0 lowpriorityscore=0 malwarescore=0 clxscore=1015 mlxlogscore=999 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use the associated kvm ioctl operation to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting. Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 27 ++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 12 ++++++++++ 5 files changed, 95 insertions(+), 5 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index acc91af64c..5043b8c85c 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break; default: - if (pbdev->summary_ind) { + if (pbdev->interp && (pbdev->fh & FH_MASK_ENABLE)) { + /* Interpreted devices were using interrupt forwarding */ + s390_pci_kvm_aif_disable(pbdev); + } else if (pbdev->summary_ind) { pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) { @@ -1072,6 +1075,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } else { DPRINTF("zPCI interpretation facilities missing.\n"); pbdev->interp = false; + pbdev->forwarding_assist = false; } } pbdev->iommu->dma_limit = s390_pci_start_dma_count(s, pbdev); @@ -1080,11 +1084,13 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (!pbdev->interp) { /* Do vfio passthrough but intercept for I/O */ pbdev->fh |= FH_SHM_VFIO; + pbdev->forwarding_assist = false; } } else { pbdev->fh |= FH_SHM_EMUL; /* Always intercept emulated devices */ pbdev->interp = false; + pbdev->forwarding_assist = false; } if (s390_pci_msix_init(pbdev) && !pbdev->interp) { @@ -1241,7 +1247,10 @@ static void s390_pcihost_reset(DeviceState *dev) /* Process all pending unplug requests */ QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { if (pbdev->unplug_requested) { - if (pbdev->summary_ind) { + if (pbdev->interp && (pbdev->fh & FH_MASK_ENABLE)) { + /* Interpreted devices were using interrupt forwarding */ + s390_pci_kvm_aif_disable(pbdev); + } else if (pbdev->summary_ind) { pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) { @@ -1379,7 +1388,10 @@ static void s390_pci_device_reset(DeviceState *dev) break; } - if (pbdev->summary_ind) { + if (pbdev->interp && (pbdev->fh & FH_MASK_ENABLE)) { + /* Interpreted devices were using interrupt forwarding */ + s390_pci_kvm_aif_disable(pbdev); + } else if (pbdev->summary_ind) { pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) { @@ -1425,6 +1437,8 @@ static Property s390_pci_device_properties[] = { DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid), DEFINE_PROP_STRING("target", S390PCIBusDevice, target), DEFINE_PROP_BOOL("interpret", S390PCIBusDevice, interp, true), + DEFINE_PROP_BOOL("forwarding_assist", S390PCIBusDevice, forwarding_assist, + true), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 92ea7b73e4..f7b01e2059 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -1102,6 +1102,32 @@ static void fmb_update(void *opaque) timer_mod(pbdev->fmb_timer, t + pbdev->pci_group->zpci_group.mui); } +static int mpcifc_reg_int_interp(S390PCIBusDevice *pbdev, ZpciFib *fib) +{ + int rc; + + rc = s390_pci_kvm_aif_enable(pbdev, fib, pbdev->forwarding_assist); + if (rc) { + DPRINTF("Failed to enable interrupt forwarding\n"); + return rc; + } + + return 0; +} + +static int mpcifc_dereg_int_interp(S390PCIBusDevice *pbdev, ZpciFib *fib) +{ + int rc; + + rc = s390_pci_kvm_aif_disable(pbdev); + if (rc) { + DPRINTF("Failed to disable interrupt forwarding\n"); + return rc; + } + + return 0; +} + int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, uintptr_t ra) { @@ -1156,7 +1182,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, switch (oc) { case ZPCI_MOD_FC_REG_INT: - if (pbdev->summary_ind) { + if (pbdev->interp) { + if (mpcifc_reg_int_interp(pbdev, &fib)) { + cc = ZPCI_PCI_LS_ERR; + s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); + } + } else if (pbdev->summary_ind) { cc = ZPCI_PCI_LS_ERR; s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); } else if (reg_irqs(env, pbdev, fib)) { @@ -1165,7 +1196,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, } break; case ZPCI_MOD_FC_DEREG_INT: - if (!pbdev->summary_ind) { + if (pbdev->interp) { + if (mpcifc_dereg_int_interp(pbdev, &fib)) { + cc = ZPCI_PCI_LS_ERR; + s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); + } + } else if (!pbdev->summary_ind) { cc = ZPCI_PCI_LS_ERR; s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); } else { diff --git a/hw/s390x/s390-pci-kvm.c b/hw/s390x/s390-pci-kvm.c index 755ea0618a..ebb862abd0 100644 --- a/hw/s390x/s390-pci-kvm.c +++ b/hw/s390x/s390-pci-kvm.c @@ -16,6 +16,7 @@ #include "kvm/kvm_s390x.h" #include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-vfio.h" bool s390_pci_kvm_zpciop_allowed(void) @@ -110,3 +111,29 @@ int s390_pci_kvm_interp_disable(S390PCIBusDevice *pbdev) return rc; } + +int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist) +{ + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op = KVM_S390_ZPCIOP_REG_INT, + .u.reg_int.ibv = fib->aibv, + .u.reg_int.sb = fib->aisb, + .u.reg_int.noi = FIB_DATA_NOI(fib->data), + .u.reg_int.isc = FIB_DATA_ISC(fib->data), + .u.reg_int.sbo = FIB_DATA_AISBO(fib->data), + .u.reg_int.flags = (assist) ? 0 : KVM_S390_ZPCIOP_REGINT_HOST + }; + + return kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); +} + +int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) +{ + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op = KVM_S390_ZPCIOP_DEREG_INT + }; + + return kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); +} diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h index a9843dfe97..5b09f0cf2f 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -351,6 +351,7 @@ struct S390PCIBusDevice { bool pci_unplug_request_processed; bool unplug_requested; bool interp; + bool forwarding_assist; QTAILQ_ENTRY(S390PCIBusDevice) link; }; diff --git a/include/hw/s390x/s390-pci-kvm.h b/include/hw/s390x/s390-pci-kvm.h index 6b2528cf82..09004d3f6c 100644 --- a/include/hw/s390x/s390-pci-kvm.h +++ b/include/hw/s390x/s390-pci-kvm.h @@ -13,6 +13,7 @@ #define HW_S390_PCI_KVM_H #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-inst.h" #ifdef CONFIG_KVM bool s390_pci_kvm_zpciop_allowed(void); @@ -20,6 +21,8 @@ int s390_pci_kvm_plug(S390PCIBusDevice *pbdev); int s390_pci_kvm_unplug(S390PCIBusDevice *pbdev); int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev); int s390_pci_kvm_interp_disable(S390PCIBusDevice *pbdev); +int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist); +int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev); #else static inline bool s390_pci_kvm_zpciop_allowed(void) { @@ -41,6 +44,15 @@ static inline int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev) { return -EINVAL; } +static inline int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, + bool assist) +{ + return -EINVAL; +} +static inline int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) +{ + return -EINVAL; +} #endif #endif From patchwork Mon Mar 14 19:49:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00465C433F5 for ; Mon, 14 Mar 2022 19:51:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244718AbiCNTwX (ORCPT ); Mon, 14 Mar 2022 15:52:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244644AbiCNTwS (ORCPT ); Mon, 14 Mar 2022 15:52:18 -0400 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 147ED3EF07 for ; Mon, 14 Mar 2022 12:50:49 -0700 (PDT) Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVvM035035; Mon, 14 Mar 2022 19:50:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=aXvx6PyzZldBHixFGOuJEyHVe3aRM30bqORsujKWtaE=; b=MtKzs5cQOMx/96y9bMPqwWVZ0cZQc/K8Prj+TCpibFWEZW9HB7Yg1ej6po509MMusLUh re6jTkyXte0RSXCwAauyXwL/bT3H9VJwDo0Wb31zTgMA+tkp+ZSGO5W1XRRcaEBK/YVF yJSgW+antnj38s5VEKi80TkKzQD9eO46jGVBya/qrHiJJ78t8j9vpK1AkIqC4+SE3U+q lnSDE3uRhNAmXGjQ0XaHS48iEzebLuSAtiOL3fDGdTcO2yyopr6CZ8cEiep1oR9FjvMa R4hHM0rp+vqmvi7oLAVh1mwWM9Z3fVh77LA9id7H4s6o+nAEfTxZ09Yw9z3pKHaes5Jx Bw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6d9gqd3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:16 +0000 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJlql3035819; Mon, 14 Mar 2022 19:50:15 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6d9gqct-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:15 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlCxF009569; Mon, 14 Mar 2022 19:50:14 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma01dal.us.ibm.com with ESMTP id 3erk59ccf6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:14 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJoCr811272674 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:50:12 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 89CBC11206B; Mon, 14 Mar 2022 19:50:12 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 23032112069; Mon, 14 Mar 2022 19:50:07 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:50:06 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 08/11] s390x/pci: use KVM-managed IOMMU for interpretation Date: Mon, 14 Mar 2022 15:49:17 -0400 Message-Id: <20220314194920.58888-9-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: qqQOzCgN7a76QUnleGULRcEGONjE3Gxn X-Proofpoint-ORIG-GUID: PooLvps3kk9YHOM0rwBrdSlf8YLAq4Ui X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 suspectscore=0 phishscore=0 impostorscore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 clxscore=1015 mlxlogscore=999 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When interpreting zPCI instructions, KVM will control the IOMMU mappings in response to RPCIT instructions rather than relying on mapping ioctls from userspace. Mark the vfio device in pre_plug so that the appropriate iommu domain will be allocated on the host during VFIO_SET_IOMMU. Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-bus.c | 11 +++++++++++ hw/s390x/s390-pci-vfio.c | 22 ++++++++++++++++++++++ include/hw/s390x/s390-pci-vfio.h | 5 +++++ 3 files changed, 38 insertions(+) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 5043b8c85c..513a276711 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -950,6 +950,17 @@ static void s390_pcihost_pre_plug(HotplugHandler *hotplug_dev, DeviceState *dev, error_setg(errp, "multifunction not supported in s390"); return; } + /* + * If we have a vfio-pci device that wishes to use interpretation + * we must update the host IOMMU domain ops. + */ + if (s390_pci_kvm_zpciop_allowed() && + object_dynamic_cast(OBJECT(dev), "vfio-pci")) { + if (s390_pci_set_kvm_iommu(s, dev)) { + error_setg(errp, "KVM IOMMU not available for interpretation"); + return; + } + } } else if (object_dynamic_cast(OBJECT(dev), TYPE_S390_PCI_DEVICE)) { S390PCIBusDevice *pbdev = S390_PCI_DEVICE(dev); diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index 4bf0a7e22d..7808e8d939 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -324,3 +324,25 @@ void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) return; } + +/* + * This function will determine if the specified VFIOPCIDevice is linked to a + * zPCI device that requests interpretation support. In this case, we must + * inform vfio that the KVM-managed IOMMU should be requested when the + * VFIO_SET_IOMMU ioctl is issued. + */ +int s390_pci_set_kvm_iommu(S390pciState *s, DeviceState *dev) +{ + VFIOPCIDevice *vdev = VFIO_PCI(dev); + S390PCIBusDevice *pbdev = s390_pci_find_dev_by_target(s, dev->id); + + if (!pbdev) { + return -ENODEV; + } + + if (pbdev->interp) { + vdev->kvm_managed_iommu = true; + } + + return 0; +} diff --git a/include/hw/s390x/s390-pci-vfio.h b/include/hw/s390x/s390-pci-vfio.h index 0c2e4b5175..5026f978c2 100644 --- a/include/hw/s390x/s390-pci-vfio.h +++ b/include/hw/s390x/s390-pci-vfio.h @@ -22,6 +22,7 @@ S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt); bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, uint32_t *fh); void s390_pci_get_clp_info(S390PCIBusDevice *pbdev); +int s390_pci_set_kvm_iommu(S390pciState *s, DeviceState *dev); #else static inline bool s390_pci_update_dma_avail(int fd, unsigned int *avail) { @@ -40,6 +41,10 @@ static inline bool s390_pci_get_host_fh(S390PCIBusDevice *pbdev, return false; } static inline void s390_pci_get_clp_info(S390PCIBusDevice *pbdev) { } +static inline int s390_pci_set_kvm_iommu(S390pciState *s, DeviceState *dev) +{ + return -EINVAL; +} #endif #endif From patchwork Mon Mar 14 19:49:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC43DC4332F for ; Mon, 14 Mar 2022 19:52:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242627AbiCNTxp (ORCPT ); Mon, 14 Mar 2022 15:53:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244727AbiCNTwY (ORCPT ); Mon, 14 Mar 2022 15:52:24 -0400 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA40ACD0 for ; Mon, 14 Mar 2022 12:50:56 -0700 (PDT) Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVUD035046; Mon, 14 Mar 2022 19:50:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=MXSsAsZuVnM/s6D6tfMaszmULKzq0LzIfX0tqQlVki4=; b=GMLWyWjHJ36+VdAJHGtsDNXo8PvtITHKQm7NVG2+7YlcefuBJbr7KHl3ICPBgQBcAEBL Ut14k9fR8mCmYR8AKpIVqUpMG+EtXW8p/RAwqm2N8tb/Fs+ViUXTnKEUU584PI/K5imj WUVK1neGxDQytr6rtmYJfjy7gA+Wr4AvZ/kVXOhN7/Nw7zjC51XSoPCJWy2dIQMdIuLo J1AyVWW/c8qKTs3sMdtarQVEQLVwCyCqcuXIdbnSdIQidCFD6gmM6URnjA1waHocei/0 Q5Z3PobXP2apcdEGy5A9oqP1JwAiVtZxDUArCgpincZtBQkC77+bNO0vNyslkYhi+Zj+ 9Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6d9gqep-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:21 +0000 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJoK3D004883; Mon, 14 Mar 2022 19:50:20 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 3et6d9gqec-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:20 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJl8BA004527; Mon, 14 Mar 2022 19:50:19 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma04wdc.us.ibm.com with ESMTP id 3erk59g81j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:19 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJoIeZ11338090 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:50:18 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E7AB4112062; Mon, 14 Mar 2022 19:50:17 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1653F112069; Mon, 14 Mar 2022 19:50:13 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:50:12 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 09/11] s390x/pci: use I/O Address Translation assist when interpreting Date: Mon, 14 Mar 2022 15:49:18 -0400 Message-Id: <20220314194920.58888-10-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: NbFzwao48man5RQf4j-kVcGkoa3sDd4g X-Proofpoint-ORIG-GUID: edorsVmVLiMcbCy9gktR_3R4-slSyxuU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 suspectscore=0 phishscore=0 impostorscore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 clxscore=1015 mlxlogscore=924 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Allow the underlying kvm host to handle the Refresh PCI Translation instruction intercepts. Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-bus.c | 6 ++--- hw/s390x/s390-pci-inst.c | 42 +++++++++++++++++++++++++++++--- hw/s390x/s390-pci-kvm.c | 21 ++++++++++++++++ include/hw/s390x/s390-pci-inst.h | 2 +- include/hw/s390x/s390-pci-kvm.h | 10 ++++++++ 5 files changed, 74 insertions(+), 7 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 513a276711..32894fbe9c 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -197,7 +197,7 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) { - pci_dereg_ioat(pbdev->iommu); + pci_dereg_ioat(pbdev); } pbdev->state = ZPCI_FS_STANDBY; rc = SCLP_RC_NORMAL_COMPLETION; @@ -1265,7 +1265,7 @@ static void s390_pcihost_reset(DeviceState *dev) pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) { - pci_dereg_ioat(pbdev->iommu); + pci_dereg_ioat(pbdev); } pbdev->state = ZPCI_FS_STANDBY; s390_pci_perform_unplug(pbdev); @@ -1406,7 +1406,7 @@ static void s390_pci_device_reset(DeviceState *dev) pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) { - pci_dereg_ioat(pbdev->iommu); + pci_dereg_ioat(pbdev); } fmb_timer_free(pbdev); diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index f7b01e2059..86bb04e859 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -969,6 +969,19 @@ int pci_dereg_irqs(S390PCIBusDevice *pbdev) return 0; } +static int reg_ioat_interp(S390PCIBusDevice *pbdev, uint64_t iota) +{ + int rc; + + rc = s390_pci_kvm_ioat_enable(pbdev, iota); + if (rc) { + return rc; + } + + pbdev->iommu->enabled = true; + return 0; +} + static int reg_ioat(CPUS390XState *env, S390PCIBusDevice *pbdev, ZpciFib fib, uintptr_t ra) { @@ -986,6 +999,16 @@ static int reg_ioat(CPUS390XState *env, S390PCIBusDevice *pbdev, ZpciFib fib, return -EINVAL; } + /* If this is an interpreted device, we must use the IOAT assist */ + if (pbdev->interp) { + if (reg_ioat_interp(pbdev, g_iota)) { + error_report("failure starting ioat assist"); + s390_program_interrupt(env, PGM_OPERAND, ra); + return -EINVAL; + } + return 0; + } + /* currently we only support designation type 1 with translation */ if (!(dt == ZPCI_IOTA_RTTO && t)) { error_report("unsupported ioat dt %d t %d", dt, t); @@ -1002,8 +1025,21 @@ static int reg_ioat(CPUS390XState *env, S390PCIBusDevice *pbdev, ZpciFib fib, return 0; } -void pci_dereg_ioat(S390PCIIOMMU *iommu) +static void dereg_ioat_interp(S390PCIBusDevice *pbdev) +{ + s390_pci_kvm_ioat_disable(pbdev); + pbdev->iommu->enabled = false; +} + +void pci_dereg_ioat(S390PCIBusDevice *pbdev) { + S390PCIIOMMU *iommu = pbdev->iommu; + + if (pbdev->interp) { + dereg_ioat_interp(pbdev); + return; + } + s390_pci_iommu_disable(iommu); iommu->pba = 0; iommu->pal = 0; @@ -1228,7 +1264,7 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, cc = ZPCI_PCI_LS_ERR; s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); } else { - pci_dereg_ioat(pbdev->iommu); + pci_dereg_ioat(pbdev); } break; case ZPCI_MOD_FC_REREG_IOAT: @@ -1239,7 +1275,7 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, cc = ZPCI_PCI_LS_ERR; s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); } else { - pci_dereg_ioat(pbdev->iommu); + pci_dereg_ioat(pbdev); if (reg_ioat(env, pbdev, fib, ra)) { cc = ZPCI_PCI_LS_ERR; s390_set_status_code(env, r1, ZPCI_MOD_ST_INSUF_RES); diff --git a/hw/s390x/s390-pci-kvm.c b/hw/s390x/s390-pci-kvm.c index ebb862abd0..2332efd676 100644 --- a/hw/s390x/s390-pci-kvm.c +++ b/hw/s390x/s390-pci-kvm.c @@ -137,3 +137,24 @@ int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) return kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); } + +int s390_pci_kvm_ioat_enable(S390PCIBusDevice *pbdev, uint64_t iota) +{ + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op = KVM_S390_ZPCIOP_REG_IOAT, + .u.reg_ioat.iota = iota + }; + + return kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); +} + +int s390_pci_kvm_ioat_disable(S390PCIBusDevice *pbdev) +{ + struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op = KVM_S390_ZPCIOP_DEREG_IOAT + }; + + return kvm_vm_ioctl(kvm_state, KVM_S390_ZPCI_OP, &args); +} diff --git a/include/hw/s390x/s390-pci-inst.h b/include/hw/s390x/s390-pci-inst.h index a55c448aad..13566fb7b4 100644 --- a/include/hw/s390x/s390-pci-inst.h +++ b/include/hw/s390x/s390-pci-inst.h @@ -99,7 +99,7 @@ typedef struct ZpciFib { } QEMU_PACKED ZpciFib; int pci_dereg_irqs(S390PCIBusDevice *pbdev); -void pci_dereg_ioat(S390PCIIOMMU *iommu); +void pci_dereg_ioat(S390PCIBusDevice *pbdev); int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra); int pcilg_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra); int pcistg_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra); diff --git a/include/hw/s390x/s390-pci-kvm.h b/include/hw/s390x/s390-pci-kvm.h index 09004d3f6c..e52e92d0f0 100644 --- a/include/hw/s390x/s390-pci-kvm.h +++ b/include/hw/s390x/s390-pci-kvm.h @@ -23,6 +23,8 @@ int s390_pci_kvm_interp_enable(S390PCIBusDevice *pbdev); int s390_pci_kvm_interp_disable(S390PCIBusDevice *pbdev); int s390_pci_kvm_aif_enable(S390PCIBusDevice *pbdev, ZpciFib *fib, bool assist); int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev); +int s390_pci_kvm_ioat_enable(S390PCIBusDevice *pbdev, uint64_t iota); +int s390_pci_kvm_ioat_disable(S390PCIBusDevice *pbdev); #else static inline bool s390_pci_kvm_zpciop_allowed(void) { @@ -53,6 +55,14 @@ static inline int s390_pci_kvm_aif_disable(S390PCIBusDevice *pbdev) { return -EINVAL; } +int s390_pci_kvm_ioat_enable(S390PCIBusDevice *pbdev, uint64_t iota) +{ + return -EINVAL; +} +int s390_pci_kvm_ioat_disable(S390PCIBusDevice *pbdev) +{ + return -EINVAL; +} #endif #endif From patchwork Mon Mar 14 19:49:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780728 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 736A6C433F5 for ; Mon, 14 Mar 2022 19:52:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244694AbiCNTxQ (ORCPT ); Mon, 14 Mar 2022 15:53:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244809AbiCNTwr (ORCPT ); Mon, 14 Mar 2022 15:52:47 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 507CB3EA85 for ; Mon, 14 Mar 2022 12:51:09 -0700 (PDT) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlUB4009362; Mon, 14 Mar 2022 19:50:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=t6qDx6aIkaJNO6w25jYBkIGJQJBd3FbBbGDVuHgbZFU=; b=W8U9VX2F2nZmBk6kRwVafNhWmAkNNVytn8RmjbA9OK6gFQf2ZNQVQFRj2bXf7S0BdsAj YatnO9oX8JWRm1gRCMwZVYd4C7LOzYrbOHIc8GffhtsvVlpnwNugZEDBpe0vw7jVoJ2G gLiDsvDYnJcvxMkvacl9IlZgGOD+OWg81/6QNd7v3Ha6Ev8euMM8atxwEFUxMaQGqdq+ N1WoJoZpz5otx475EChLBZ80Cw7azpA4vSrL5X/9U+qyr4qtoKgFm2hOlbPoK+/CHR8G tDmPXzQr8bPowhpO8jG32hF8oxD5KbXsVCYjmq+MufRCJNvHncH8XjMfB86lTfF3Upev kQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6d2rgqy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:25 +0000 Received: from m0098413.ppops.net (m0098413.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJoPXv016408; Mon, 14 Mar 2022 19:50:25 GMT Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6d2rgqu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:25 +0000 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJl7Pm014276; Mon, 14 Mar 2022 19:50:24 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma02wdc.us.ibm.com with ESMTP id 3erk59g8ee-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:24 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJoNaf21102856 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:50:23 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 67341112065; Mon, 14 Mar 2022 19:50:23 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 633F9112067; Mon, 14 Mar 2022 19:50:18 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:50:18 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 10/11] s390x/pci: use dtsm provided from vfio capabilities for interpreted devices Date: Mon, 14 Mar 2022 15:49:19 -0400 Message-Id: <20220314194920.58888-11-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: wX7fPWbhAbDP7X3xxu6sRtWjyoW1uFe0 X-Proofpoint-ORIG-GUID: GuLl2RhXq6RWhrtcOldE04OhZXY54EXc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxlogscore=999 clxscore=1015 spamscore=0 impostorscore=0 phishscore=0 mlxscore=0 adultscore=0 bulkscore=0 suspectscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When using the IOAT assist via interpretation, we should advertise what the host driver supports, not QEMU. Reviewed-by: Pierre Morel Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-vfio.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index 7808e8d939..def4e90a40 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -181,7 +181,11 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, resgrp->i = cap->noi; resgrp->maxstbl = cap->maxstbl; resgrp->version = cap->version; - resgrp->dtsm = ZPCI_DTSM; + if (hdr->version >= 2 && pbdev->interp) { + resgrp->dtsm = cap->dtsm; + } else { + resgrp->dtsm = ZPCI_DTSM; + } } } From patchwork Mon Mar 14 19:49:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12780727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0BA7C433EF for ; Mon, 14 Mar 2022 19:52:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244676AbiCNTxP (ORCPT ); Mon, 14 Mar 2022 15:53:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244826AbiCNTws (ORCPT ); Mon, 14 Mar 2022 15:52:48 -0400 Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 734813EB94 for ; Mon, 14 Mar 2022 12:51:10 -0700 (PDT) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 22EJlVFV009716; Mon, 14 Mar 2022 19:50:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=/CK9g+YBugw9n85Z2cnuiuKsYBg6z3VT0fTH55kr5aA=; b=mJ05YM7HL8dRV/lBaL5/q7+qSj7ebtONmyDHHBvN7g3m603NcvQb54xahSqj9YkuSqwZ 2VIRlRMTVW0hcu35HjNjXeO0dLu0xd6DfLpyqIBVU5A+bCY6Rg1Vt0zCEeEsLNyZNghZ h7GTnqu66bX1jXV8Ifqqs+BzKYw8V082cr+PzbxYagNtrbRbfCvhc0cUu6kfAZbV3KZs 6iacSo1BA6ppPENq0PRys3MvPU+HLM1Y6eEoGKC5EeT+eY30OpR97IWQdinwTfIvtPS2 QYmrLR5GUW1H8E6LQTfn4hiTiuA6bW2f+BTcFeh9rixH1Y4oLpfy7hZ9Eb91/ypgPoO4 uQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6cw8q3e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:32 +0000 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 22EJlVxO009741; Mon, 14 Mar 2022 19:50:31 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com with ESMTP id 3et6cw8q35-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:31 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 22EJl7Li004522; Mon, 14 Mar 2022 19:50:31 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma04wdc.us.ibm.com with ESMTP id 3erk59g82s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Mar 2022 19:50:31 +0000 Received: from b01ledav004.gho.pok.ibm.com (b01ledav004.gho.pok.ibm.com [9.57.199.109]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 22EJoTuh44106140 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Mar 2022 19:50:29 GMT Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9AD62112062; Mon, 14 Mar 2022 19:50:29 +0000 (GMT) Received: from b01ledav004.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1BF4611206E; Mon, 14 Mar 2022 19:50:24 +0000 (GMT) Received: from li-c92d2ccc-254b-11b2-a85c-a700b5bfb098.ibm.com.com (unknown [9.211.32.184]) by b01ledav004.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 14 Mar 2022 19:50:23 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: alex.williamson@redhat.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, farman@linux.ibm.com, pmorel@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, mst@redhat.com, pbonzini@redhat.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Subject: [PATCH v4 11/11] s390x/pci: let intercept devices have separate PCI groups Date: Mon, 14 Mar 2022 15:49:20 -0400 Message-Id: <20220314194920.58888-12-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220314194920.58888-1-mjrosato@linux.ibm.com> References: <20220314194920.58888-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 2A0vnuTlbeu-4oJrJrbovqev4NxeZ-at X-Proofpoint-GUID: u3Mgcy7UkSGS2tzD-BzKTCDfSE8IKDvh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.850,Hydra:6.0.425,FMLib:17.11.64.514 definitions=2022-03-14_13,2022-03-14_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 impostorscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2203140116 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's use the reserved pool of simulated PCI groups to allow intercept devices to have separate groups from interpreted devices as some group values may be different. If we run out of simulated PCI groups, subsequent intercept devices just get the default group. Furthermore, if we encounter any PCI groups from hostdevs that are marked as simulated, let's just assign them to the default group to avoid conflicts between host simulated groups and our own simulated groups. Reviewed-by: Pierre Morel Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-bus.c | 19 ++++++++++++++-- hw/s390x/s390-pci-vfio.c | 40 ++++++++++++++++++++++++++++++--- include/hw/s390x/s390-pci-bus.h | 6 ++++- 3 files changed, 59 insertions(+), 6 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 32894fbe9c..e932f72e89 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -748,13 +748,14 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn) object_unref(OBJECT(iommu)); } -S390PCIGroup *s390_group_create(int id) +S390PCIGroup *s390_group_create(int id, int host_id) { S390PCIGroup *group; S390pciState *s = s390_get_phb(); group = g_new0(S390PCIGroup, 1); group->id = id; + group->host_id = host_id; QTAILQ_INSERT_TAIL(&s->zpci_groups, group, link); return group; } @@ -772,12 +773,25 @@ S390PCIGroup *s390_group_find(int id) return NULL; } +S390PCIGroup *s390_group_find_host_sim(int host_id) +{ + S390PCIGroup *group; + S390pciState *s = s390_get_phb(); + + QTAILQ_FOREACH(group, &s->zpci_groups, link) { + if (group->id >= ZPCI_SIM_GRP_START && group->host_id == host_id) { + return group; + } + } + return NULL; +} + static void s390_pci_init_default_group(void) { S390PCIGroup *group; ClpRspQueryPciGrp *resgrp; - group = s390_group_create(ZPCI_DEFAULT_FN_GRP); + group = s390_group_create(ZPCI_DEFAULT_FN_GRP, ZPCI_DEFAULT_FN_GRP); resgrp = &group->zpci_group; resgrp->fr = 1; resgrp->dasm = 0; @@ -825,6 +839,7 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp) NULL, g_free); s->zpci_table = g_hash_table_new_full(g_int_hash, g_int_equal, NULL, NULL); s->bus_no = 0; + s->next_sim_grp = ZPCI_SIM_GRP_START; QTAILQ_INIT(&s->pending_sei); QTAILQ_INIT(&s->zpci_devs); QTAILQ_INIT(&s->zpci_dma_limit); diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index def4e90a40..de24c4ddf0 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -150,13 +150,17 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, { struct vfio_info_cap_header *hdr; struct vfio_device_info_cap_zpci_group *cap; + S390pciState *s = s390_get_phb(); ClpRspQueryPciGrp *resgrp; VFIOPCIDevice *vpci = container_of(pbdev->pdev, VFIOPCIDevice, pdev); hdr = vfio_get_device_info_cap(info, VFIO_DEVICE_INFO_CAP_ZPCI_GROUP); - /* If capability not provided, just use the default group */ - if (hdr == NULL) { + /* + * If capability not provided or the underlying hostdev is simulated, just + * use the default group. + */ + if (hdr == NULL || pbdev->zpci_fn.pfgid >= ZPCI_SIM_GRP_START) { trace_s390_pci_clp_cap(vpci->vbasedev.name, VFIO_DEVICE_INFO_CAP_ZPCI_GROUP); pbdev->zpci_fn.pfgid = ZPCI_DEFAULT_FN_GRP; @@ -165,11 +169,41 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, } cap = (void *) hdr; + /* + * For an intercept device, let's use an existing simulated group if one + * one was already created for other intercept devices in this group. + * If not, create a new simulated group if any are still available. + * If all else fails, just fall back on the default group. + */ + if (!pbdev->interp) { + pbdev->pci_group = s390_group_find_host_sim(pbdev->zpci_fn.pfgid); + if (pbdev->pci_group) { + /* Use existing simulated group */ + pbdev->zpci_fn.pfgid = pbdev->pci_group->id; + return; + } else { + if (s->next_sim_grp == ZPCI_DEFAULT_FN_GRP) { + /* All out of simulated groups, use default */ + trace_s390_pci_clp_cap(vpci->vbasedev.name, + VFIO_DEVICE_INFO_CAP_ZPCI_GROUP); + pbdev->zpci_fn.pfgid = ZPCI_DEFAULT_FN_GRP; + pbdev->pci_group = s390_group_find(ZPCI_DEFAULT_FN_GRP); + return; + } else { + /* We can assign a new simulated group */ + pbdev->zpci_fn.pfgid = s->next_sim_grp; + s->next_sim_grp++; + /* Fall through to create the new sim group using CLP info */ + } + } + } + /* See if the PCI group is already defined, create if not */ pbdev->pci_group = s390_group_find(pbdev->zpci_fn.pfgid); if (!pbdev->pci_group) { - pbdev->pci_group = s390_group_create(pbdev->zpci_fn.pfgid); + pbdev->pci_group = s390_group_create(pbdev->zpci_fn.pfgid, + pbdev->zpci_fn.pfgid); resgrp = &pbdev->pci_group->zpci_group; if (cap->flags & VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH) { diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h index 5b09f0cf2f..0605fcea24 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -315,13 +315,16 @@ typedef struct ZpciFmb { QEMU_BUILD_BUG_MSG(offsetof(ZpciFmb, fmt0) != 48, "padding in ZpciFmb"); #define ZPCI_DEFAULT_FN_GRP 0xFF +#define ZPCI_SIM_GRP_START 0xF0 typedef struct S390PCIGroup { ClpRspQueryPciGrp zpci_group; int id; + int host_id; QTAILQ_ENTRY(S390PCIGroup) link; } S390PCIGroup; -S390PCIGroup *s390_group_create(int id); +S390PCIGroup *s390_group_create(int id, int host_id); S390PCIGroup *s390_group_find(int id); +S390PCIGroup *s390_group_find_host_sim(int host_id); struct S390PCIBusDevice { DeviceState qdev; @@ -370,6 +373,7 @@ struct S390pciState { QTAILQ_HEAD(, S390PCIBusDevice) zpci_devs; QTAILQ_HEAD(, S390PCIDMACount) zpci_dma_limit; QTAILQ_HEAD(, S390PCIGroup) zpci_groups; + uint8_t next_sim_grp; }; S390pciState *s390_get_phb(void);