From patchwork Wed Dec 9 20:27:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11962493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95DDDC4167B for ; Wed, 9 Dec 2020 20:30:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 645FA22B39 for ; Wed, 9 Dec 2020 20:30:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388570AbgLIUaT (ORCPT ); Wed, 9 Dec 2020 15:30:19 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:6860 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387663AbgLIU36 (ORCPT ); Wed, 9 Dec 2020 15:29:58 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KCmPm171107; Wed, 9 Dec 2020 15:29:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=Ll19ePRbwoEsUoHXPbtitX6SHMdAUB9AHe5X2OcjHU8=; b=J1419j+jfIh9qofTLjUfl/UKmjhRo27Sq/elTRn0Z0SjmZ7pb6ATHmubxTIr5+OYrvQt prn8a+u0CmhFvXfhI7++SYJVPlE0NEs6n+pvNzcPqysGQ6kXym2lJ6AY1n5wHO0OcO3d vFXSmvwmU6vFtC0oc+M7PP1tm73ASW9zta5YR6QBnp25kwhFn9onp0QPyF4sLAmyI23o yKjjQjpODm2QXfnqx73YDeJtt7qJJhimEcJZvx9gYETH2Rv+QQ0F70fXsf6WBtoH42Rq kvZ2rGP54/4HT9G6m6dfivQBSxoN3yr6jfTaCaO12C2a6iXwGOF7KSxw4y36rgKYGlcH lA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 35avp1pys3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:29:17 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 0B9KLIMd007118; Wed, 9 Dec 2020 15:29:17 -0500 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 35avp1pyrq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:29:17 -0500 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KMAL7015879; Wed, 9 Dec 2020 20:29:16 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma01wdc.us.ibm.com with ESMTP id 3581u9gdy1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 20:29:16 +0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0B9KRxWU29884698 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 9 Dec 2020 20:27:59 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1CE64B2068; Wed, 9 Dec 2020 20:27:59 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EE889B2066; Wed, 9 Dec 2020 20:27:56 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.37.122]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 9 Dec 2020 20:27:56 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC 1/4] s390/pci: track alignment/length strictness for zpci_dev Date: Wed, 9 Dec 2020 15:27:47 -0500 Message-Id: <1607545670-1557-2-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> References: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2020-12-09_16:2020-12-09,2020-12-09 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 adultscore=0 impostorscore=0 malwarescore=0 clxscore=1015 suspectscore=0 mlxscore=0 spamscore=0 priorityscore=1501 bulkscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012090140 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Some zpci device types (e.g., ISM) follow different rules for length and alignment of pci instructions. Recognize this and keep track of it in the zpci_dev. Signed-off-by: Matthew Rosato Reviewed-by: Niklas Schnelle Reviewed-by: Pierre Morel --- arch/s390/include/asm/pci.h | 3 ++- arch/s390/include/asm/pci_clp.h | 4 +++- arch/s390/pci/pci_clp.c | 1 + 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index 2126289..f16ffba 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -133,7 +133,8 @@ struct zpci_dev { u8 has_hp_slot : 1; u8 is_physfn : 1; u8 util_str_avail : 1; - u8 reserved : 4; + u8 relaxed_align : 1; + u8 reserved : 3; unsigned int devfn; /* DEVFN part of the RID*/ struct mutex lock; diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h index 1f4b666..9fb7cbf 100644 --- a/arch/s390/include/asm/pci_clp.h +++ b/arch/s390/include/asm/pci_clp.h @@ -150,7 +150,9 @@ struct clp_rsp_query_pci_grp { u16 : 4; u16 noi : 12; /* number of interrupts */ u8 version; - u8 : 6; + u8 : 4; + u8 relaxed_align : 1; /* Relax length and alignment rules */ + u8 : 1; u8 frame : 1; u8 refresh : 1; /* TLB refresh mode */ u16 reserved2; diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c index 153720d..630f8fc 100644 --- a/arch/s390/pci/pci_clp.c +++ b/arch/s390/pci/pci_clp.c @@ -103,6 +103,7 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev, zdev->max_msi = response->noi; zdev->fmb_update = response->mui; zdev->version = response->version; + zdev->relaxed_align = response->relaxed_align; switch (response->version) { case 1: From patchwork Wed Dec 9 20:27:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11962497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEBD8C4167B for ; Wed, 9 Dec 2020 20:31:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9562823B99 for ; Wed, 9 Dec 2020 20:31:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388501AbgLIU3E (ORCPT ); Wed, 9 Dec 2020 15:29:04 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58175 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728450AbgLIU2t (ORCPT ); Wed, 9 Dec 2020 15:28:49 -0500 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KDeAl133362; Wed, 9 Dec 2020 15:28:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=RNAgxCZ6LPLzPrY2Z1vCKZGyclLScwFuVytng/DHOrI=; b=BaNzHybLSyGcz2Sz614G/ZI94sRwAUslOKYZcl5LfDQyrRpAKCWJyokL+VNlBeU/N321 hCaSRP0tHNgXFcVNpoyKpdBdnw+VOK0N2ao8g9AMqTWrlE/rXrrasZLjU/5E1UckyBqg 3HicBlNrvVcoKp7HsT+TKUP9NyTC/j/R6w8e44JzAZCvV+NPvEdVZTbeGvkeSS0Wecr2 9l1mo35QeC+SCTE49MiZiqCK7TQjRjN3/Bv1PmM/uxqOCSUwMtaOan7Ty9k+C+c0QaO/ d9gL7TFHKmH+4CRmuverkIbHTBj12e6DE5Pyh+m/brFgf+yC3205XrTPmN0P0CZwGT5m Jw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 35b3gdkgx9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:28:05 -0500 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 0B9KDlYl134069; Wed, 9 Dec 2020 15:28:04 -0500 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 35b3gdkgx0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:28:04 -0500 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KM7iX015859; Wed, 9 Dec 2020 20:28:03 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma01wdc.us.ibm.com with ESMTP id 3581u9gds6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 20:28:03 +0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0B9KS1va26083762 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 9 Dec 2020 20:28:01 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CD589B205F; Wed, 9 Dec 2020 20:28:01 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 90053B2065; Wed, 9 Dec 2020 20:27:59 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.37.122]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 9 Dec 2020 20:27:59 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC 2/4] vfio-pci/zdev: Pass the relaxed alignment flag Date: Wed, 9 Dec 2020 15:27:48 -0500 Message-Id: <1607545670-1557-3-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> References: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2020-12-09_16:2020-12-09,2020-12-09 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 adultscore=0 phishscore=0 malwarescore=0 mlxlogscore=953 priorityscore=1501 clxscore=1011 mlxscore=0 bulkscore=0 impostorscore=0 spamscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012090137 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use an additional bit of the VFIO_DEVICE_INFO_CAP_ZPCI_GROUP flags field to pass whether or not the associated device supports relaxed length and alignment for some I/O operations. Signed-off-by: Matthew Rosato Acked-by: Niklas Schnelle Reviewed-by: Pierre Morel --- drivers/vfio/pci/vfio_pci_zdev.c | 2 ++ include/uapi/linux/vfio_zdev.h | 1 + 2 files changed, 3 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c index 2296856..57e19ff 100644 --- a/drivers/vfio/pci/vfio_pci_zdev.c +++ b/drivers/vfio/pci/vfio_pci_zdev.c @@ -60,6 +60,8 @@ static int zpci_group_cap(struct zpci_dev *zdev, struct vfio_pci_device *vdev, .version = zdev->version }; + if (zdev->relaxed_align) + cap.flags |= VFIO_DEVICE_INFO_ZPCI_FLAG_RELAXED; return vfio_info_add_capability(caps, &cap.header, sizeof(cap)); } diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h index b430939..b0b6596 100644 --- a/include/uapi/linux/vfio_zdev.h +++ b/include/uapi/linux/vfio_zdev.h @@ -43,6 +43,7 @@ struct vfio_device_info_cap_zpci_group { __u64 msi_addr; /* MSI address */ __u64 flags; #define VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH 1 /* Program-specified TLB refresh */ +#define VFIO_DEVICE_INFO_ZPCI_FLAG_RELAXED 2 /* Relaxed Length and Alignment */ __u16 mui; /* Measurement Block Update Interval */ __u16 noi; /* Maximum number of MSIs */ __u16 maxstbl; /* Maximum Store Block Length */ From patchwork Wed Dec 9 20:27:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11962499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25E54C2BB40 for ; Wed, 9 Dec 2020 20:31:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EEEA323CD1 for ; Wed, 9 Dec 2020 20:31:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388497AbgLIU3E (ORCPT ); Wed, 9 Dec 2020 15:29:04 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:53246 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388480AbgLIU2t (ORCPT ); Wed, 9 Dec 2020 15:28:49 -0500 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KBsar179225; Wed, 9 Dec 2020 15:28:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=0u9A1QuSjmbJ/rAaQOhcdu29Aoy0Lx70TAbNYINn/vc=; b=dSI5+1J2e7lMJBti3Y5U6i3/Iu8kZucyZ/zrGtITPFVLVAHT7g6Xs6XtHDwB0F8P/Dsj lKGdocBPMu/GDSBpADOSk1yNA5Ix1h6G03KXrCkOL8whNzR1YNjeZ6n/EIc6ZlCepENU DzXSxLom8aK/cVUMraRp9ILnbKKGiWeExDqwBYnU8PT2W6rS9r5rUFHSE6LI+RMuI6x6 9kFwCmN3286LtLPfEFTpcEtEwn18ociJHRuvmgKf8vSzehPv1A48mOKeadk2ABvBillo qxwYYKCB8snKOTLcO6QkjBlYVmU5xs/jckZWF8EisysCNAplgsDykq3KleFipwI3nCqU og== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 35b5b9ge9v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:28:07 -0500 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 0B9KCCHP180949; Wed, 9 Dec 2020 15:28:07 -0500 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 35b5b9ge9k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:28:07 -0500 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KM99t015871; Wed, 9 Dec 2020 20:28:05 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma01wdc.us.ibm.com with ESMTP id 3581u9gdsg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 20:28:05 +0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0B9KS4Ti32834132 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 9 Dec 2020 20:28:04 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1B487B206B; Wed, 9 Dec 2020 20:28:04 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2691EB2065; Wed, 9 Dec 2020 20:28:02 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.37.122]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 9 Dec 2020 20:28:02 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC 3/4] s390/pci: Get hardware-reported max store block length Date: Wed, 9 Dec 2020 15:27:49 -0500 Message-Id: <1607545670-1557-4-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> References: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2020-12-09_16:2020-12-09,2020-12-09 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 bulkscore=0 mlxlogscore=999 impostorscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxscore=0 priorityscore=1501 clxscore=1015 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012090137 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We'll need to know this information for vfio passthrough. Signed-off-by: Matthew Rosato Reviewed-by: Pierre Morel --- arch/s390/include/asm/pci.h | 1 + arch/s390/include/asm/pci_clp.h | 3 ++- arch/s390/pci/pci_clp.c | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index f16ffba..04f1f48 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -181,6 +181,7 @@ struct zpci_dev { atomic64_t mapped_pages; atomic64_t unmapped_pages; + u16 maxstbl; u8 version; enum pci_bus_speed max_bus_speed; diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h index 9fb7cbf..ddd1604 100644 --- a/arch/s390/include/asm/pci_clp.h +++ b/arch/s390/include/asm/pci_clp.h @@ -155,7 +155,8 @@ struct clp_rsp_query_pci_grp { u8 : 1; u8 frame : 1; u8 refresh : 1; /* TLB refresh mode */ - u16 reserved2; + u16 : 3; + u16 maxstbl : 13; u16 mui; u16 : 16; u16 maxfaal; diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c index 630f8fc..99b64fe 100644 --- a/arch/s390/pci/pci_clp.c +++ b/arch/s390/pci/pci_clp.c @@ -104,6 +104,7 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev, zdev->fmb_update = response->mui; zdev->version = response->version; zdev->relaxed_align = response->relaxed_align; + zdev->maxstbl = response->maxstbl; switch (response->version) { case 1: From patchwork Wed Dec 9 20:27:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11962503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D396C4361B for ; Wed, 9 Dec 2020 20:32:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C636122B39 for ; Wed, 9 Dec 2020 20:32:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388483AbgLIU3D (ORCPT ); Wed, 9 Dec 2020 15:29:03 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58812 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726558AbgLIU2v (ORCPT ); Wed, 9 Dec 2020 15:28:51 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KEMs5143084; Wed, 9 Dec 2020 15:28:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=Z1DohrlWHPwbNtbny3gPW2Q2RZuygh3dJiI7rT2LcSc=; b=Gjzxr/KZuoFHciAQ65JthWD1jxigBhciyTgMIEmiBckCw5v+MzZVGevDHD32rTYFMJxH 2qVGzEe3UdG4p/MS1vv+CCa4vB6i7jqKQuAtONx8j9hYnF92d83RtU2yJop6kAAJ/Jzs rERJKcu8E96tF3stQEqf57DrcpE6R9byCw3u2a7lQGXjdRPae7tXc+M0qZXK+/mmtHNc EdHa2icSPpL9+i2TtgB/5OdRiaV9SCyga4uKE5w2U07ugONIiFsTcRySGQQ+2IxKB+CV 9M2nN6oMQFXu4F+DwHyFQIEY6Y112LwkCbWht/nAbbH1dzm5IZbeGS74iYdEbpOXMT8L uw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 35ayxnbe8m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:28:10 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 0B9KO7G9183523; Wed, 9 Dec 2020 15:28:09 -0500 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 35ayxnbe8a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 15:28:09 -0500 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 0B9KMZXh008882; Wed, 9 Dec 2020 20:28:08 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma03dal.us.ibm.com with ESMTP id 3581u9jrmt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 09 Dec 2020 20:28:08 +0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 0B9KS6fx32768292 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 9 Dec 2020 20:28:07 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B84E5B206B; Wed, 9 Dec 2020 20:28:06 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7A493B2064; Wed, 9 Dec 2020 20:28:04 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.37.122]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 9 Dec 2020 20:28:04 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC 4/4] vfio-pci/zdev: Introduce the zPCI I/O vfio region Date: Wed, 9 Dec 2020 15:27:50 -0500 Message-Id: <1607545670-1557-5-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> References: <1607545670-1557-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2020-12-09_16:2020-12-09,2020-12-09 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 lowpriorityscore=0 phishscore=0 impostorscore=0 adultscore=0 suspectscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012090137 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Some s390 PCI devices (e.g. ISM) perform I/O operations that have very specific requirements in terms of alignment as well as the patterns in which the data is read/written. Allowing these to proceed through the typical vfio_pci_bar_rw path will cause them to be broken in up in such a way that these requirements can't be guaranteed. In addition, ISM devices do not support the MIO codepaths that might be triggered on vfio I/O coming from userspace; we must be able to ensure that these devices use the non-MIO instructions. To facilitate this, provide a new vfio region by which non-MIO instructions can be passed directly to the host kernel s390 PCI layer, to be reliably issued as non-MIO instructions. This patch introduces the new vfio VFIO_REGION_SUBTYPE_IBM_ZPCI_IO region and implements the ability to pass PCISTB and PCILG instructions over it, as these are what is required for ISM devices. Signed-off-by: Matthew Rosato --- drivers/vfio/pci/vfio_pci.c | 8 ++ drivers/vfio/pci/vfio_pci_private.h | 6 ++ drivers/vfio/pci/vfio_pci_zdev.c | 158 ++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 4 + include/uapi/linux/vfio_zdev.h | 32 ++++++++ 5 files changed, 208 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index e619017..241b6fb 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -409,6 +409,14 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev) } } + if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV)) { + ret = vfio_pci_zdev_io_init(vdev); + if (ret && ret != -ENODEV) { + pci_warn(pdev, "Failed to setup zPCI I/O region\n"); + return ret; + } + } + vfio_pci_probe_mmaps(vdev); return 0; diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h index 5c90e56..bc49980 100644 --- a/drivers/vfio/pci/vfio_pci_private.h +++ b/drivers/vfio/pci/vfio_pci_private.h @@ -217,12 +217,18 @@ static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) #ifdef CONFIG_VFIO_PCI_ZDEV extern int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, struct vfio_info_cap *caps); +extern int vfio_pci_zdev_io_init(struct vfio_pci_device *vdev); #else static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, struct vfio_info_cap *caps) { return -ENODEV; } + +static inline int vfio_pci_zdev_io_init(struct vfio_pci_device *vdev) +{ + return -ENODEV; +} #endif #endif /* VFIO_PCI_PRIVATE_H */ diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c index 57e19ff..a962043 100644 --- a/drivers/vfio/pci/vfio_pci_zdev.c +++ b/drivers/vfio/pci/vfio_pci_zdev.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "vfio_pci_private.h" @@ -143,3 +144,160 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, return ret; } + +static size_t vfio_pci_zdev_io_rw(struct vfio_pci_device *vdev, + char __user *buf, size_t count, + loff_t *ppos, bool iswrite) +{ + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS; + struct vfio_region_zpci_io *region = vdev->region[i].data; + struct zpci_dev *zdev = to_zpci(vdev->pdev); + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; + void *base = region; + struct page *gpage; + void *gaddr; + u64 *data; + int ret; + u64 req; + + if ((!vdev->pdev->bus) || (!zdev)) + return -ENODEV; + + if (pos >= vdev->region[i].size) + return -EINVAL; + + count = min(count, (size_t)(vdev->region[i].size - pos)); + + if (!iswrite) { + /* Only allow reads to the _hdr area */ + if (pos + count > offsetof(struct vfio_region_zpci_io, req)) + return -EFAULT; + if (copy_to_user(buf, base + pos, count)) + return -EFAULT; + return count; + } + + /* Only allow writes to the _req area */ + if (pos < offsetof(struct vfio_region_zpci_io, req)) + return -EFAULT; + if (copy_from_user(base + pos, buf, count)) + return -EFAULT; + + /* + * Read operations are limited to 8B + */ + if ((region->req.flags & VFIO_ZPCI_IO_FLAG_READ) && + (region->req.len > 8)) { + return -EIO; + } + + /* + * Block write operations are limited to hardware-reported max + */ + if ((region->req.flags & VFIO_ZPCI_IO_FLAG_BLOCKW) && + (region->req.len > zdev->maxstbl)) { + return -EIO; + } + + /* + * While some devices may allow relaxed alignment for the PCISTB + * instruction, the VFIO region requires the input buffer to be on a + * DWORD boundary for all operations for simplicity. + */ + if (!IS_ALIGNED(region->req.gaddr, sizeof(uint64_t))) + return -EIO; + + /* + * For now, the largest allowed block I/O is advertised as PAGE_SIZE, + * and cannot exceed a page boundary - so a single page is enough. The + * guest should have validated this but let's double-check that the + * request will not cross a page boundary. + */ + if (((region->req.gaddr & ~PAGE_MASK) + + region->req.len - 1) & PAGE_MASK) { + return -EIO; + } + + mutex_lock(&zdev->lock); + + ret = get_user_pages_fast(region->req.gaddr & PAGE_MASK, 1, 0, &gpage); + if (ret <= 0) { + count = -EIO; + goto out; + } + gaddr = page_address(gpage); + gaddr += (region->req.gaddr & ~PAGE_MASK); + data = (u64 *)gaddr; + + req = ZPCI_CREATE_REQ(zdev->fh, region->req.pcias, region->req.len); + + /* Perform the requested I/O operation */ + if (region->req.flags & VFIO_ZPCI_IO_FLAG_READ) { + /* PCILG */ + ret = __zpci_load(data, req, + region->req.offset); + } else if (region->req.flags & VFIO_ZPCI_IO_FLAG_BLOCKW) { + /* PCISTB */ + ret = __zpci_store_block(data, req, + region->req.offset); + } else { + /* Undefined Operation or none provided */ + count = -EIO; + } + if (ret < 0) + count = -EIO; + + put_page(gpage); + +out: + mutex_unlock(&zdev->lock); + return count; +} + +static void vfio_pci_zdev_io_release(struct vfio_pci_device *vdev, + struct vfio_pci_region *region) +{ + kfree(region->data); +} + +static const struct vfio_pci_regops vfio_pci_zdev_io_regops = { + .rw = vfio_pci_zdev_io_rw, + .release = vfio_pci_zdev_io_release, +}; + +int vfio_pci_zdev_io_init(struct vfio_pci_device *vdev) +{ + struct vfio_region_zpci_io *region; + struct zpci_dev *zdev; + int ret; + + if (!vdev->pdev->bus) + return -ENODEV; + + zdev = to_zpci(vdev->pdev); + if (!zdev) + return -ENODEV; + + region = kmalloc(sizeof(struct vfio_region_zpci_io), GFP_KERNEL); + + ret = vfio_pci_register_dev_region(vdev, + PCI_VENDOR_ID_IBM | VFIO_REGION_TYPE_PCI_VENDOR_TYPE, + VFIO_REGION_SUBTYPE_IBM_ZPCI_IO, + &vfio_pci_zdev_io_regops, + sizeof(struct vfio_region_zpci_io), + VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE, + region); + + if (ret) { + kfree(region); + return ret; + } + + /* Setup the initial header information */ + region->hdr.flags = 0; + region->hdr.max = zdev->maxstbl; + region->hdr.reserved = 0; + region->hdr.reserved2 = 0; + + return ret; +} diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 2f313a2..6fbaec3 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -338,6 +338,10 @@ struct vfio_region_info_cap_type { * to do TLB invalidation on a GPU. */ #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1) +/* + * IBM zPCI I/O region + */ +#define VFIO_REGION_SUBTYPE_IBM_ZPCI_IO (2) /* sub-types for VFIO_REGION_TYPE_GFX */ #define VFIO_REGION_SUBTYPE_GFX_EDID (1) diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h index b0b6596..22d3408 100644 --- a/include/uapi/linux/vfio_zdev.h +++ b/include/uapi/linux/vfio_zdev.h @@ -76,4 +76,36 @@ struct vfio_device_info_cap_zpci_pfip { __u8 pfip[]; }; +/** + * VFIO_REGION_SUBTYPE_IBM_ZPCI_IO - VFIO zPCI PCI Direct I/O Region + * + * This region is used to transfer I/O operations from the guest directly + * to the host zPCI I/O layer. + * + * The _hdr area is user-readable and is used to provide setup information. + * The _req area is user-writable and is used to provide the I/O operation. + */ +struct vfio_zpci_io_hdr { + __u64 flags; + __u16 max; /* Max block operation size allowed */ + __u16 reserved; + __u32 reserved2; +}; + +struct vfio_zpci_io_req { + __u64 flags; +#define VFIO_ZPCI_IO_FLAG_READ (1 << 0) /* Read Operation Specified */ +#define VFIO_ZPCI_IO_FLAG_BLOCKW (1 << 1) /* Block Write Operation Specified */ + __u64 gaddr; /* Address of guest data */ + __u64 offset; /* Offset into target PCI Address Space */ + __u32 reserved; + __u16 len; /* Length of guest operation */ + __u8 pcias; /* Target PCI Address Space */ + __u8 reserved2; +}; + +struct vfio_region_zpci_io { + struct vfio_zpci_io_hdr hdr; + struct vfio_zpci_io_req req; +}; #endif