From patchwork Tue Jan 19 20:02:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12030681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0F8BC433E0 for ; Tue, 19 Jan 2021 20:07:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8E0F82310B for ; Tue, 19 Jan 2021 20:07:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729548AbhASUHa (ORCPT ); Tue, 19 Jan 2021 15:07:30 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:52826 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728613AbhASUDW (ORCPT ); Tue, 19 Jan 2021 15:03:22 -0500 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10JK2WFA177644; Tue, 19 Jan 2021 15:02:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=Ll19ePRbwoEsUoHXPbtitX6SHMdAUB9AHe5X2OcjHU8=; b=ohL4nccilAUlyQ016hVzZFb16hXWwJmdELJEsFv/ZMNGmOd5J60ao+MAidVdzXvTY2DW 6eipx2VcitJQcgNXp55TrzcAH+6w+/zMEVFUcHT95iwRgbWTTfXI2hLeVtishY4+ACB0 g/6So5+dYgvW4Av+3IVUne+Tw7fWBOX6jk2Y2U0XAd7qqaGv1duwePk5qUHxNL+s4Khk GSltmzZhkiy/mHjeGzikmRcYEzwtPt2+DKYsGDh5J03Pndk/PbpKeY/naafpfT0aZcgM o3+MyLoObbBAbRKpp7KtXRO/1tfuhWW594iBF3f5BWvNnFzY3wPlmF9QJcBmJEeMNGFH uA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 366607g5tr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:02:40 -0500 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10JK2aeN178104; Tue, 19 Jan 2021 15:02:39 -0500 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0b-001b2d01.pphosted.com with ESMTP id 366607g5tc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:02:39 -0500 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10JK2U4X025577; Tue, 19 Jan 2021 20:02:39 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma05wdc.us.ibm.com with ESMTP id 363qs90q6v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 20:02:39 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10JK2b9630933286 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 Jan 2021 20:02:37 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7783EAC05B; Tue, 19 Jan 2021 20:02:37 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 87A93AC059; Tue, 19 Jan 2021 20:02:35 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.211.56.144]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 19 Jan 2021 20:02:35 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] s390/pci: track alignment/length strictness for zpci_dev Date: Tue, 19 Jan 2021 15:02:27 -0500 Message-Id: <1611086550-32765-2-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> References: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-19_09:2021-01-18,2021-01-19 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 mlxscore=0 impostorscore=0 spamscore=0 phishscore=0 adultscore=0 clxscore=1015 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101190106 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Some zpci device types (e.g., ISM) follow different rules for length and alignment of pci instructions. Recognize this and keep track of it in the zpci_dev. Signed-off-by: Matthew Rosato Reviewed-by: Niklas Schnelle Reviewed-by: Pierre Morel --- arch/s390/include/asm/pci.h | 3 ++- arch/s390/include/asm/pci_clp.h | 4 +++- arch/s390/pci/pci_clp.c | 1 + 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index 2126289..f16ffba 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -133,7 +133,8 @@ struct zpci_dev { u8 has_hp_slot : 1; u8 is_physfn : 1; u8 util_str_avail : 1; - u8 reserved : 4; + u8 relaxed_align : 1; + u8 reserved : 3; unsigned int devfn; /* DEVFN part of the RID*/ struct mutex lock; diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h index 1f4b666..9fb7cbf 100644 --- a/arch/s390/include/asm/pci_clp.h +++ b/arch/s390/include/asm/pci_clp.h @@ -150,7 +150,9 @@ struct clp_rsp_query_pci_grp { u16 : 4; u16 noi : 12; /* number of interrupts */ u8 version; - u8 : 6; + u8 : 4; + u8 relaxed_align : 1; /* Relax length and alignment rules */ + u8 : 1; u8 frame : 1; u8 refresh : 1; /* TLB refresh mode */ u16 reserved2; diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c index 153720d..630f8fc 100644 --- a/arch/s390/pci/pci_clp.c +++ b/arch/s390/pci/pci_clp.c @@ -103,6 +103,7 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev, zdev->max_msi = response->noi; zdev->fmb_update = response->mui; zdev->version = response->version; + zdev->relaxed_align = response->relaxed_align; switch (response->version) { case 1: From patchwork Tue Jan 19 20:02:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12030677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB897C433E0 for ; Tue, 19 Jan 2021 20:06:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C03123107 for ; Tue, 19 Jan 2021 20:06:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730117AbhASUDt (ORCPT ); Tue, 19 Jan 2021 15:03:49 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:61966 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729548AbhASUDY (ORCPT ); Tue, 19 Jan 2021 15:03:24 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10JK2Kfa178336; Tue, 19 Jan 2021 15:02:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=RNAgxCZ6LPLzPrY2Z1vCKZGyclLScwFuVytng/DHOrI=; b=rie9I8LGhJ4ToZKylnItpHBD0ukeNZ95UOfU2ytWOdSZEURz4bH8gYVOMpIHFal+prkE CZ2I2HnaOm1VLNMRGjyMYGnttGPMBwRdJlNzt7eVkJTtlN/1nOjpz1bhB6BzLS2Af7T9 lHliANrF4R3+j04gd2/8eG55FT/F4v1WqZ4ZctLo+X++Ob+QuifLPbQZunjL+ssiP7mR ZCcJsrD9nt90I/PAzn/CsDjd+JTuKDIESTWaN864+/LKr0WG6n+amYz5uH5pRP508qRa yHSDZeZi3VjzJnqlWwGn1DnnboSHjFegaLgfsKXn7sUHOeiq2Up/A8F/0s+PRZ0RmXe1 Rw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 36643wk6gs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:02:43 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10JK2gOU180554; Tue, 19 Jan 2021 15:02:42 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 36643wk6ge-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:02:42 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10JK2Rti023177; Tue, 19 Jan 2021 20:02:41 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma01dal.us.ibm.com with ESMTP id 363qs9pwtp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 20:02:41 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10JK2dwb43778360 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 Jan 2021 20:02:39 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D4EABAC059; Tue, 19 Jan 2021 20:02:39 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C4C79AC05B; Tue, 19 Jan 2021 20:02:37 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.211.56.144]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 19 Jan 2021 20:02:37 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] vfio-pci/zdev: Pass the relaxed alignment flag Date: Tue, 19 Jan 2021 15:02:28 -0500 Message-Id: <1611086550-32765-3-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> References: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-19_09:2021-01-18,2021-01-19 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 mlxlogscore=999 clxscore=1015 impostorscore=0 phishscore=0 priorityscore=1501 lowpriorityscore=0 malwarescore=0 suspectscore=0 adultscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101190106 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use an additional bit of the VFIO_DEVICE_INFO_CAP_ZPCI_GROUP flags field to pass whether or not the associated device supports relaxed length and alignment for some I/O operations. Signed-off-by: Matthew Rosato Acked-by: Niklas Schnelle Reviewed-by: Pierre Morel --- drivers/vfio/pci/vfio_pci_zdev.c | 2 ++ include/uapi/linux/vfio_zdev.h | 1 + 2 files changed, 3 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c index 2296856..57e19ff 100644 --- a/drivers/vfio/pci/vfio_pci_zdev.c +++ b/drivers/vfio/pci/vfio_pci_zdev.c @@ -60,6 +60,8 @@ static int zpci_group_cap(struct zpci_dev *zdev, struct vfio_pci_device *vdev, .version = zdev->version }; + if (zdev->relaxed_align) + cap.flags |= VFIO_DEVICE_INFO_ZPCI_FLAG_RELAXED; return vfio_info_add_capability(caps, &cap.header, sizeof(cap)); } diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h index b430939..b0b6596 100644 --- a/include/uapi/linux/vfio_zdev.h +++ b/include/uapi/linux/vfio_zdev.h @@ -43,6 +43,7 @@ struct vfio_device_info_cap_zpci_group { __u64 msi_addr; /* MSI address */ __u64 flags; #define VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH 1 /* Program-specified TLB refresh */ +#define VFIO_DEVICE_INFO_ZPCI_FLAG_RELAXED 2 /* Relaxed Length and Alignment */ __u16 mui; /* Measurement Block Update Interval */ __u16 noi; /* Maximum number of MSIs */ __u16 maxstbl; /* Maximum Store Block Length */ From patchwork Tue Jan 19 20:02:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12030679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5303CC433DB for ; Tue, 19 Jan 2021 20:07:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F21A12310B for ; Tue, 19 Jan 2021 20:07:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729690AbhASUD5 (ORCPT ); Tue, 19 Jan 2021 15:03:57 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:18150 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730011AbhASUDZ (ORCPT ); Tue, 19 Jan 2021 15:03:25 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10JJqFAR140713; Tue, 19 Jan 2021 15:02:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=0u9A1QuSjmbJ/rAaQOhcdu29Aoy0Lx70TAbNYINn/vc=; b=HxAEHm0+pcIGLUcfmvqC1ocr8s9HeQ0oDrxj22soqEKJkx0V0jSSTCP3Fal3fCzToFpW CJPFDi46anjZG7hpH3ZNoRJz+6bXyiXXEyve3fm6gziV+ArH+FMx9lC+9TnlN8dCOgYg OXsFPM6QqZNn2BzNho74zr37158Zw/LwYxFwDA6JrrwWYliuNPZhp8brgRD/ZcdDBEZX 6sOpAku7UYHdBkkT11LRO1XXmpqgivYGF1E/kZlJIsmUjYwUEU5BogZC82+/MjL2PsJm XMUMkbouJU5J/z2xT1NqCnEJfnjwInytlt9G4CiTOEOi4SEI+JZRItgtbQXZ4KJ4ATCk fA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3665y307qw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:02:45 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10JJsaCD147680; Tue, 19 Jan 2021 15:02:45 -0500 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 3665y307qd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:02:44 -0500 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10JJpV1K030023; Tue, 19 Jan 2021 20:02:43 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma03wdc.us.ibm.com with ESMTP id 363qs90pe8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 20:02:43 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10JK2g5G30146978 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 Jan 2021 20:02:42 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 23C8DAC060; Tue, 19 Jan 2021 20:02:42 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 26AB3AC059; Tue, 19 Jan 2021 20:02:40 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.211.56.144]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 19 Jan 2021 20:02:39 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] s390/pci: Get hardware-reported max store block length Date: Tue, 19 Jan 2021 15:02:29 -0500 Message-Id: <1611086550-32765-4-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> References: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-19_09:2021-01-18,2021-01-19 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 impostorscore=0 mlxscore=0 adultscore=0 phishscore=0 bulkscore=0 malwarescore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101190106 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We'll need to know this information for vfio passthrough. Signed-off-by: Matthew Rosato Reviewed-by: Pierre Morel --- arch/s390/include/asm/pci.h | 1 + arch/s390/include/asm/pci_clp.h | 3 ++- arch/s390/pci/pci_clp.c | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index f16ffba..04f1f48 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -181,6 +181,7 @@ struct zpci_dev { atomic64_t mapped_pages; atomic64_t unmapped_pages; + u16 maxstbl; u8 version; enum pci_bus_speed max_bus_speed; diff --git a/arch/s390/include/asm/pci_clp.h b/arch/s390/include/asm/pci_clp.h index 9fb7cbf..ddd1604 100644 --- a/arch/s390/include/asm/pci_clp.h +++ b/arch/s390/include/asm/pci_clp.h @@ -155,7 +155,8 @@ struct clp_rsp_query_pci_grp { u8 : 1; u8 frame : 1; u8 refresh : 1; /* TLB refresh mode */ - u16 reserved2; + u16 : 3; + u16 maxstbl : 13; u16 mui; u16 : 16; u16 maxfaal; diff --git a/arch/s390/pci/pci_clp.c b/arch/s390/pci/pci_clp.c index 630f8fc..99b64fe 100644 --- a/arch/s390/pci/pci_clp.c +++ b/arch/s390/pci/pci_clp.c @@ -104,6 +104,7 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev, zdev->fmb_update = response->mui; zdev->version = response->version; zdev->relaxed_align = response->relaxed_align; + zdev->maxstbl = response->maxstbl; switch (response->version) { case 1: From patchwork Tue Jan 19 20:02:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 12030715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91240C433DB for ; Tue, 19 Jan 2021 20:27:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 21B1123107 for ; Tue, 19 Jan 2021 20:27:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390547AbhASU1a (ORCPT ); Tue, 19 Jan 2021 15:27:30 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:32792 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390223AbhASUYf (ORCPT ); Tue, 19 Jan 2021 15:24:35 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10JK2K9D178309; Tue, 19 Jan 2021 15:23:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=IQMgiGhUZZdFvXz8BaA2KHszZfRA+WzBAJ9mKgLnEmI=; b=DFsYynOchPiPyu0+jZDW6h+RAqjFIP1vN1A5vcWiG3TRX0pewXaRAW1/x1rx3RNr8+8v Gb3Xbog8eQnN31UeI1AcFR7eSAsoFwtmNN8qTo1YRGh+B35B1iI5KUvOsfu3rIVcVxpu L1Pj8FpnUsll9JRyG6Mur5bW5O2fq7VmA99CzRoMtQqP7Shl1xmt6zTfNSDZl5c19RHk lNPq/2Kv+B+iiffMV4R+MynSzq/Mz+kCWn8nIUjpXK6d0T7k9iA8lkEOvpzRwViVRhl8 p/rb1cIYfGp/GxqMSuAqwhyTH8CtPd17l3bqoCWY5I15mXKoQhlHQ0j+aGQULPJbq73X Vw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 36643wku3w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:23:53 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10JK2Ofe178627; Tue, 19 Jan 2021 15:23:53 -0500 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 36643wku3e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 15:23:53 -0500 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10JJpTbq030018; Tue, 19 Jan 2021 20:02:46 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma03wdc.us.ibm.com with ESMTP id 363qs90peb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Jan 2021 20:02:46 +0000 Received: from b01ledav006.gho.pok.ibm.com (b01ledav006.gho.pok.ibm.com [9.57.199.111]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10JK2ill28115438 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 Jan 2021 20:02:44 GMT Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9469BAC059; Tue, 19 Jan 2021 20:02:44 +0000 (GMT) Received: from b01ledav006.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6C808AC05F; Tue, 19 Jan 2021 20:02:42 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.211.56.144]) by b01ledav006.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 19 Jan 2021 20:02:42 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com, schnelle@linux.ibm.com Cc: pmorel@linux.ibm.com, borntraeger@de.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, gerald.schaefer@linux.ibm.com, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] vfio-pci/zdev: Introduce the zPCI I/O vfio region Date: Tue, 19 Jan 2021 15:02:30 -0500 Message-Id: <1611086550-32765-5-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> References: <1611086550-32765-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-19_09:2021-01-18,2021-01-19 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 mlxlogscore=999 clxscore=1015 impostorscore=0 phishscore=0 priorityscore=1501 lowpriorityscore=0 malwarescore=0 suspectscore=0 adultscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101190106 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Some s390 PCI devices (e.g. ISM) perform I/O operations that have very specific requirements in terms of alignment as well as the patterns in which the data is read/written. Allowing these to proceed through the typical vfio_pci_bar_rw path will cause them to be broken in up in such a way that these requirements can't be guaranteed. In addition, ISM devices do not support the MIO codepaths that might be triggered on vfio I/O coming from userspace; we must be able to ensure that these devices use the non-MIO instructions. To facilitate this, provide a new vfio region by which non-MIO instructions can be passed directly to the host kernel s390 PCI layer, to be reliably issued as non-MIO instructions. This patch introduces the new vfio VFIO_REGION_SUBTYPE_IBM_ZPCI_IO region and implements the ability to pass PCISTB and PCILG instructions over it, as these are what is required for ISM devices. Signed-off-by: Matthew Rosato --- drivers/vfio/pci/vfio_pci.c | 8 ++ drivers/vfio/pci/vfio_pci_private.h | 6 ++ drivers/vfio/pci/vfio_pci_zdev.c | 158 ++++++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 4 + include/uapi/linux/vfio_zdev.h | 33 ++++++++ 5 files changed, 209 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 706de3e..e1c156e 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -407,6 +407,14 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev) } } + if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV)) { + ret = vfio_pci_zdev_io_init(vdev); + if (ret && ret != -ENODEV) { + pci_warn(pdev, "Failed to setup zPCI I/O region\n"); + return ret; + } + } + vfio_pci_probe_mmaps(vdev); return 0; diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pci_private.h index 5c90e56..bc49980 100644 --- a/drivers/vfio/pci/vfio_pci_private.h +++ b/drivers/vfio/pci/vfio_pci_private.h @@ -217,12 +217,18 @@ static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) #ifdef CONFIG_VFIO_PCI_ZDEV extern int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, struct vfio_info_cap *caps); +extern int vfio_pci_zdev_io_init(struct vfio_pci_device *vdev); #else static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, struct vfio_info_cap *caps) { return -ENODEV; } + +static inline int vfio_pci_zdev_io_init(struct vfio_pci_device *vdev) +{ + return -ENODEV; +} #endif #endif /* VFIO_PCI_PRIVATE_H */ diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_zdev.c index 57e19ff..a962043 100644 --- a/drivers/vfio/pci/vfio_pci_zdev.c +++ b/drivers/vfio/pci/vfio_pci_zdev.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "vfio_pci_private.h" @@ -143,3 +144,160 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_device *vdev, return ret; } + +static size_t vfio_pci_zdev_io_rw(struct vfio_pci_device *vdev, + char __user *buf, size_t count, + loff_t *ppos, bool iswrite) +{ + unsigned int i = VFIO_PCI_OFFSET_TO_INDEX(*ppos) - VFIO_PCI_NUM_REGIONS; + struct vfio_region_zpci_io *region = vdev->region[i].data; + struct zpci_dev *zdev = to_zpci(vdev->pdev); + loff_t pos = *ppos & VFIO_PCI_OFFSET_MASK; + void *base = region; + struct page *gpage; + void *gaddr; + u64 *data; + int ret; + u64 req; + + if ((!vdev->pdev->bus) || (!zdev)) + return -ENODEV; + + if (pos >= vdev->region[i].size) + return -EINVAL; + + count = min(count, (size_t)(vdev->region[i].size - pos)); + + if (!iswrite) { + /* Only allow reads to the _hdr area */ + if (pos + count > offsetof(struct vfio_region_zpci_io, req)) + return -EFAULT; + if (copy_to_user(buf, base + pos, count)) + return -EFAULT; + return count; + } + + /* Only allow writes to the _req area */ + if (pos < offsetof(struct vfio_region_zpci_io, req)) + return -EFAULT; + if (copy_from_user(base + pos, buf, count)) + return -EFAULT; + + /* + * Read operations are limited to 8B + */ + if ((region->req.flags & VFIO_ZPCI_IO_FLAG_READ) && + (region->req.len > 8)) { + return -EIO; + } + + /* + * Block write operations are limited to hardware-reported max + */ + if ((region->req.flags & VFIO_ZPCI_IO_FLAG_BLOCKW) && + (region->req.len > zdev->maxstbl)) { + return -EIO; + } + + /* + * While some devices may allow relaxed alignment for the PCISTB + * instruction, the VFIO region requires the input buffer to be on a + * DWORD boundary for all operations for simplicity. + */ + if (!IS_ALIGNED(region->req.gaddr, sizeof(uint64_t))) + return -EIO; + + /* + * For now, the largest allowed block I/O is advertised as PAGE_SIZE, + * and cannot exceed a page boundary - so a single page is enough. The + * guest should have validated this but let's double-check that the + * request will not cross a page boundary. + */ + if (((region->req.gaddr & ~PAGE_MASK) + + region->req.len - 1) & PAGE_MASK) { + return -EIO; + } + + mutex_lock(&zdev->lock); + + ret = get_user_pages_fast(region->req.gaddr & PAGE_MASK, 1, 0, &gpage); + if (ret <= 0) { + count = -EIO; + goto out; + } + gaddr = page_address(gpage); + gaddr += (region->req.gaddr & ~PAGE_MASK); + data = (u64 *)gaddr; + + req = ZPCI_CREATE_REQ(zdev->fh, region->req.pcias, region->req.len); + + /* Perform the requested I/O operation */ + if (region->req.flags & VFIO_ZPCI_IO_FLAG_READ) { + /* PCILG */ + ret = __zpci_load(data, req, + region->req.offset); + } else if (region->req.flags & VFIO_ZPCI_IO_FLAG_BLOCKW) { + /* PCISTB */ + ret = __zpci_store_block(data, req, + region->req.offset); + } else { + /* Undefined Operation or none provided */ + count = -EIO; + } + if (ret < 0) + count = -EIO; + + put_page(gpage); + +out: + mutex_unlock(&zdev->lock); + return count; +} + +static void vfio_pci_zdev_io_release(struct vfio_pci_device *vdev, + struct vfio_pci_region *region) +{ + kfree(region->data); +} + +static const struct vfio_pci_regops vfio_pci_zdev_io_regops = { + .rw = vfio_pci_zdev_io_rw, + .release = vfio_pci_zdev_io_release, +}; + +int vfio_pci_zdev_io_init(struct vfio_pci_device *vdev) +{ + struct vfio_region_zpci_io *region; + struct zpci_dev *zdev; + int ret; + + if (!vdev->pdev->bus) + return -ENODEV; + + zdev = to_zpci(vdev->pdev); + if (!zdev) + return -ENODEV; + + region = kmalloc(sizeof(struct vfio_region_zpci_io), GFP_KERNEL); + + ret = vfio_pci_register_dev_region(vdev, + PCI_VENDOR_ID_IBM | VFIO_REGION_TYPE_PCI_VENDOR_TYPE, + VFIO_REGION_SUBTYPE_IBM_ZPCI_IO, + &vfio_pci_zdev_io_regops, + sizeof(struct vfio_region_zpci_io), + VFIO_REGION_INFO_FLAG_READ | VFIO_REGION_INFO_FLAG_WRITE, + region); + + if (ret) { + kfree(region); + return ret; + } + + /* Setup the initial header information */ + region->hdr.flags = 0; + region->hdr.max = zdev->maxstbl; + region->hdr.reserved = 0; + region->hdr.reserved2 = 0; + + return ret; +} diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index d181277..5547f9b 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -338,6 +338,10 @@ struct vfio_region_info_cap_type { * to do TLB invalidation on a GPU. */ #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1) +/* + * IBM zPCI I/O region + */ +#define VFIO_REGION_SUBTYPE_IBM_ZPCI_IO (2) /* sub-types for VFIO_REGION_TYPE_GFX */ #define VFIO_REGION_SUBTYPE_GFX_EDID (1) diff --git a/include/uapi/linux/vfio_zdev.h b/include/uapi/linux/vfio_zdev.h index b0b6596..830acca4 100644 --- a/include/uapi/linux/vfio_zdev.h +++ b/include/uapi/linux/vfio_zdev.h @@ -76,4 +76,37 @@ struct vfio_device_info_cap_zpci_pfip { __u8 pfip[]; }; +/** + * VFIO_REGION_SUBTYPE_IBM_ZPCI_IO - VFIO zPCI PCI Direct I/O Region + * + * This region is used to transfer I/O operations from the guest directly + * to the host zPCI I/O layer. The same instruction requested by the guest + * (e.g. PCISTB) will always be used, even if the MIO variant is available. + * + * The _hdr area is user-readable and is used to provide setup information. + * The _req area is user-writable and is used to provide the I/O operation. + */ +struct vfio_zpci_io_hdr { + __u64 flags; + __u16 max; /* Max block operation size allowed */ + __u16 reserved; + __u32 reserved2; +}; + +struct vfio_zpci_io_req { + __u64 flags; +#define VFIO_ZPCI_IO_FLAG_READ (1 << 0) /* Read Operation Specified */ +#define VFIO_ZPCI_IO_FLAG_BLOCKW (1 << 1) /* Block Write Operation Specified */ + __u64 gaddr; /* Address of guest data */ + __u64 offset; /* Offset into target PCI Address Space */ + __u32 reserved; + __u16 len; /* Length of guest operation */ + __u8 pcias; /* Target PCI Address Space */ + __u8 reserved2; +}; + +struct vfio_region_zpci_io { + struct vfio_zpci_io_hdr hdr; + struct vfio_zpci_io_req req; +}; #endif