From patchwork Fri Mar 17 03:17:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dong Jia Shi X-Patchwork-Id: 9629941 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9400D60249 for ; Fri, 17 Mar 2017 07:05:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 838A22869C for ; Fri, 17 Mar 2017 07:05:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 782ED286A0; Fri, 17 Mar 2017 07:05:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B53CD2869C for ; Fri, 17 Mar 2017 07:05:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751066AbdCQHFa (ORCPT ); Fri, 17 Mar 2017 03:05:30 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:37982 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751170AbdCQHF3 (ORCPT ); Fri, 17 Mar 2017 03:05:29 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v2H3DsDd120633 for ; Thu, 16 Mar 2017 23:18:08 -0400 Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.152]) by mx0a-001b2d01.pphosted.com with ESMTP id 297x6aeraq-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Thu, 16 Mar 2017 23:18:08 -0400 Received: from localhost by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 16 Mar 2017 21:18:07 -0600 Received: from b03cxnp08025.gho.boulder.ibm.com (9.17.130.17) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 16 Mar 2017 21:18:04 -0600 Received: from b03ledav003.gho.boulder.ibm.com (b03ledav003.gho.boulder.ibm.com [9.17.130.234]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v2H3I3HD8454620; Thu, 16 Mar 2017 20:18:03 -0700 Received: from b03ledav003.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2D0736A03F; Thu, 16 Mar 2017 21:18:03 -0600 (MDT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by b03ledav003.gho.boulder.ibm.com (Postfix) with ESMTP id 4B8586A03B; Thu, 16 Mar 2017 21:18:01 -0600 (MDT) From: Dong Jia Shi To: kvm@vger.kernel.org, linux-s390@vger.kernel.org, qemu-devel@nongnu.org Cc: bjsdjshi@linux.vnet.ibm.com, renxiaof@linux.vnet.ibm.com, cornelia.huck@de.ibm.com, borntraeger@de.ibm.com, agraf@suse.com, alex.williamson@redhat.com, pmorel@linux.vnet.ibm.com, pasic@linux.vnet.ibm.com Subject: [PATCH v4 04/16] vfio: ccw: basic implementation for vfio_ccw driver Date: Fri, 17 Mar 2017 04:17:31 +0100 X-Mailer: git-send-email 2.8.4 In-Reply-To: <20170317031743.40128-1-bjsdjshi@linux.vnet.ibm.com> References: <20170317031743.40128-1-bjsdjshi@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17031703-0016-0000-0000-000006640A5F X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006796; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000206; SDB=6.00834930; UDB=6.00410081; IPR=6.00612585; BA=6.00005215; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014684; XFM=3.00000013; UTC=2017-03-17 03:18:06 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17031703-0017-0000-0000-0000383C3220 Message-Id: <20170317031743.40128-5-bjsdjshi@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-03-17_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1703170025 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To make vfio support subchannel devices, we need a css driver for the vfio subchannels. This patch adds a basic vfio-ccw subchannel driver for this purpose. To enable VFIO for vfio-ccw, enable S390_CCW_IOMMU config option and configure VFIO as required. Acked-by: Pierre Morel Signed-off-by: Dong Jia Shi --- arch/s390/Kconfig | 10 ++ arch/s390/include/asm/isc.h | 1 + drivers/iommu/Kconfig | 8 ++ drivers/s390/cio/Makefile | 3 + drivers/s390/cio/vfio_ccw_drv.c | 262 ++++++++++++++++++++++++++++++++++++ drivers/s390/cio/vfio_ccw_private.h | 25 ++++ 6 files changed, 309 insertions(+) create mode 100644 drivers/s390/cio/vfio_ccw_drv.c create mode 100644 drivers/s390/cio/vfio_ccw_private.h diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index a2dcef0..b53f0359 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -672,6 +672,16 @@ config EADM_SCH To compile this driver as a module, choose M here: the module will be called eadm_sch. +config VFIO_CCW + def_tristate n + prompt "Support for VFIO-CCW subchannels" + depends on S390_CCW_IOMMU && VFIO + help + This driver allows usage of I/O subchannels via VFIO-CCW. + + To compile this driver as a module, choose M here: the + module will be called vfio_ccw. + endmenu menu "Dump support" diff --git a/arch/s390/include/asm/isc.h b/arch/s390/include/asm/isc.h index 68d7d68..8a0b721 100644 --- a/arch/s390/include/asm/isc.h +++ b/arch/s390/include/asm/isc.h @@ -16,6 +16,7 @@ #define CONSOLE_ISC 1 /* console I/O subchannel */ #define EADM_SCH_ISC 4 /* EADM subchannels */ #define CHSC_SCH_ISC 7 /* CHSC subchannels */ +#define VFIO_CCW_ISC IO_SCH_ISC /* VFIO-CCW I/O subchannels */ /* Adapter interrupts. */ #define QDIO_AIRQ_ISC IO_SCH_ISC /* I/O subchannel in qdio mode */ #define PCI_ISC 2 /* PCI I/O subchannels */ diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 37e204f..6ee3a25 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -327,6 +327,14 @@ config S390_IOMMU help Support for the IOMMU API for s390 PCI devices. +config S390_CCW_IOMMU + bool "S390 CCW IOMMU Support" + depends on S390 && CCW + select IOMMU_API + help + Enables bits of IOMMU API required by VFIO. The iommu_ops + is not implemented as it is not necessary for VFIO. + config MTK_IOMMU bool "MTK IOMMU Support" depends on ARM || ARM64 diff --git a/drivers/s390/cio/Makefile b/drivers/s390/cio/Makefile index 3ab9aed..3d7390e 100644 --- a/drivers/s390/cio/Makefile +++ b/drivers/s390/cio/Makefile @@ -17,3 +17,6 @@ obj-$(CONFIG_CCWGROUP) += ccwgroup.o qdio-objs := qdio_main.o qdio_thinint.o qdio_debug.o qdio_setup.o obj-$(CONFIG_QDIO) += qdio.o + +vfio_ccw-objs += vfio_ccw_drv.o +obj-$(CONFIG_VFIO_CCW) += vfio_ccw.o diff --git a/drivers/s390/cio/vfio_ccw_drv.c b/drivers/s390/cio/vfio_ccw_drv.c new file mode 100644 index 0000000..de4f4f1 --- /dev/null +++ b/drivers/s390/cio/vfio_ccw_drv.c @@ -0,0 +1,262 @@ +/* + * VFIO based Physical Subchannel device driver + * + * Copyright IBM Corp. 2017 + * + * Author(s): Dong Jia Shi + * Xiao Feng Ren + */ + +#include +#include +#include +#include + +#include + +#include "vfio_ccw_private.h" + +/* + * Helpers + */ +static int vfio_ccw_sch_quiesce(struct subchannel *sch) +{ + struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev); + DECLARE_COMPLETION_ONSTACK(completion); + int iretry, ret = 0; + + spin_lock_irq(sch->lock); + if (!sch->schib.pmcw.ena) + goto out_unlock; + ret = cio_disable_subchannel(sch); + if (ret != -EBUSY) + goto out_unlock; + + do { + iretry = 255; + + ret = cio_cancel_halt_clear(sch, &iretry); + while (ret == -EBUSY) { + /* + * Flush all I/O and wait for + * cancel/halt/clear completion. + */ + private->completion = &completion; + spin_unlock_irq(sch->lock); + + wait_for_completion_timeout(&completion, 3*HZ); + + spin_lock_irq(sch->lock); + private->completion = NULL; + ret = cio_cancel_halt_clear(sch, &iretry); + }; + + ret = cio_disable_subchannel(sch); + } while (ret == -EBUSY); + +out_unlock: + spin_unlock_irq(sch->lock); + return ret; +} + +/* + * Sysfs interfaces + */ +static ssize_t chpids_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct subchannel *sch = to_subchannel(dev); + struct chsc_ssd_info *ssd = &sch->ssd_info; + ssize_t ret = 0; + int chp; + int mask; + + for (chp = 0; chp < 8; chp++) { + mask = 0x80 >> chp; + if (ssd->path_mask & mask) + ret += sprintf(buf + ret, "%02x ", ssd->chpid[chp].id); + else + ret += sprintf(buf + ret, "00 "); + } + ret += sprintf(buf+ret, "\n"); + return ret; +} + +static ssize_t pimpampom_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct subchannel *sch = to_subchannel(dev); + struct pmcw *pmcw = &sch->schib.pmcw; + + return sprintf(buf, "%02x %02x %02x\n", + pmcw->pim, pmcw->pam, pmcw->pom); +} + +static DEVICE_ATTR(chpids, 0444, chpids_show, NULL); +static DEVICE_ATTR(pimpampom, 0444, pimpampom_show, NULL); + +static struct attribute *vfio_subchannel_attrs[] = { + &dev_attr_chpids.attr, + &dev_attr_pimpampom.attr, + NULL, +}; + +static struct attribute_group vfio_subchannel_attr_group = { + .attrs = vfio_subchannel_attrs, +}; + +/* + * Css driver callbacks + */ +static void vfio_ccw_sch_irq(struct subchannel *sch) +{ + struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev); + + inc_irq_stat(IRQIO_CIO); + + if (!private) + return; + + if (private->completion) + complete(private->completion); +} + +static int vfio_ccw_sch_probe(struct subchannel *sch) +{ + struct pmcw *pmcw = &sch->schib.pmcw; + struct vfio_ccw_private *private; + int ret; + + if (pmcw->qf) { + dev_warn(&sch->dev, "vfio: ccw: does not support QDIO: %s\n", + dev_name(&sch->dev)); + return -ENODEV; + } + + private = kzalloc(sizeof(*private), GFP_KERNEL | GFP_DMA); + if (!private) + return -ENOMEM; + private->sch = sch; + dev_set_drvdata(&sch->dev, private); + + spin_lock_irq(sch->lock); + sch->isc = VFIO_CCW_ISC; + ret = cio_enable_subchannel(sch, (u32)(unsigned long)sch); + spin_unlock_irq(sch->lock); + if (ret) + goto out_free; + + ret = sysfs_create_group(&sch->dev.kobj, &vfio_subchannel_attr_group); + if (ret) + goto out_disable; + + return 0; + +out_disable: + cio_disable_subchannel(sch); +out_free: + dev_set_drvdata(&sch->dev, NULL); + kfree(private); + return ret; +} + +static int vfio_ccw_sch_remove(struct subchannel *sch) +{ + struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev); + + vfio_ccw_sch_quiesce(sch); + + sysfs_remove_group(&sch->dev.kobj, &vfio_subchannel_attr_group); + + dev_set_drvdata(&sch->dev, NULL); + + kfree(private); + + return 0; +} + +static void vfio_ccw_sch_shutdown(struct subchannel *sch) +{ + vfio_ccw_sch_quiesce(sch); +} + +/** + * vfio_ccw_sch_event - process subchannel event + * @sch: subchannel + * @process: non-zero if function is called in process context + * + * An unspecified event occurred for this subchannel. Adjust data according + * to the current operational state of the subchannel. Return zero when the + * event has been handled sufficiently or -EAGAIN when this function should + * be called again in process context. + */ +static int vfio_ccw_sch_event(struct subchannel *sch, int process) +{ + unsigned long flags; + + spin_lock_irqsave(sch->lock, flags); + if (!device_is_registered(&sch->dev)) + goto out_unlock; + + if (work_pending(&sch->todo_work)) + goto out_unlock; + + if (cio_update_schib(sch)) { + /* Not operational. */ + css_sched_sch_todo(sch, SCH_TODO_UNREG); + + /* + * TODO: + * Probably we should send the machine check to the guest. + */ + goto out_unlock; + } + +out_unlock: + spin_unlock_irqrestore(sch->lock, flags); + + return 0; +} + +static struct css_device_id vfio_ccw_sch_ids[] = { + { .match_flags = 0x1, .type = SUBCHANNEL_TYPE_IO, }, + { /* end of list */ }, +}; +MODULE_DEVICE_TABLE(css, vfio_ccw_sch_ids); + +static struct css_driver vfio_ccw_sch_driver = { + .drv = { + .name = "vfio_ccw", + .owner = THIS_MODULE, + }, + .subchannel_type = vfio_ccw_sch_ids, + .irq = vfio_ccw_sch_irq, + .probe = vfio_ccw_sch_probe, + .remove = vfio_ccw_sch_remove, + .shutdown = vfio_ccw_sch_shutdown, + .sch_event = vfio_ccw_sch_event, +}; + +static int __init vfio_ccw_sch_init(void) +{ + int ret; + + isc_register(VFIO_CCW_ISC); + ret = css_driver_register(&vfio_ccw_sch_driver); + if (ret) + isc_unregister(VFIO_CCW_ISC); + + return ret; +} + +static void __exit vfio_ccw_sch_exit(void) +{ + css_driver_unregister(&vfio_ccw_sch_driver); + isc_unregister(VFIO_CCW_ISC); +} +module_init(vfio_ccw_sch_init); +module_exit(vfio_ccw_sch_exit); + +MODULE_LICENSE("GPL v2"); diff --git a/drivers/s390/cio/vfio_ccw_private.h b/drivers/s390/cio/vfio_ccw_private.h new file mode 100644 index 0000000..38d69a5 --- /dev/null +++ b/drivers/s390/cio/vfio_ccw_private.h @@ -0,0 +1,25 @@ +/* + * Private stuff for vfio_ccw driver + * + * Copyright IBM Corp. 2017 + * + * Author(s): Dong Jia Shi + * Xiao Feng Ren + */ + +#ifndef _VFIO_CCW_PRIVATE_H_ +#define _VFIO_CCW_PRIVATE_H_ + +#include "css.h" + +/** + * struct vfio_ccw_private + * @sch: pointer to the subchannel + * @completion: synchronization helper of the I/O completion + */ +struct vfio_ccw_private { + struct subchannel *sch; + struct completion *completion; +} __aligned(8); + +#endif