From patchwork Thu Jul 26 19:54:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Borntraeger X-Patchwork-Id: 10546381 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6CA9D14BC for ; Thu, 26 Jul 2018 19:54:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F84F2860A for ; Thu, 26 Jul 2018 19:54:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 43DFF2BD26; Thu, 26 Jul 2018 19:54:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FFBA2860A for ; Thu, 26 Jul 2018 19:54:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731993AbeGZVNS (ORCPT ); Thu, 26 Jul 2018 17:13:18 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58876 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731962AbeGZVNR (ORCPT ); Thu, 26 Jul 2018 17:13:17 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w6QJsO08049736 for ; Thu, 26 Jul 2018 15:54:55 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2kfjk0ng3c-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 26 Jul 2018 15:54:55 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 26 Jul 2018 20:54:53 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 26 Jul 2018 20:54:49 +0100 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w6QJslXW39911594 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 26 Jul 2018 19:54:47 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1EFC1AE04D; Thu, 26 Jul 2018 22:54:54 +0100 (BST) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F31F0AE045; Thu, 26 Jul 2018 22:54:53 +0100 (BST) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Thu, 26 Jul 2018 22:54:53 +0100 (BST) Received: by tuxmaker.boeblingen.de.ibm.com (Postfix, from userid 25651) id 0FB9820F5B5; Thu, 26 Jul 2018 21:54:47 +0200 (CEST) From: Christian Borntraeger To: Janosch Frank , alex.williamson@redhat.com, Kirti Wankhede Cc: KVM , Cornelia Huck , Christian Borntraeger , David Hildenbrand , linux-s390@vger.kernel.org, freude@de.ibm.com, schwidefsky@de.ibm.com, pmorel@linux.vnet.ibm.com, Halil Pasic , Tony Krowiak , Michael Mueller , buendgen@de.ibm.com, Boris Fiuczynski , jjherne@linux.ibm.com, Farhan Ali , berrange@redhat.com Subject: [PATCH v7 14/22] s390: vfio-ap: sysfs interface to activate mdev matrix Date: Thu, 26 Jul 2018 21:54:21 +0200 X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180726195429.31960-1-borntraeger@de.ibm.com> References: <20180726195429.31960-1-borntraeger@de.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 18072619-0016-0000-0000-000001EC78A4 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18072619-0017-0000-0000-000032417FE2 Message-Id: <20180726195429.31960-15-borntraeger@de.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-07-26_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1806210000 definitions=main-1807260204 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Tony Krowiak Provides a sysfs interface to activate AP matrix configured for the mediated matrix device. To activate the mdev matrix the APQNs that can be derived from the cross product of adapter and domain IDs must: 1. Be reserved by the AP bus for use by KVM 2. Not be assigned to another activated matrix mdev The relevant sysfs structures are: /sys/devices/vfio_ap ... [matrix] ...... [mdev_supported_types] ......... [vfio_ap-passthrough] ............ [devices] ...............[$uuid] .................. activate To activate the matrix configured for the matrix mdev, write a 1 to the activate file: echo 1 > activate To deactivate the matrix configured for the matrix mdev, write a 0 to the activate file: echo 0 > activate To view whether the matrix configured for the mdev is activated, print the activate file: cat activate 0: means not activated 1: means activated Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic Tested-by: Michael Mueller Tested-by: Farhan Ali Tested-by: Pierre Morel Signed-off-by: Christian Borntraeger --- drivers/s390/crypto/vfio_ap_ops.c | 244 ++++++++++++++++++++++++-- drivers/s390/crypto/vfio_ap_private.h | 1 + 2 files changed, 233 insertions(+), 12 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 9258902d9f70..675aa97612f6 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -52,6 +52,12 @@ static int vfio_ap_mdev_remove(struct mdev_device *mdev) struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); mutex_lock(&matrix_dev.lock); + if (matrix_mdev->activated) { + pr_warn("%s: %s: mdev remove failed: mdev %s is activated", + VFIO_AP_MODULE_NAME, __func__, matrix_mdev->name); + mutex_unlock(&matrix_dev.lock); + return -EBUSY; + } list_del(&matrix_mdev->list); mutex_unlock(&matrix_dev.lock); kfree(matrix_mdev); @@ -125,6 +131,15 @@ static ssize_t assign_adapter_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apid = matrix_mdev->matrix.apm_max; + mutex_lock(&matrix_dev.lock); + + if (matrix_mdev->activated) { + pr_warn("%s: %s: assign adapter '%s' failed: mdev %s is activated\n", + VFIO_AP_MODULE_NAME, __func__, buf, matrix_mdev->name); + ret = -EBUSY; + goto done; + } + ret = kstrtoul(buf, 0, &apid); if (ret || (apid > max_apid)) { pr_warn("%s: %s: adapter id '%s' not a value from 0 to %02lu(%#04lx)\n", @@ -133,17 +148,17 @@ static ssize_t assign_adapter_store(struct device *dev, if (!ret) ret = -EINVAL; - return ret; + goto done; } /* Set the bit in the AP mask (APM) corresponding to the AP adapter * number (APID). The bits in the mask, from most significant to least * significant bit, correspond to APIDs 0-255. */ - mutex_lock(&matrix_dev.lock); set_bit_inv(apid, matrix_mdev->matrix.apm); ret = count; +done: mutex_unlock(&matrix_dev.lock); return ret; @@ -175,6 +190,15 @@ static ssize_t unassign_adapter_store(struct device *dev, unsigned long max_apid = matrix_mdev->matrix.apm_max; + mutex_lock(&matrix_dev.lock); + + if (matrix_mdev->activated) { + pr_warn("%s: %s: unassign adapter '%s' failed: mdev %s is activated\n", + VFIO_AP_MODULE_NAME, __func__, buf, matrix_mdev->name); + ret = -EBUSY; + goto done; + } + ret = kstrtoul(buf, 0, &apid); if (ret || (apid > max_apid)) { pr_warn("%s: %s: adapter id '%s' must be a value from 0 to %02lu(%#04lx)\n", @@ -183,10 +207,9 @@ static ssize_t unassign_adapter_store(struct device *dev, if (!ret) ret = -EINVAL; - return ret; + goto done; } - mutex_lock(&matrix_dev.lock); if (!test_bit_inv(apid, matrix_mdev->matrix.apm)) { pr_warn("%s: %s: adapter id %02lu(%#04lx) not assigned\n", VFIO_AP_MODULE_NAME, __func__, apid, apid); @@ -214,6 +237,15 @@ static ssize_t assign_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; + mutex_lock(&matrix_dev.lock); + + if (matrix_mdev->activated) { + pr_warn("%s: %s: assign domain '%s' failed: mdev %s is activated\n", + VFIO_AP_MODULE_NAME, __func__, buf, matrix_mdev->name); + ret = -EBUSY; + goto done; + } + ret = kstrtoul(buf, 0, &apqi); if (ret || (apqi > max_apqi)) { pr_warn("%s: %s: domain id '%s' not a value from 0 to %02lu(%#04lx)\n", @@ -222,17 +254,17 @@ static ssize_t assign_domain_store(struct device *dev, if (!ret) ret = -EINVAL; - return ret; + goto done; } /* Set the bit in the AQM (bitmask) corresponding to the AP domain * number (APQI). The bits in the mask, from most significant to least * significant, correspond to numbers 0-255. */ - mutex_lock(&matrix_dev.lock); set_bit_inv(apqi, matrix_mdev->matrix.aqm); ret = count; +done: mutex_unlock(&matrix_dev.lock); return ret; @@ -249,15 +281,23 @@ static ssize_t unassign_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; + mutex_lock(&matrix_dev.lock); + + if (matrix_mdev->activated) { + pr_warn("%s: %s: unassign domain '%s' failed: mdev %s is activated\n", + VFIO_AP_MODULE_NAME, __func__, buf, matrix_mdev->name); + ret = -EBUSY; + goto done; + } + ret = kstrtoul(buf, 0, &apqi); if (ret || (apqi > max_apqi)) { pr_warn("%s: %s: domain id '%s' not a value from 0 to %02lu(%#04lx)\n", VFIO_AP_MODULE_NAME, __func__, buf, max_apqi, max_apqi); ret = -EINVAL; - return ret; + goto done; } - mutex_lock(&matrix_dev.lock); if (!test_bit_inv(apqi, matrix_mdev->matrix.aqm)) { pr_warn("%s: %s: domain %02lu(%#04lx) not assigned\n", VFIO_AP_MODULE_NAME, __func__, apqi, apqi); @@ -301,6 +341,15 @@ static ssize_t assign_control_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long maxid = matrix_mdev->matrix.adm_max; + mutex_lock(&matrix_dev.lock); + + if (matrix_mdev->activated) { + pr_warn("%s: %s: assign control domain '%s' failed: mdev %s is activated\n", + VFIO_AP_MODULE_NAME, __func__, buf, matrix_mdev->name); + ret = -EBUSY; + goto done; + } + ret = kstrtoul(buf, 0, &id); if (ret || (id > maxid)) { pr_warn("%s: %s: control domain id '%s' not a value from 0 to %02lu(%#04lx)\n", @@ -309,7 +358,7 @@ static ssize_t assign_control_domain_store(struct device *dev, if (!ret) ret = -EINVAL; - return ret; + goto done; } /* Set the bit in the ADM (bitmask) corresponding to the AP control @@ -317,10 +366,10 @@ static ssize_t assign_control_domain_store(struct device *dev, * least significant, correspond to IDs 0 up to the one less than the * number of control domains that can be assigned. */ - mutex_lock(&matrix_dev.lock); set_bit_inv(id, matrix_mdev->matrix.adm); ret = count; +done: mutex_unlock(&matrix_dev.lock); return ret; @@ -351,16 +400,24 @@ static ssize_t unassign_control_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_domid = matrix_mdev->matrix.adm_max; + mutex_lock(&matrix_dev.lock); + + if (matrix_mdev->activated) { + pr_warn("%s: %s: unassign control domain '%s' failed: mdev %s is activated\n", + VFIO_AP_MODULE_NAME, __func__, buf, matrix_mdev->name); + ret = -EBUSY; + goto done; + } + ret = kstrtoul(buf, 0, &domid); if (ret || (domid > max_domid)) { pr_warn("%s: %s: control domain id '%s' not a value from 0 to %02lu(%#04lx)\n", VFIO_AP_MODULE_NAME, __func__, buf, max_domid, max_domid); ret = -EINVAL; - return ret; + goto done; } - mutex_lock(&matrix_dev.lock); if (!test_bit_inv(domid, matrix_mdev->matrix.adm)) { pr_warn("%s: %s: control domain id %02lu(%#04lx) is not assigned\n", VFIO_AP_MODULE_NAME, __func__, domid, domid); @@ -434,6 +491,168 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr, } DEVICE_ATTR_RO(matrix); +static int vfio_ap_verify_queues_reserved(struct ap_matrix_mdev *matrix_mdev) +{ + unsigned long apid, apqi; + int ret = 0; + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, + matrix_mdev->matrix.apm_max + 1) { + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, + matrix_mdev->matrix.aqm_max + 1) { + if (!ap_owned_by_def_drv((int)apid, (int)apqi)) + continue; + + /* + * We want to log every APQN that is not reserved by + * the driver, so record the return code, log a message + * and allow the loop to continue + */ + ret = -EPERM; + pr_warn("%s: activate for %s failed: queue %02lx.%04lx owned by default driver\n", + VFIO_AP_MODULE_NAME, matrix_mdev->name, apid, + apqi); + } + } + + return ret; +} + +static void vfio_ap_mdev_log_sharing_err(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid, unsigned long apqi) +{ + pr_warn("%s: AP queue %02lx.%04lx is assigned to %s device\n", __func__, + apid, apqi, matrix_mdev->name); +} + +/** + * vfio_ap_mdev_verify_no_sharing + * + * Verifies that the APQNs derived from the cross product of the AP adapter IDs + * and AP queue indexes comprising the AP matrix are not configured for another + * activated mediated device. AP queue sharing is not allowed. + * + * @kvm: the KVM guest + * @matrix: the AP matrix + * + * Returns 0 if the APQNs are valid, otherwise; returns -EBUSY. + */ +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) +{ + int nbits; + struct ap_matrix_mdev *lstdev; + unsigned long apid, apqi; + unsigned long apm[BITS_TO_LONGS(matrix_mdev->matrix.apm_max + 1)]; + unsigned long aqm[BITS_TO_LONGS(matrix_mdev->matrix.aqm_max + 1)]; + + list_for_each_entry(lstdev, &matrix_dev.mdev_list, list) { + if ((matrix_mdev == lstdev) || !lstdev->activated) + continue; + + memset(apm, 0, sizeof(apm)); + memset(aqm, 0, sizeof(aqm)); + + /* + * We work on full longs, as we can only exclude the leftover + * bits in non-inverse order. The leftover is all zeros. + */ + nbits = sizeof(apm) * BITS_PER_BYTE; + if (!bitmap_and(apm, matrix_mdev->matrix.apm, + lstdev->matrix.apm, nbits)) + continue; + + nbits = sizeof(aqm) * BITS_PER_BYTE; + if (!bitmap_and(aqm, matrix_mdev->matrix.aqm, + lstdev->matrix.aqm, nbits)) + continue; + + goto sharing_err; + } + return 0; + +sharing_err: + + for_each_set_bit_inv(apid, apm, matrix_mdev->matrix.apm_max + 1) + for_each_set_bit_inv(apqi, aqm, + matrix_mdev->matrix.aqm_max + 1) + vfio_ap_mdev_log_sharing_err(lstdev, apid, apqi); + + return -EPERM; +} + +static int vfio_ap_mdev_activate(struct ap_matrix_mdev *matrix_mdev) +{ + int ret = 0; + + if (matrix_mdev->activated) + return 0; + + ret = vfio_ap_verify_queues_reserved(matrix_mdev); + if (ret) + return ret; + + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + if (ret) + return ret; + + matrix_mdev->activated = 1; + + return 0; +} + +static int vfio_ap_mdev_deactivate(struct ap_matrix_mdev *matrix_mdev) +{ + if (!matrix_mdev->activated) + return 0; + + matrix_mdev->activated = false; + + return 0; +} + +static ssize_t activate_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + int ret; + unsigned int activate; + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + + ret = kstrtouint(buf, 0, &activate); + if (ret || ((activate != 0) && (activate != 1))) { + pr_warn("%s: %s: input to activate '%s' not 0 or 1\n", + VFIO_AP_MODULE_NAME, __func__, buf); + + if (!ret) + ret = -EINVAL; + + return ret; + } + + mutex_lock(&matrix_dev.lock); + + ret = (activate) ? vfio_ap_mdev_activate(matrix_mdev) : + vfio_ap_mdev_deactivate(matrix_mdev); + if (ret) + goto done; + + ret = count; + +done: + mutex_unlock(&matrix_dev.lock); + + return ret; +} + +static ssize_t activate_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + + return sprintf(buf, "%d\n", matrix_mdev->activated); +} +DEVICE_ATTR_RW(activate); static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_assign_adapter.attr, @@ -444,6 +663,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_unassign_control_domain.attr, &dev_attr_control_domains.attr, &dev_attr_matrix.attr, + &dev_attr_activate.attr, NULL, }; diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index c2fd4a3e0ae3..df1996e6fce3 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -63,6 +63,7 @@ struct ap_matrix_mdev { const char *name; struct list_head list; struct ap_matrix matrix; + bool activated; }; static inline struct device *to_device(struct ap_matrix_dev *matrix_dev)