From patchwork Tue Oct 2 07:06:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Borntraeger X-Patchwork-Id: 10623123 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C13DD13BB for ; Tue, 2 Oct 2018 07:07:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD59F28704 for ; Tue, 2 Oct 2018 07:07:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A02DC2873F; Tue, 2 Oct 2018 07:07:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 944F628704 for ; Tue, 2 Oct 2018 07:07:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726811AbeJBNtV (ORCPT ); Tue, 2 Oct 2018 09:49:21 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:48346 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726776AbeJBNsu (ORCPT ); Tue, 2 Oct 2018 09:48:50 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w9274W7Z113995 for ; Tue, 2 Oct 2018 03:07:04 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2mv0v8p436-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 02 Oct 2018 03:07:04 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 2 Oct 2018 08:07:02 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 2 Oct 2018 08:06:59 +0100 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w9276wgT66715702 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 2 Oct 2018 07:06:58 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4103E52052; Tue, 2 Oct 2018 10:06:36 +0100 (BST) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTPS id 29B755204E; Tue, 2 Oct 2018 10:06:36 +0100 (BST) Received: by tuxmaker.boeblingen.de.ibm.com (Postfix, from userid 25651) id EEA5520F63D; Tue, 2 Oct 2018 09:06:57 +0200 (CEST) From: Christian Borntraeger To: Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= Cc: KVM , Cornelia Huck , Christian Borntraeger , linux-s390 , Janosch Frank , David Hildenbrand , Martin Schwidefsky Subject: [GIT PULL 06/28] s390: vfio-ap: sysfs interfaces to configure adapters Date: Tue, 2 Oct 2018 09:06:33 +0200 X-Mailer: git-send-email 2.17.0 In-Reply-To: <20181002070655.224110-1-borntraeger@de.ibm.com> References: <20181002070655.224110-1-borntraeger@de.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 18100207-4275-0000-0000-000002C3A0C8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18100207-4276-0000-0000-000037CEAED8 Message-Id: <20181002070655.224110-7-borntraeger@de.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-10-02_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1810020072 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Tony Krowiak Introduces two new sysfs attributes for the VFIO mediated matrix device for assigning AP adapters to and unassigning AP adapters from a mediated matrix device. The IDs of the AP adapters assigned to the mediated matrix device will be stored in an AP mask (APM). The bits in the APM, from most significant to least significant bit, correspond to AP adapter IDs (APID) 0 to 255. On some systems, the maximum allowable adapter number may be less than 255 - depending upon the host's AP configuration - and assignment may be rejected if the input adapter ID exceeds the limit. When an adapter is assigned, the bit corresponding to the APID will be set in the APM. Likewise, when an adapter is unassigned, the bit corresponding to the APID will be cleared from the APM. In order to successfully assign an adapter, the APQNs derived from the adapter ID being assigned and the queue indexes of all domains previously assigned: 1. Must be bound to the vfio_ap device driver. 2. Must not be assigned to any other mediated matrix device If there are no domains assigned to the mdev, then there must be an AP queue bound to the vfio_ap device driver with an APQN containing the APID, otherwise all domains subsequently assigned will fail because there will be no AP queues bound with an APQN containing the adapter ID. Assigning or un-assigning an AP adapter will be rejected if a guest using the mediated matrix device is running. The relevant sysfs structures are: /sys/devices/vfio_ap/matrix/ ...... [mdev_supported_types] ......... [vfio_ap-passthrough] ............ [devices] ...............[$uuid] .................. assign_adapter .................. unassign_adapter To assign an adapter to the $uuid mediated matrix device's APM, write the APID to the assign_adapter file. To unassign an adapter, write the APID to the unassign_adapter file. The APID is specified using conventional semantics: If it begins with 0x the number will be parsed as a hexadecimal number; if it begins with a 0 the number will be parsed as an octal number; otherwise, it will be parsed as a decimal number. For example, to assign adapter 173 (0xad) to the mediated matrix device $uuid: echo 173 > assign_adapter or echo 0xad > assign_adapter or echo 0255 > assign_adapter To unassign adapter 173 (0xad): echo 173 > unassign_adapter or echo 0xad > unassign_adapter or echo 0255 > unassign_adapter Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic Tested-by: Michael Mueller Tested-by: Farhan Ali Tested-by: Pierre Morel Reviewed-by: Cornelia Huck Message-Id: <20180925231641.4954-7-akrowiak@linux.vnet.ibm.com> Signed-off-by: Christian Borntraeger --- drivers/s390/crypto/vfio_ap_ops.c | 292 ++++++++++++++++++++++++++++++ 1 file changed, 292 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 99ed30315f56..8a63473c5ae7 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -106,9 +106,301 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = { NULL, }; +struct vfio_ap_queue_reserved { + unsigned long *apid; + unsigned long *apqi; + bool reserved; +}; + +/** + * vfio_ap_has_queue + * + * @dev: an AP queue device + * @data: a struct vfio_ap_queue_reserved reference + * + * Flags whether the AP queue device (@dev) has a queue ID containing the APQN, + * apid or apqi specified in @data: + * + * - If @data contains both an apid and apqi value, then @data will be flagged + * as reserved if the APID and APQI fields for the AP queue device matches + * + * - If @data contains only an apid value, @data will be flagged as + * reserved if the APID field in the AP queue device matches + * + * - If @data contains only an apqi value, @data will be flagged as + * reserved if the APQI field in the AP queue device matches + * + * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if + * @data does not contain either an apid or apqi. + */ +static int vfio_ap_has_queue(struct device *dev, void *data) +{ + struct vfio_ap_queue_reserved *qres = data; + struct ap_queue *ap_queue = to_ap_queue(dev); + ap_qid_t qid; + unsigned long id; + + if (qres->apid && qres->apqi) { + qid = AP_MKQID(*qres->apid, *qres->apqi); + if (qid == ap_queue->qid) + qres->reserved = true; + } else if (qres->apid && !qres->apqi) { + id = AP_QID_CARD(ap_queue->qid); + if (id == *qres->apid) + qres->reserved = true; + } else if (!qres->apid && qres->apqi) { + id = AP_QID_QUEUE(ap_queue->qid); + if (id == *qres->apqi) + qres->reserved = true; + } else { + return -EINVAL; + } + + return 0; +} + +/** + * vfio_ap_verify_queue_reserved + * + * @matrix_dev: a mediated matrix device + * @apid: an AP adapter ID + * @apqi: an AP queue index + * + * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device + * driver according to the following rules: + * + * - If both @apid and @apqi are not NULL, then there must be an AP queue + * device bound to the vfio_ap driver with the APQN identified by @apid and + * @apqi + * + * - If only @apid is not NULL, then there must be an AP queue device bound + * to the vfio_ap driver with an APQN containing @apid + * + * - If only @apqi is not NULL, then there must be an AP queue device bound + * to the vfio_ap driver with an APQN containing @apqi + * + * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL. + */ +static int vfio_ap_verify_queue_reserved(unsigned long *apid, + unsigned long *apqi) +{ + int ret; + struct vfio_ap_queue_reserved qres; + + qres.apid = apid; + qres.apqi = apqi; + qres.reserved = false; + + ret = driver_for_each_device(matrix_dev->device.driver, NULL, &qres, + vfio_ap_has_queue); + if (ret) + return ret; + + if (qres.reserved) + return 0; + + return -EADDRNOTAVAIL; +} + +static int +vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + int ret; + unsigned long apqi; + unsigned long nbits = matrix_mdev->matrix.aqm_max + 1; + + if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits) + return vfio_ap_verify_queue_reserved(&apid, NULL); + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) { + ret = vfio_ap_verify_queue_reserved(&apid, &apqi); + if (ret) + return ret; + } + + return 0; +} + +/** + * vfio_ap_mdev_verify_no_sharing + * + * Verifies that the APQNs derived from the cross product of the AP adapter IDs + * and AP queue indexes comprising the AP matrix are not configured for another + * mediated device. AP queue sharing is not allowed. + * + * @matrix_mdev: the mediated matrix device + * + * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE. + */ +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) +{ + struct ap_matrix_mdev *lstdev; + DECLARE_BITMAP(apm, AP_DEVICES); + DECLARE_BITMAP(aqm, AP_DOMAINS); + + list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) { + if (matrix_mdev == lstdev) + continue; + + memset(apm, 0, sizeof(apm)); + memset(aqm, 0, sizeof(aqm)); + + /* + * We work on full longs, as we can only exclude the leftover + * bits in non-inverse order. The leftover is all zeros. + */ + if (!bitmap_and(apm, matrix_mdev->matrix.apm, + lstdev->matrix.apm, AP_DEVICES)) + continue; + + if (!bitmap_and(aqm, matrix_mdev->matrix.aqm, + lstdev->matrix.aqm, AP_DOMAINS)) + continue; + + return -EADDRINUSE; + } + + return 0; +} + +/** + * assign_adapter_store + * + * @dev: the matrix device + * @attr: the mediated matrix device's assign_adapter attribute + * @buf: a buffer containing the AP adapter number (APID) to + * be assigned + * @count: the number of bytes in @buf + * + * Parses the APID from @buf and sets the corresponding bit in the mediated + * matrix device's APM. + * + * Returns the number of bytes processed if the APID is valid; otherwise, + * returns one of the following errors: + * + * 1. -EINVAL + * The APID is not a valid number + * + * 2. -ENODEV + * The APID exceeds the maximum value configured for the system + * + * 3. -EADDRNOTAVAIL + * An APQN derived from the cross product of the APID being assigned + * and the APQIs previously assigned is not bound to the vfio_ap device + * driver; or, if no APQIs have yet been assigned, the APID is not + * contained in an APQN bound to the vfio_ap device driver. + * + * 4. -EADDRINUSE + * An APQN derived from the cross product of the APID being assigned + * and the APQIs previously assigned is being used by another mediated + * matrix device + */ +static ssize_t assign_adapter_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + int ret; + unsigned long apid; + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + + ret = kstrtoul(buf, 0, &apid); + if (ret) + return ret; + + if (apid > matrix_mdev->matrix.apm_max) + return -ENODEV; + + /* + * Set the bit in the AP mask (APM) corresponding to the AP adapter + * number (APID). The bits in the mask, from most significant to least + * significant bit, correspond to APIDs 0-255. + */ + mutex_lock(&matrix_dev->lock); + + ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid); + if (ret) + goto done; + + set_bit_inv(apid, matrix_mdev->matrix.apm); + + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + if (ret) + goto share_err; + + ret = count; + goto done; + +share_err: + clear_bit_inv(apid, matrix_mdev->matrix.apm); +done: + mutex_unlock(&matrix_dev->lock); + + return ret; +} +static DEVICE_ATTR_WO(assign_adapter); + +/** + * unassign_adapter_store + * + * @dev: the matrix device + * @attr: the mediated matrix device's unassign_adapter attribute + * @buf: a buffer containing the adapter number (APID) to be unassigned + * @count: the number of bytes in @buf + * + * Parses the APID from @buf and clears the corresponding bit in the mediated + * matrix device's APM. + * + * Returns the number of bytes processed if the APID is valid; otherwise, + * returns one of the following errors: + * -EINVAL if the APID is not a number + * -ENODEV if the APID it exceeds the maximum value configured for the + * system + */ +static ssize_t unassign_adapter_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + int ret; + unsigned long apid; + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + + ret = kstrtoul(buf, 0, &apid); + if (ret) + return ret; + + if (apid > matrix_mdev->matrix.apm_max) + return -ENODEV; + + mutex_lock(&matrix_dev->lock); + clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); + mutex_unlock(&matrix_dev->lock); + + return count; +} +DEVICE_ATTR_WO(unassign_adapter); + +static struct attribute *vfio_ap_mdev_attrs[] = { + &dev_attr_assign_adapter.attr, + &dev_attr_unassign_adapter.attr, + NULL +}; + +static struct attribute_group vfio_ap_mdev_attr_group = { + .attrs = vfio_ap_mdev_attrs +}; + +static const struct attribute_group *vfio_ap_mdev_attr_groups[] = { + &vfio_ap_mdev_attr_group, + NULL +}; + static const struct mdev_parent_ops vfio_ap_matrix_ops = { .owner = THIS_MODULE, .supported_type_groups = vfio_ap_mdev_type_groups, + .mdev_attr_groups = vfio_ap_mdev_attr_groups, .create = vfio_ap_mdev_create, .remove = vfio_ap_mdev_remove, };