From patchwork Tue Apr 7 19:20:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478933 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AE09D92A for ; Tue, 7 Apr 2020 19:21:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 84F2A2082F for ; Tue, 7 Apr 2020 19:21:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727026AbgDGTUf (ORCPT ); Tue, 7 Apr 2020 15:20:35 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:7208 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726339AbgDGTUe (ORCPT ); Tue, 7 Apr 2020 15:20:34 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J37UY034644; Tue, 7 Apr 2020 15:20:32 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx33xe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:32 -0400 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J3LIG036064; Tue, 7 Apr 2020 15:20:32 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx33x3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:31 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKDlF007745; Tue, 7 Apr 2020 19:20:30 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma05wdc.us.ibm.com with ESMTP id 306hv6jffp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:30 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKTSo31064438 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:29 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3C76128060; Tue, 7 Apr 2020 19:20:29 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7772328071; Tue, 7 Apr 2020 19:20:28 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:28 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 01/15] s390/vfio-ap: store queue struct in hash table for quick access Date: Tue, 7 Apr 2020 15:20:01 -0400 Message-Id: <20200407192015.19887-2-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 phishscore=0 impostorscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 suspectscore=3 adultscore=0 clxscore=1015 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070153 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Rather than looping over potentially 65535 objects, let's store the structures for caching information about queue devices bound to the vfio_ap device driver in a hash table keyed by APQN. Signed-off-by: Tony Krowiak Reviewed-by: Cornelia Huck --- drivers/s390/crypto/vfio_ap_drv.c | 28 +++------ drivers/s390/crypto/vfio_ap_ops.c | 90 ++++++++++++++------------- drivers/s390/crypto/vfio_ap_private.h | 10 ++- 3 files changed, 60 insertions(+), 68 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index be2520cc010b..e9c226c0730e 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -51,15 +51,9 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids); */ static int vfio_ap_queue_dev_probe(struct ap_device *apdev) { - struct vfio_ap_queue *q; - - q = kzalloc(sizeof(*q), GFP_KERNEL); - if (!q) - return -ENOMEM; - dev_set_drvdata(&apdev->device, q); - q->apqn = to_ap_queue(&apdev->device)->qid; - q->saved_isc = VFIO_AP_ISC_INVALID; - return 0; + struct ap_queue *queue = to_ap_queue(&apdev->device); + + return vfio_ap_mdev_probe_queue(queue); } /** @@ -70,18 +64,9 @@ static int vfio_ap_queue_dev_probe(struct ap_device *apdev) */ static void vfio_ap_queue_dev_remove(struct ap_device *apdev) { - struct vfio_ap_queue *q; - int apid, apqi; - - mutex_lock(&matrix_dev->lock); - q = dev_get_drvdata(&apdev->device); - dev_set_drvdata(&apdev->device, NULL); - apid = AP_QID_CARD(q->apqn); - apqi = AP_QID_QUEUE(q->apqn); - vfio_ap_mdev_reset_queue(apid, apqi, 1); - vfio_ap_irq_disable(q); - kfree(q); - mutex_unlock(&matrix_dev->lock); + struct ap_queue *queue = to_ap_queue(&apdev->device); + + vfio_ap_mdev_remove_queue(queue); } static void vfio_ap_matrix_dev_release(struct device *dev) @@ -135,6 +120,7 @@ static int vfio_ap_matrix_dev_create(void) mutex_init(&matrix_dev->lock); INIT_LIST_HEAD(&matrix_dev->mdev_list); + hash_init(matrix_dev->qtable); dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME); matrix_dev->device.parent = root_device; diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 5c0f53c6dde7..134860934fe7 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -26,45 +26,16 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev); -static int match_apqn(struct device *dev, const void *data) -{ - struct vfio_ap_queue *q = dev_get_drvdata(dev); - - return (q->apqn == *(int *)(data)) ? 1 : 0; -} - -/** - * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list - * @matrix_mdev: the associated mediated matrix - * @apqn: The queue APQN - * - * Retrieve a queue with a specific APQN from the list of the - * devices of the vfio_ap_drv. - * Verify that the APID and the APQI are set in the matrix. - * - * Returns the pointer to the associated vfio_ap_queue - */ -static struct vfio_ap_queue *vfio_ap_get_queue( - struct ap_matrix_mdev *matrix_mdev, - int apqn) +struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn) { struct vfio_ap_queue *q; - struct device *dev; - - if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm)) - return NULL; - if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm)) - return NULL; - - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &apqn, match_apqn); - if (!dev) - return NULL; - q = dev_get_drvdata(dev); - q->matrix_mdev = matrix_mdev; - put_device(dev); - return q; + hash_for_each_possible(matrix_dev->qtable, q, qnode, apqn) { + if (q && (apqn == q->apqn)) + return q; + } + + return NULL; } /** @@ -293,10 +264,11 @@ static int handle_pqap(struct kvm_vcpu *vcpu) matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook, struct ap_matrix_mdev, pqap_hook); - q = vfio_ap_get_queue(matrix_mdev, apqn); + q = vfio_ap_get_queue(apqn); if (!q) goto out_unlock; + q->matrix_mdev = matrix_mdev; status = vcpu->run->s.regs.gprs[1]; /* If IR bit(16) is set we enable the interrupt */ @@ -1116,16 +1088,11 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, static void vfio_ap_irq_disable_apqn(int apqn) { - struct device *dev; struct vfio_ap_queue *q; - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &apqn, match_apqn); - if (dev) { - q = dev_get_drvdata(dev); + q = vfio_ap_get_queue(apqn); + if (q) vfio_ap_irq_disable(q); - put_device(dev); - } } int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, @@ -1302,3 +1269,38 @@ void vfio_ap_mdev_unregister(void) { mdev_unregister_device(&matrix_dev->device); } + +int vfio_ap_mdev_probe_queue(struct ap_queue *queue) +{ + struct vfio_ap_queue *q; + + q = kzalloc(sizeof(*q), GFP_KERNEL); + if (!q) + return -ENOMEM; + + mutex_lock(&matrix_dev->lock); + dev_set_drvdata(&queue->ap_dev.device, q); + q->apqn = queue->qid; + q->saved_isc = VFIO_AP_ISC_INVALID; + hash_add(matrix_dev->qtable, &q->qnode, q->apqn); + mutex_unlock(&matrix_dev->lock); + + return 0; +} + +void vfio_ap_mdev_remove_queue(struct ap_queue *queue) +{ + struct vfio_ap_queue *q; + int apid, apqi; + + mutex_lock(&matrix_dev->lock); + q = dev_get_drvdata(&queue->ap_dev.device); + dev_set_drvdata(&queue->ap_dev.device, NULL); + apid = AP_QID_CARD(q->apqn); + apqi = AP_QID_QUEUE(q->apqn); + vfio_ap_mdev_reset_queue(apid, apqi, 1); + vfio_ap_irq_disable(q); + hash_del(&q->qnode); + kfree(q); + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index f46dde56b464..e1f8c82cc55d 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -18,6 +18,7 @@ #include #include #include +#include #include "ap_bus.h" @@ -43,6 +44,7 @@ struct ap_matrix_dev { struct list_head mdev_list; struct mutex lock; struct ap_driver *vfio_ap_drv; + DECLARE_HASHTABLE(qtable, 8); }; extern struct ap_matrix_dev *matrix_dev; @@ -90,8 +92,6 @@ struct ap_matrix_mdev { extern int vfio_ap_mdev_register(void); extern void vfio_ap_mdev_unregister(void); -int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, - unsigned int retry); struct vfio_ap_queue { struct ap_matrix_mdev *matrix_mdev; @@ -99,6 +99,10 @@ struct vfio_ap_queue { int apqn; #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; + struct hlist_node qnode; }; -struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q); + +int vfio_ap_mdev_probe_queue(struct ap_queue *queue); +void vfio_ap_mdev_remove_queue(struct ap_queue *queue); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Tue Apr 7 19:20:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5444392C for ; Tue, 7 Apr 2020 19:21:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 31EDD2075E for ; Tue, 7 Apr 2020 19:21:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727905AbgDGTVs (ORCPT ); Tue, 7 Apr 2020 15:21:48 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:47692 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726921AbgDGTUe (ORCPT ); Tue, 7 Apr 2020 15:20:34 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J3L9F129785; Tue, 7 Apr 2020 15:20:32 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3082hyk5w6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:32 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J3N6S129912; Tue, 7 Apr 2020 15:20:32 -0400 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com with ESMTP id 3082hyk5vy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:32 -0400 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKC3h013711; Tue, 7 Apr 2020 19:20:31 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma04wdc.us.ibm.com with ESMTP id 306hv6afgb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:31 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKUrS41157052 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:30 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 11F6628060; Tue, 7 Apr 2020 19:20:30 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 51E8D28068; Tue, 7 Apr 2020 19:20:29 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:29 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 02/15] s390/vfio-ap: manage link between queue struct and matrix mdev Date: Tue, 7 Apr 2020 15:20:02 -0400 Message-Id: <20200407192015.19887-3-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 spamscore=0 mlxscore=0 malwarescore=0 bulkscore=0 adultscore=0 phishscore=0 lowpriorityscore=0 clxscore=1015 suspectscore=11 mlxlogscore=999 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org A vfio_ap_queue structure is created for each queue device probed. To ensure that the matrix mdev to which a queue's APQN is assigned is linked to the queue structure as long as the queue device is bound to the vfio_ap device driver, let's go ahead and manage these links when the queue device is probed and removed as well as whenever an adapter or domain is assigned to or unassigned from the matrix mdev. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 75 ++++++++++++++++++++++++++++--- 1 file changed, 70 insertions(+), 5 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 134860934fe7..00699bd72d2b 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -148,7 +148,6 @@ struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) status.response_code); end_free: vfio_ap_free_aqic_resources(q); - q->matrix_mdev = NULL; return status; } @@ -250,7 +249,6 @@ static int handle_pqap(struct kvm_vcpu *vcpu) struct vfio_ap_queue *q; struct ap_queue_status qstatus = { .response_code = AP_RESPONSE_Q_NOT_AVAIL, }; - struct ap_matrix_mdev *matrix_mdev; /* If we do not use the AIV facility just go to userland */ if (!(vcpu->arch.sie_block->eca & ECA_AIV)) @@ -261,14 +259,11 @@ static int handle_pqap(struct kvm_vcpu *vcpu) if (!vcpu->kvm->arch.crypto.pqap_hook) goto out_unlock; - matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook, - struct ap_matrix_mdev, pqap_hook); q = vfio_ap_get_queue(apqn); if (!q) goto out_unlock; - q->matrix_mdev = matrix_mdev; status = vcpu->run->s.regs.gprs[1]; /* If IR bit(16) is set we enable the interrupt */ @@ -536,6 +531,31 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) return 0; } +/** + * vfio_ap_mdev_qlinks_for_apid + * + * @matrix_mdev: a matrix mediated device + * @apqi: the APID of one or more APQNs assigned to @matrix_mdev + * + * Set the link to @matrix_mdev for each queue device bound to the vfio_ap + * device driver with an APQN assigned to @matrix_mdev with the specified @apid. + * + * Note: If @matrix_mdev is NULL, the link to @matrix_mdev will be severed. + */ +static void vfio_ap_mdev_qlinks_for_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + unsigned long apqi; + struct vfio_ap_queue *q; + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, + matrix_mdev->matrix.aqm_max + 1) { + q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); + if (q) + q->matrix_mdev = matrix_mdev; + } +} + /** * assign_adapter_store * @@ -605,6 +625,7 @@ static ssize_t assign_adapter_store(struct device *dev, if (ret) goto share_err; + vfio_ap_mdev_qlinks_for_apid(matrix_mdev, apid); ret = count; goto done; @@ -656,6 +677,7 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); + vfio_ap_mdev_qlinks_for_apid(NULL, apid); mutex_unlock(&matrix_dev->lock); return count; @@ -682,6 +704,31 @@ vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev, return 0; } +/** + * vfio_ap_mdev_qlinks_for_apqi + * + * @matrix_mdev: a matrix mediated device + * @apqi: the APQI of one or more APQNs assigned to @matrix_mdev + * + * Set the link to @matrix_mdev for each queue device bound to the vfio_ap + * device driver with an APQN assigned to @matrix_mdev with the specified @apqi. + * + * Note: If @matrix_mdev is NULL, the link to @matrix_mdev will be severed. + */ +static void vfio_ap_mdev_qlinks_for_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + unsigned long apid; + struct vfio_ap_queue *q; + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, + matrix_mdev->matrix.apm_max + 1) { + q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); + if (q) + q->matrix_mdev = matrix_mdev; + } +} + /** * assign_domain_store * @@ -746,6 +793,7 @@ static ssize_t assign_domain_store(struct device *dev, if (ret) goto share_err; + vfio_ap_mdev_qlinks_for_apqi(matrix_mdev, apqi); ret = count; goto done; @@ -798,6 +846,7 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); + vfio_ap_mdev_qlinks_for_apqi(NULL, apqi); mutex_unlock(&matrix_dev->lock); return count; @@ -1270,6 +1319,21 @@ void vfio_ap_mdev_unregister(void) mdev_unregister_device(&matrix_dev->device); } +static void vfio_ap_mdev_for_queue(struct vfio_ap_queue *q) +{ + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); + struct ap_matrix_mdev *matrix_mdev; + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (test_bit_inv(apid, matrix_mdev->matrix.apm) && + test_bit_inv(apqi, matrix_mdev->matrix.aqm)) { + q->matrix_mdev = matrix_mdev; + break; + } + } +} + int vfio_ap_mdev_probe_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; @@ -1282,6 +1346,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue) dev_set_drvdata(&queue->ap_dev.device, q); q->apqn = queue->qid; q->saved_isc = VFIO_AP_ISC_INVALID; + vfio_ap_mdev_for_queue(q); hash_add(matrix_dev->qtable, &q->qnode, q->apqn); mutex_unlock(&matrix_dev->lock); From patchwork Tue Apr 7 19:20:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478909 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AFB6C174A for ; Tue, 7 Apr 2020 19:20:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 976722075E for ; Tue, 7 Apr 2020 19:20:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727121AbgDGTUg (ORCPT ); Tue, 7 Apr 2020 15:20:36 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:19248 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727002AbgDGTUf (ORCPT ); Tue, 7 Apr 2020 15:20:35 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J36qc034588; Tue, 7 Apr 2020 15:20:34 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx33y5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:34 -0400 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037JHF3u087179; Tue, 7 Apr 2020 15:20:33 -0400 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx33xp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:33 -0400 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKCYK010870; Tue, 7 Apr 2020 19:20:32 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma02wdc.us.ibm.com with ESMTP id 306hv6jdkx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:32 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKUtO45678932 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:30 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CBE4C2805E; Tue, 7 Apr 2020 19:20:30 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 29B8A2805C; Tue, 7 Apr 2020 19:20:30 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:30 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 03/15] s390/zcrypt: driver callback to indicate resource in use Date: Tue, 7 Apr 2020 15:20:03 -0400 Message-Id: <20200407192015.19887-4-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 phishscore=0 impostorscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 suspectscore=3 adultscore=0 clxscore=1015 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070153 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduces a new driver callback to prevent a root user from unbinding an AP queue from its device driver if the queue is in use. The intent of this callback is to provide a driver with the means to prevent a root user from inadvertently taking a queue away from a guest and giving it to the host while the guest is still using it. The callback will be invoked whenever a change to the AP bus's sysfs apmask or aqmask attributes would result in one or more AP queues being removed from its driver. If the callback responds in the affirmative for any driver queried, the change to the apmask or aqmask will be rejected with a device in use error. For this patch, only non-default drivers will be queried. Currently, there is only one non-default driver, the vfio_ap device driver. The vfio_ap device driver manages AP queues passed through to one or more guests and we don't want to unexpectedly take AP resources away from guests which are most likely independently administered. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/ap_bus.c | 144 +++++++++++++++++++++++++++++++++-- drivers/s390/crypto/ap_bus.h | 4 + 2 files changed, 142 insertions(+), 6 deletions(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index 5256e3ce84e5..af15c095e76a 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -35,6 +35,7 @@ #include #include #include +#include #include "ap_bus.h" #include "ap_debug.h" @@ -995,9 +996,11 @@ int ap_parse_mask_str(const char *str, newmap = kmalloc(size, GFP_KERNEL); if (!newmap) return -ENOMEM; - if (mutex_lock_interruptible(lock)) { - kfree(newmap); - return -ERESTARTSYS; + if (lock) { + if (mutex_lock_interruptible(lock)) { + kfree(newmap); + return -ERESTARTSYS; + } } if (*str == '+' || *str == '-') { @@ -1009,7 +1012,10 @@ int ap_parse_mask_str(const char *str, } if (rc == 0) memcpy(bitmap, newmap, size); - mutex_unlock(lock); + + if (lock) + mutex_unlock(lock); + kfree(newmap); return rc; } @@ -1196,12 +1202,75 @@ static ssize_t apmask_show(struct bus_type *bus, char *buf) return rc; } +int __verify_card_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newapm = (unsigned long *)data; + + /* + * If the reserved bits do not identify cards reserved for use by the + * non-default driver, there is no need to verify the driver is using + * the queues. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(newapm, ap_perms.aqm)) + rc = -EADDRINUSE; + + module_put(drv->owner); + + return rc; +} + +static int apmask_commit(unsigned long *newapm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DEVICES)]; + + /* + * Check if any bits in the apmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newapm, ap_perms.apm, AP_DEVICES)) { + rc = bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_card_reservations); + if (rc) + return rc; + } + + memcpy(ap_perms.apm, newapm, APMASKSIZE); + + return 0; +} + static ssize_t apmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + unsigned long newapm[BITS_TO_LONGS(AP_DEVICES)]; + + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + memcpy(newapm, ap_perms.apm, APMASKSIZE); + + rc = ap_parse_mask_str(buf, newapm, AP_DEVICES, NULL); + if (rc) + goto done; + + rc = apmask_commit(newapm); + if (rc) + goto done; - rc = ap_parse_mask_str(buf, ap_perms.apm, AP_DEVICES, &ap_perms_mutex); +done: + mutex_unlock(&ap_perms_mutex); if (rc) return rc; @@ -1227,12 +1296,75 @@ static ssize_t aqmask_show(struct bus_type *bus, char *buf) return rc; } +int __verify_queue_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newaqm = (unsigned long *)data; + + /* + * If the reserved bits do not identify queues reserved for use by the + * non-default driver, there is no need to verify the driver is using + * the queues. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(ap_perms.apm, newaqm)) + rc = -EADDRINUSE; + + module_put(drv->owner); + + return rc; +} + +static int aqmask_commit(unsigned long *newaqm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DOMAINS)]; + + /* + * Check if any bits in the aqmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newaqm, ap_perms.aqm, AP_DOMAINS)) { + rc = bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_queue_reservations); + if (rc) + return rc; + } + + memcpy(ap_perms.aqm, newaqm, AQMASKSIZE); + + return 0; +} + static ssize_t aqmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + unsigned long newaqm[BITS_TO_LONGS(AP_DEVICES)]; - rc = ap_parse_mask_str(buf, ap_perms.aqm, AP_DOMAINS, &ap_perms_mutex); + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + memcpy(newaqm, ap_perms.aqm, AQMASKSIZE); + + rc = ap_parse_mask_str(buf, newaqm, AP_DOMAINS, NULL); + if (rc) + goto done; + + rc = aqmask_commit(newaqm); + if (rc) + goto done; + +done: + mutex_unlock(&ap_perms_mutex); if (rc) return rc; diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 4348fdff1c61..86bf5e224ba4 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -138,6 +138,7 @@ struct ap_driver { void (*remove)(struct ap_device *); void (*suspend)(struct ap_device *); void (*resume)(struct ap_device *); + bool (*in_use)(unsigned long *apm, unsigned long *aqm); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) @@ -266,6 +267,9 @@ void ap_queue_init_state(struct ap_queue *aq); struct ap_card *ap_card_create(int id, int queue_depth, int raw_device_type, int comp_device_type, unsigned int functions); +#define APMASKSIZE (BITS_TO_LONGS(AP_DEVICES) * sizeof(unsigned long)) +#define AQMASKSIZE (BITS_TO_LONGS(AP_DOMAINS) * sizeof(unsigned long)) + struct ap_perms { unsigned long ioctlm[BITS_TO_LONGS(AP_IOCTLS)]; unsigned long apm[BITS_TO_LONGS(AP_DEVICES)]; From patchwork Tue Apr 7 19:20:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55CB292C for ; Tue, 7 Apr 2020 19:21:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 33E632075E for ; Tue, 7 Apr 2020 19:21:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727966AbgDGTVl (ORCPT ); Tue, 7 Apr 2020 15:21:41 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:30920 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727080AbgDGTUg (ORCPT ); Tue, 7 Apr 2020 15:20:36 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J9NAO100761; Tue, 7 Apr 2020 15:20:35 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 306mje1u39-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:35 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037JA0Vp102439; Tue, 7 Apr 2020 15:20:34 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 306mje1u30-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:34 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKCFf012934; Tue, 7 Apr 2020 19:20:33 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma01dal.us.ibm.com with ESMTP id 306hv6h2rt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:33 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKVJh42008880 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:31 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A801C28058; Tue, 7 Apr 2020 19:20:31 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ED2812805C; Tue, 7 Apr 2020 19:20:30 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:30 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 04/15] s390/vfio-ap: implement in-use callback for vfio_ap driver Date: Tue, 7 Apr 2020 15:20:04 -0400 Message-Id: <20200407192015.19887-5-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 mlxscore=0 priorityscore=1501 suspectscore=3 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's implement the callback to indicate when an APQN is in use by the vfio_ap device driver. The callback is invoked whenever a change to the apmask or aqmask would result in one or more queue devices being removed from the driver. The vfio_ap device driver will indicate a resource is in use if the APQN of any of the queue devices to be removed are assigned to any of the matrix mdevs under the driver's control. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 1 + drivers/s390/crypto/vfio_ap_ops.c | 47 +++++++++++++++++---------- drivers/s390/crypto/vfio_ap_private.h | 2 ++ 3 files changed, 33 insertions(+), 17 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index e9c226c0730e..5197a1fe14d4 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -174,6 +174,7 @@ static int __init vfio_ap_init(void) memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv)); vfio_ap_drv.probe = vfio_ap_queue_dev_probe; vfio_ap_drv.remove = vfio_ap_queue_dev_remove; + vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 00699bd72d2b..8ece0d52ff4c 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -500,7 +500,9 @@ vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, * * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE. */ -static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, + unsigned long *mdev_apm, + unsigned long *mdev_aqm) { struct ap_matrix_mdev *lstdev; DECLARE_BITMAP(apm, AP_DEVICES); @@ -517,12 +519,10 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) * We work on full longs, as we can only exclude the leftover * bits in non-inverse order. The leftover is all zeros. */ - if (!bitmap_and(apm, matrix_mdev->matrix.apm, - lstdev->matrix.apm, AP_DEVICES)) + if (!bitmap_and(apm, mdev_apm, lstdev->matrix.apm, AP_DEVICES)) continue; - if (!bitmap_and(aqm, matrix_mdev->matrix.aqm, - lstdev->matrix.aqm, AP_DOMAINS)) + if (!bitmap_and(aqm, mdev_aqm, lstdev->matrix.aqm, AP_DOMAINS)) continue; return -EADDRINUSE; @@ -594,6 +594,7 @@ static ssize_t assign_adapter_store(struct device *dev, { int ret; unsigned long apid; + DECLARE_BITMAP(apm, AP_DEVICES); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); @@ -619,18 +620,18 @@ static ssize_t assign_adapter_store(struct device *dev, if (ret) goto done; - set_bit_inv(apid, matrix_mdev->matrix.apm); + memset(apm, 0, sizeof(apm)); + set_bit_inv(apid, apm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm, + matrix_mdev->matrix.aqm); if (ret) - goto share_err; + goto done; + set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_qlinks_for_apid(matrix_mdev, apid); ret = count; - goto done; -share_err: - clear_bit_inv(apid, matrix_mdev->matrix.apm); done: mutex_unlock(&matrix_dev->lock); @@ -767,6 +768,7 @@ static ssize_t assign_domain_store(struct device *dev, { int ret; unsigned long apqi; + DECLARE_BITMAP(aqm, AP_DOMAINS); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; @@ -787,18 +789,18 @@ static ssize_t assign_domain_store(struct device *dev, if (ret) goto done; - set_bit_inv(apqi, matrix_mdev->matrix.aqm); + memset(aqm, 0, sizeof(aqm)); + set_bit_inv(apqi, aqm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, + matrix_mdev->matrix.apm, aqm); if (ret) - goto share_err; + goto done; + set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_qlinks_for_apqi(matrix_mdev, apqi); ret = count; - goto done; -share_err: - clear_bit_inv(apqi, matrix_mdev->matrix.aqm); done: mutex_unlock(&matrix_dev->lock); @@ -1369,3 +1371,14 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) kfree(q); mutex_unlock(&matrix_dev->lock); } + +bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) +{ + bool in_use; + + mutex_lock(&matrix_dev->lock); + in_use = vfio_ap_mdev_verify_no_sharing(NULL, apm, aqm) ? true : false; + mutex_unlock(&matrix_dev->lock); + + return in_use; +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index e1f8c82cc55d..4b6e144bab17 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -105,4 +105,6 @@ struct vfio_ap_queue { int vfio_ap_mdev_probe_queue(struct ap_queue *queue); void vfio_ap_mdev_remove_queue(struct ap_queue *queue); +bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Tue Apr 7 19:20:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478929 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E77B292C for ; Tue, 7 Apr 2020 19:21:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C543B208E4 for ; Tue, 7 Apr 2020 19:21:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727909AbgDGTVZ (ORCPT ); Tue, 7 Apr 2020 15:21:25 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58938 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727078AbgDGTUg (ORCPT ); Tue, 7 Apr 2020 15:20:36 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J3QW7175408; Tue, 7 Apr 2020 15:20:35 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082j93ej5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:35 -0400 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J3XMm176261; Tue, 7 Apr 2020 15:20:35 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082j93eht-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:35 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKC8x026989; Tue, 7 Apr 2020 19:20:34 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma02dal.us.ibm.com with ESMTP id 306hv71382-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:34 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKWoe53477690 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:32 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7488628058; Tue, 7 Apr 2020 19:20:32 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BBCBF2805A; Tue, 7 Apr 2020 19:20:31 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:31 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 05/15] s390/vfio-ap: introduce shadow CRYCB Date: Tue, 7 Apr 2020 15:20:05 -0400 Message-Id: <20200407192015.19887-6-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 malwarescore=0 suspectscore=3 impostorscore=0 bulkscore=0 lowpriorityscore=0 phishscore=0 spamscore=0 mlxlogscore=999 adultscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070153 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's introduce a shadow copy of the KVM guest's CRYCB and maintain it for the lifespan of the guest. The shadow CRYCB will be used to provide the AP configuration for a KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 31 +++++++++++++++++++++------ drivers/s390/crypto/vfio_ap_private.h | 1 + 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 8ece0d52ff4c..b8b678032ab7 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -280,14 +280,32 @@ static int handle_pqap(struct kvm_vcpu *vcpu) return 0; } +static void vfio_ap_matrix_clear(struct ap_matrix *matrix) +{ + bitmap_clear(matrix->apm, 0, AP_DEVICES); + bitmap_clear(matrix->aqm, 0, AP_DOMAINS); + bitmap_clear(matrix->adm, 0, AP_DOMAINS); +} + static void vfio_ap_matrix_init(struct ap_config_info *info, struct ap_matrix *matrix) { + vfio_ap_matrix_clear(matrix); matrix->apm_max = info->apxa ? info->Na : 63; matrix->aqm_max = info->apxa ? info->Nd : 15; matrix->adm_max = info->apxa ? info->Nd : 15; } +static bool vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + if (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd) { + kvm_arch_crypto_set_masks(matrix_mdev->kvm, + matrix_mdev->shadow_crycb.apm, + matrix_mdev->shadow_crycb.aqm, + matrix_mdev->shadow_crycb.adm); + } +} + static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) { struct ap_matrix_mdev *matrix_mdev; @@ -303,6 +321,7 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) matrix_mdev->mdev = mdev; vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); + vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_crycb); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; matrix_mdev->pqap_hook.owner = THIS_MODULE; @@ -1126,13 +1145,9 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (ret) return NOTIFY_DONE; - /* If there is no CRYCB pointer, then we can't copy the masks */ - if (!matrix_mdev->kvm->arch.crypto.crycbd) - return NOTIFY_DONE; - - kvm_arch_crypto_set_masks(matrix_mdev->kvm, matrix_mdev->matrix.apm, - matrix_mdev->matrix.aqm, - matrix_mdev->matrix.adm); + memcpy(&matrix_mdev->shadow_crycb, &matrix_mdev->matrix, + sizeof(matrix_mdev->shadow_crycb)); + vfio_ap_mdev_commit_crycb(matrix_mdev); return NOTIFY_OK; } @@ -1247,6 +1262,8 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev) kvm_put_kvm(matrix_mdev->kvm); matrix_mdev->kvm = NULL; } + + vfio_ap_matrix_clear(&matrix_mdev->shadow_crycb); mutex_unlock(&matrix_dev->lock); vfio_unregister_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY, diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 4b6e144bab17..87cc270c3212 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -83,6 +83,7 @@ struct ap_matrix { struct ap_matrix_mdev { struct list_head node; struct ap_matrix matrix; + struct ap_matrix shadow_crycb; struct notifier_block group_notifier; struct notifier_block iommu_notifier; struct kvm *kvm; From patchwork Tue Apr 7 19:20:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478931 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55B3A92A for ; Tue, 7 Apr 2020 19:21:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3F51F2082F for ; Tue, 7 Apr 2020 19:21:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727849AbgDGTVY (ORCPT ); Tue, 7 Apr 2020 15:21:24 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:38456 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727136AbgDGTUh (ORCPT ); Tue, 7 Apr 2020 15:20:37 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J38sr034767; Tue, 7 Apr 2020 15:20:36 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx3408-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:36 -0400 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J3k4G038238; Tue, 7 Apr 2020 15:20:36 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx33ys-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:36 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKRAP027262; Tue, 7 Apr 2020 19:20:35 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma02dal.us.ibm.com with ESMTP id 306hv71389-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:35 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKXW350790726 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:33 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 432F22805E; Tue, 7 Apr 2020 19:20:33 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 89B642805A; Tue, 7 Apr 2020 19:20:32 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:32 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 06/15] s390/vfio-ap: sysfs attribute to display the guest CRYCB Date: Tue, 7 Apr 2020 15:20:06 -0400 Message-Id: <20200407192015.19887-7-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 phishscore=0 impostorscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 suspectscore=3 adultscore=0 clxscore=1015 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070153 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The matrix of adapters and domains configured in a guest's CRYCB may differ from the matrix of adapters and domains assigned to the matrix mdev, so this patch introduces a sysfs attribute to display the CRYCB of a guest using the matrix mdev. For a matrix mdev denoted by $uuid, the crycb for a guest using the matrix mdev can be displayed as follows: cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix If a guest is not using the matrix mdev at the time the crycb is displayed, an error (ENODEV) will be returned. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 58 +++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index b8b678032ab7..beb2e68a2b1c 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1041,6 +1041,63 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr, } static DEVICE_ATTR_RO(matrix); +static ssize_t guest_matrix_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + char *bufpos = buf; + unsigned long apid; + unsigned long apqi; + unsigned long apid1; + unsigned long apqi1; + unsigned long napm_bits = matrix_mdev->shadow_crycb.apm_max + 1; + unsigned long naqm_bits = matrix_mdev->shadow_crycb.aqm_max + 1; + int nchars = 0; + int n; + + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return -ENODEV; + + apid1 = find_first_bit_inv(matrix_mdev->shadow_crycb.apm, napm_bits); + apqi1 = find_first_bit_inv(matrix_mdev->shadow_crycb.aqm, naqm_bits); + + mutex_lock(&matrix_dev->lock); + + if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) { + for_each_set_bit_inv(apid, matrix_mdev->shadow_crycb.apm, + napm_bits) { + for_each_set_bit_inv(apqi, + matrix_mdev->shadow_crycb.aqm, + naqm_bits) { + n = sprintf(bufpos, "%02lx.%04lx\n", apid, + apqi); + bufpos += n; + nchars += n; + } + } + } else if (apid1 < napm_bits) { + for_each_set_bit_inv(apid, matrix_mdev->shadow_crycb.apm, + napm_bits) { + n = sprintf(bufpos, "%02lx.\n", apid); + bufpos += n; + nchars += n; + } + } else if (apqi1 < naqm_bits) { + for_each_set_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm, + naqm_bits) { + n = sprintf(bufpos, ".%04lx\n", apqi); + bufpos += n; + nchars += n; + } + } + + mutex_unlock(&matrix_dev->lock); + + return nchars; +} +static DEVICE_ATTR_RO(guest_matrix); + static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_assign_adapter.attr, &dev_attr_unassign_adapter.attr, @@ -1050,6 +1107,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_unassign_control_domain.attr, &dev_attr_control_domains.attr, &dev_attr_matrix.attr, + &dev_attr_guest_matrix.attr, NULL, }; From patchwork Tue Apr 7 19:20:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478925 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9454492A for ; Tue, 7 Apr 2020 19:21:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 69C7C2078C for ; Tue, 7 Apr 2020 19:21:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727247AbgDGTUj (ORCPT ); Tue, 7 Apr 2020 15:20:39 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:25266 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727192AbgDGTUi (ORCPT ); Tue, 7 Apr 2020 15:20:38 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J91CS120236; Tue, 7 Apr 2020 15:20:36 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 308ye2g8rq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:36 -0400 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J9eou123167; Tue, 7 Apr 2020 15:20:36 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0b-001b2d01.pphosted.com with ESMTP id 308ye2g8rh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:36 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKZDF027895; Tue, 7 Apr 2020 19:20:35 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma03wdc.us.ibm.com with ESMTP id 306hv6ae53-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:35 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKYsv48890254 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:34 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0A5AD28059; Tue, 7 Apr 2020 19:20:34 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 589DE28064; Tue, 7 Apr 2020 19:20:33 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:33 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 07/15] s390/vfio-ap: filter CRYCB bits for unavailable queue devices Date: Tue, 7 Apr 2020 15:20:07 -0400 Message-Id: <20200407192015.19887-8-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 spamscore=0 mlxlogscore=999 bulkscore=0 phishscore=0 lowpriorityscore=0 mlxscore=0 suspectscore=3 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Even though APQNs for queues that are not in the host's AP configuration may be assigned to a matrix mdev, we do not want to set bits in the guest's CRYCB for APQNs that do not reference AP queue devices bound to the vfio_ap device driver. Ideally, it would be great if such APQNs could be filtered out before setting the bits in the guest's CRYCB; however, the architecture precludes filtering individual APQNs. Consequently, either the APID or APQI must be filtered. This patch introduces code to filter the APIDs or APQIs assigned to the matrix mdev's AP configuration before assigning them to the guest's AP configuration (i.e., CRYCB). We'll start by filtering the APIDs: If an APQN assigned to the matrix mdev's AP configuration does not reference a queue device bound to the vfio_ap device driver, the APID will be filtered out (i.e., not assigned to the guest's CRYCB). If every APID assigned to the matrix mdev is filtered out, then we'll try filtering the APQI's: If an APQN assigned to the matrix mdev's AP configuration does not reference a queue device bound to the vfio_ap device driver, the APQI will be filtered out (i.e., not assigned to the guest's CRYCB). In any case, if after filtering either the APIDs or APQIs there are any APQNs that can be assigned to the guest's CRYCB, they will be assigned and the CRYCB will be hot plugged into the guest. Example ======= APQNs bound to vfio_ap device driver: 04.0004 04.0047 04.0054 05.0005 05.0047 05.0054 Assignments to matrix mdev: APIDs APQIs -> APQNs 04 0004 04.0004 05 0005 04.0005 0047 04.0047 0054 04.0054 05.0004 05.0005 05.0047 04.0054 Filter APIDs: APID 04 will be filtered because APQN 04.0005 is not bound. APID 05 will be filtered because APQN 05.0004 is not bound. APQNs remaining: None Filter APQIs: APQI 04 will be filtered because APQN 05.0004 is not bound. APQI 05 will be filtered because APQN 04.0005 is not bound. APQNs remaining: 04.0047, 04.0054, 05.0047, 05.0054 APQNs 04.0047, 04.0054, 05.0047, 05.0054 will be assigned to the CRYCB and hot plugged into the KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 150 ++++++++++++++++++++++++++++-- 1 file changed, 140 insertions(+), 10 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index beb2e68a2b1c..25b7d978e3fd 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -296,14 +296,17 @@ static void vfio_ap_matrix_init(struct ap_config_info *info, matrix->adm_max = info->apxa ? info->Nd : 15; } -static bool vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev) +static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev) { - if (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd) { - kvm_arch_crypto_set_masks(matrix_mdev->kvm, - matrix_mdev->shadow_crycb.apm, - matrix_mdev->shadow_crycb.aqm, - matrix_mdev->shadow_crycb.adm); - } + return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd); +} + +static void vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + kvm_arch_crypto_set_masks(matrix_mdev->kvm, + matrix_mdev->shadow_crycb.apm, + matrix_mdev->shadow_crycb.aqm, + matrix_mdev->shadow_crycb.adm); } static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) @@ -550,6 +553,130 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, return 0; } +static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, + struct ap_matrix *shadow_crycb, + bool filter_apids) +{ + unsigned long apid, apqi, apqn; + + memcpy(shadow_crycb, &matrix_mdev->matrix, sizeof(*shadow_crycb)); + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { + /* + * If the APID is not assigned to the host AP configuration, + * we can not assign it to the guest's AP configuration + */ + if (!test_bit_inv(apid, + (unsigned long *)matrix_dev->info.apm)) { + clear_bit_inv(apid, shadow_crycb->apm); + continue; + } + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, + AP_DOMAINS) { + /* + * If the APQI is not assigned to the host AP + * configuration, then it can not be assigned to the + * guest's AP configuration + */ + if (!test_bit_inv(apqi, (unsigned long *) + matrix_dev->info.aqm)) { + clear_bit_inv(apqi, shadow_crycb->aqm); + continue; + } + + /* + * If the APQN is not bound to the vfio_ap device + * driver, then we can't assign it to the guest's + * AP configuration. The AP architecture won't + * allow filtering of a single APQN, so if we're + * filtering APIDs, then filter the APID; otherwise, + * filter the APQI. + */ + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_queue(apqn)) { + if (filter_apids) + clear_bit_inv(apid, shadow_crycb->apm); + else + clear_bit_inv(apqi, shadow_crycb->aqm); + break; + } + } + + /* + * If we're filtering APQIs and all of them have been filtered, + * there's no need to continue filtering. + */ + if (!filter_apids) + if (bitmap_empty(shadow_crycb->aqm, AP_DOMAINS)) + break; + } + + return bitmap_weight(shadow_crycb->apm, AP_DEVICES) * + bitmap_weight(shadow_crycb->aqm, AP_DOMAINS); +} + +static bool vfio_ap_mdev_configure_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + int napm, naqm; + struct ap_matrix shadow_crycb; + + vfio_ap_matrix_init(&matrix_dev->info, &shadow_crycb); + napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES); + naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS); + + /* + * If there are no APIDs or no APQIs assigned to the matrix mdev, + * then no APQNs shall be assigned to the guest CRYCB. + */ + if ((napm != 0) || (naqm != 0)) { + /* + * Filter the APIDs assigned to the matrix mdev for APQNs that + * do not reference an AP queue device bound to the driver. + */ + napm = vfio_ap_mdev_filter_matrix(matrix_mdev, &shadow_crycb, + true); + /* + * If there are no APQNs that can be assigned to the guest's + * CRYCB after filtering, then try filtering the APQIs. + */ + if (napm == 0) { + naqm = vfio_ap_mdev_filter_matrix(matrix_mdev, + &shadow_crycb, false); + + /* + * If there are no APQNs that can be assigned to the + * matrix mdev after filtering the APQIs, then no APQNs + * shall be assigned to the guest's CRYCB. + */ + if (naqm == 0) { + bitmap_clear(shadow_crycb.apm, 0, AP_DEVICES); + bitmap_clear(shadow_crycb.aqm, 0, AP_DOMAINS); + } + } + } + + /* + * If the guest's AP configuration has not changed, then return + * indicating such. + */ + if (bitmap_equal(matrix_mdev->shadow_crycb.apm, shadow_crycb.apm, + AP_DEVICES) && + bitmap_equal(matrix_mdev->shadow_crycb.aqm, shadow_crycb.aqm, + AP_DOMAINS) && + bitmap_equal(matrix_mdev->shadow_crycb.adm, shadow_crycb.adm, + AP_DOMAINS)) + return false; + + /* + * Copy the changes to the guest's CRYCB, then return indicating that + * the guest's AP configuration has changed. + */ + memcpy(&matrix_mdev->shadow_crycb, &shadow_crycb, sizeof(shadow_crycb)); + + return true; +} + /** * vfio_ap_mdev_qlinks_for_apid * @@ -1203,9 +1330,11 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (ret) return NOTIFY_DONE; - memcpy(&matrix_mdev->shadow_crycb, &matrix_mdev->matrix, - sizeof(matrix_mdev->shadow_crycb)); - vfio_ap_mdev_commit_crycb(matrix_mdev); + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return NOTIFY_DONE; + + if (vfio_ap_mdev_configure_crycb(matrix_mdev)) + vfio_ap_mdev_commit_crycb(matrix_mdev); return NOTIFY_OK; } @@ -1315,6 +1444,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev) mutex_lock(&matrix_dev->lock); if (matrix_mdev->kvm) { kvm_arch_crypto_clear_masks(matrix_mdev->kvm); + vfio_ap_matrix_clear(&matrix_mdev->shadow_crycb); matrix_mdev->kvm->arch.crypto.pqap_hook = NULL; vfio_ap_mdev_reset_queues(mdev); kvm_put_kvm(matrix_mdev->kvm); From patchwork Tue Apr 7 19:20:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478927 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2AAF392C for ; Tue, 7 Apr 2020 19:21:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0984B2075E for ; Tue, 7 Apr 2020 19:21:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727435AbgDGTVY (ORCPT ); Tue, 7 Apr 2020 15:21:24 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:18294 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727226AbgDGTUi (ORCPT ); Tue, 7 Apr 2020 15:20:38 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J9Mv0100723; Tue, 7 Apr 2020 15:20:38 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 306mje1u4h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:38 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037JA0Vr102439; Tue, 7 Apr 2020 15:20:37 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 306mje1u42-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:37 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKVgC007987; Tue, 7 Apr 2020 19:20:36 GMT Received: from b01cxnp22033.gho.pok.ibm.com (b01cxnp22033.gho.pok.ibm.com [9.57.198.23]) by ppma01wdc.us.ibm.com with ESMTP id 306hv62fpr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:36 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKYJO10092974 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:35 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D9C872805A; Tue, 7 Apr 2020 19:20:34 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 282DC28058; Tue, 7 Apr 2020 19:20:34 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:34 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 08/15] s390/vfio_ap: add qlink from ap_matrix_mdev struct to vfio_ap_queue struct Date: Tue, 7 Apr 2020 15:20:08 -0400 Message-Id: <20200407192015.19887-9-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 mlxscore=0 priorityscore=1501 suspectscore=3 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Currently, a vfio_ap_queue struct is created for every queue device probed which is then added to a hash table of all queues probed by the vfio_ap device driver. This list could get quite large making retrieval expensive. In order to make retrieval of a vfio_ap_queue struct more efficient when we already have a pointer to the ap_matrix_mdev to which the queue's APQN is assigned, let's go ahead and add a link from the ap_matrix_mdev struct to the vfio_ap_queue struct. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 29 +++++++++++++++++++++++---- drivers/s390/crypto/vfio_ap_private.h | 2 ++ 2 files changed, 27 insertions(+), 4 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 25b7d978e3fd..6ee1ebe3f207 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -38,6 +38,19 @@ struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn) return NULL; } +struct vfio_ap_queue *vfio_ap_get_mdev_queue(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqn) +{ + struct vfio_ap_queue *q; + + hash_for_each_possible(matrix_mdev->qtable, q, mdev_qnode, apqn) { + if (q->apqn == apqn) + return q; + } + + return NULL; +} + /** * vfio_ap_wait_for_irqclear * @apqn: The AP Queue number @@ -325,6 +338,7 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) matrix_mdev->mdev = mdev; vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_crycb); + hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; matrix_mdev->pqap_hook.owner = THIS_MODULE; @@ -594,7 +608,7 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, * filter the APQI. */ apqn = AP_MKQID(apid, apqi); - if (!vfio_ap_get_queue(apqn)) { + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) { if (filter_apids) clear_bit_inv(apid, shadow_crycb->apm); else @@ -624,7 +638,6 @@ static bool vfio_ap_mdev_configure_crycb(struct ap_matrix_mdev *matrix_mdev) vfio_ap_matrix_init(&matrix_dev->info, &shadow_crycb); napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES); naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS); - /* * If there are no APIDs or no APQIs assigned to the matrix mdev, * then no APQNs shall be assigned to the guest CRYCB. @@ -636,6 +649,7 @@ static bool vfio_ap_mdev_configure_crycb(struct ap_matrix_mdev *matrix_mdev) */ napm = vfio_ap_mdev_filter_matrix(matrix_mdev, &shadow_crycb, true); + /* * If there are no APQNs that can be assigned to the guest's * CRYCB after filtering, then try filtering the APQIs. @@ -697,8 +711,10 @@ static void vfio_ap_mdev_qlinks_for_apid(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, matrix_mdev->matrix.aqm_max + 1) { q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); - if (q) + if (q) { q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, &q->mdev_qnode, q->apqn); + } } } @@ -871,8 +887,10 @@ static void vfio_ap_mdev_qlinks_for_apqi(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, matrix_mdev->matrix.apm_max + 1) { q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); - if (q) + if (q) { q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, &q->mdev_qnode, q->apqn); + } } } @@ -1536,6 +1554,7 @@ static void vfio_ap_mdev_for_queue(struct vfio_ap_queue *q) if (test_bit_inv(apid, matrix_mdev->matrix.apm) && test_bit_inv(apqi, matrix_mdev->matrix.aqm)) { q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, &q->mdev_qnode, q->apqn); break; } } @@ -1573,6 +1592,8 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) vfio_ap_mdev_reset_queue(apid, apqi, 1); vfio_ap_irq_disable(q); hash_del(&q->qnode); + if (q->matrix_mdev) + hash_del(&q->mdev_qnode); kfree(q); mutex_unlock(&matrix_dev->lock); } diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 87cc270c3212..794c60a767d2 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -89,6 +89,7 @@ struct ap_matrix_mdev { struct kvm *kvm; struct kvm_s390_module_hook pqap_hook; struct mdev_device *mdev; + DECLARE_HASHTABLE(qtable, 8); }; extern int vfio_ap_mdev_register(void); @@ -101,6 +102,7 @@ struct vfio_ap_queue { #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; struct hlist_node qnode; + struct hlist_node mdev_qnode; }; int vfio_ap_mdev_probe_queue(struct ap_queue *queue); From patchwork Tue Apr 7 19:20:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478917 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 099B092A for ; Tue, 7 Apr 2020 19:21:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D169E2075E for ; Tue, 7 Apr 2020 19:21:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727787AbgDGTU7 (ORCPT ); Tue, 7 Apr 2020 15:20:59 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:60168 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727447AbgDGTUn (ORCPT ); Tue, 7 Apr 2020 15:20:43 -0400 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J40cI043212; Tue, 7 Apr 2020 15:20:39 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nab2f3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:38 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J4JX5045292; Tue, 7 Apr 2020 15:20:38 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nab2er-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:38 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKUra011493; Tue, 7 Apr 2020 19:20:37 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma03dal.us.ibm.com with ESMTP id 306hv6s2d9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:37 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKZ8r14025540 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:35 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AE7AA28060; Tue, 7 Apr 2020 19:20:35 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EEF9C28058; Tue, 7 Apr 2020 19:20:34 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:34 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 09/15] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Date: Tue, 7 Apr 2020 15:20:09 -0400 Message-Id: <20200407192015.19887-10-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 priorityscore=1501 suspectscore=3 mlxscore=0 phishscore=0 spamscore=0 malwarescore=0 clxscore=1015 impostorscore=0 mlxlogscore=999 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The current implementation does not allow assignment of an AP adapter or domain to an mdev device if the APQNs resulting from the assignment do not reference AP queue devices that are bound to the vfio_ap device driver. This patch allows assignment of AP resources to the matrix mdev as long as the APQNs resulting from the assignment: 1. Are not reserved by the AP BUS for use by the zcrypt device drivers. 2. Are not assigned to another matrix mdev. The rationale behind this is twofold: 1. The AP architecture does not preclude assignment of APQNs to an AP configuration that are not available to the system. 2. APQNs that do not reference a queue device bound to the vfio_ap device driver will not be assigned to the guest's CRYCB, so the guest will not get access to queues not bound to the vfio_ap driver. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 213 ++++++------------------------ 1 file changed, 37 insertions(+), 176 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 6ee1ebe3f207..b5cfa3a967c0 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -409,122 +409,6 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = { NULL, }; -struct vfio_ap_queue_reserved { - unsigned long *apid; - unsigned long *apqi; - bool reserved; -}; - -/** - * vfio_ap_has_queue - * - * @dev: an AP queue device - * @data: a struct vfio_ap_queue_reserved reference - * - * Flags whether the AP queue device (@dev) has a queue ID containing the APQN, - * apid or apqi specified in @data: - * - * - If @data contains both an apid and apqi value, then @data will be flagged - * as reserved if the APID and APQI fields for the AP queue device matches - * - * - If @data contains only an apid value, @data will be flagged as - * reserved if the APID field in the AP queue device matches - * - * - If @data contains only an apqi value, @data will be flagged as - * reserved if the APQI field in the AP queue device matches - * - * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if - * @data does not contain either an apid or apqi. - */ -static int vfio_ap_has_queue(struct device *dev, void *data) -{ - struct vfio_ap_queue_reserved *qres = data; - struct ap_queue *ap_queue = to_ap_queue(dev); - ap_qid_t qid; - unsigned long id; - - if (qres->apid && qres->apqi) { - qid = AP_MKQID(*qres->apid, *qres->apqi); - if (qid == ap_queue->qid) - qres->reserved = true; - } else if (qres->apid && !qres->apqi) { - id = AP_QID_CARD(ap_queue->qid); - if (id == *qres->apid) - qres->reserved = true; - } else if (!qres->apid && qres->apqi) { - id = AP_QID_QUEUE(ap_queue->qid); - if (id == *qres->apqi) - qres->reserved = true; - } else { - return -EINVAL; - } - - return 0; -} - -/** - * vfio_ap_verify_queue_reserved - * - * @matrix_dev: a mediated matrix device - * @apid: an AP adapter ID - * @apqi: an AP queue index - * - * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device - * driver according to the following rules: - * - * - If both @apid and @apqi are not NULL, then there must be an AP queue - * device bound to the vfio_ap driver with the APQN identified by @apid and - * @apqi - * - * - If only @apid is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apid - * - * - If only @apqi is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apqi - * - * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL. - */ -static int vfio_ap_verify_queue_reserved(unsigned long *apid, - unsigned long *apqi) -{ - int ret; - struct vfio_ap_queue_reserved qres; - - qres.apid = apid; - qres.apqi = apqi; - qres.reserved = false; - - ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &qres, vfio_ap_has_queue); - if (ret) - return ret; - - if (qres.reserved) - return 0; - - return -EADDRNOTAVAIL; -} - -static int -vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, - unsigned long apid) -{ - int ret; - unsigned long apqi; - unsigned long nbits = matrix_mdev->matrix.aqm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(&apid, NULL); - - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - /** * vfio_ap_mdev_verify_no_sharing * @@ -533,6 +417,8 @@ vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, * mediated device. AP queue sharing is not allowed. * * @matrix_mdev: the mediated matrix device + * @mdev_apm: the mask identifying the adapters assigned to mdev + * @mdev_aqm: the mask identifying the domains (queues) assigned to mdev * * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE. */ @@ -545,6 +431,11 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, DECLARE_BITMAP(aqm, AP_DOMAINS); list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) { + /* + * If either of the input masks belongs to the mdev to which an + * AP resource is being assigned, then we don't need to verify + * that mdev's masks. + */ if (matrix_mdev == lstdev) continue; @@ -567,6 +458,20 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, return 0; } +static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev, + unsigned long *mdev_apm, + unsigned long *mdev_aqm) +{ + DECLARE_BITMAP(apm, AP_DEVICES); + DECLARE_BITMAP(aqm, AP_DOMAINS); + + if (bitmap_and(apm, mdev_apm, ap_perms.apm, AP_DEVICES) && + bitmap_and(aqm, mdev_aqm, ap_perms.aqm, AP_DOMAINS)) + return -EADDRNOTAVAIL; + + return vfio_ap_mdev_verify_no_sharing(matrix_mdev, mdev_apm, mdev_aqm); +} + static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, struct ap_matrix *shadow_crycb, bool filter_apids) @@ -771,33 +676,21 @@ static ssize_t assign_adapter_store(struct device *dev, if (apid > matrix_mdev->matrix.apm_max) return -ENODEV; - /* - * Set the bit in the AP mask (APM) corresponding to the AP adapter - * number (APID). The bits in the mask, from most significant to least - * significant bit, correspond to APIDs 0-255. - */ - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid); - if (ret) - goto done; - memset(apm, 0, sizeof(apm)); set_bit_inv(apid, apm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm, - matrix_mdev->matrix.aqm); - if (ret) - goto done; - + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev, apm, + matrix_mdev->matrix.aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_qlinks_for_apid(matrix_mdev, apid); - ret = count; - -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_adapter); @@ -847,26 +740,6 @@ static ssize_t unassign_adapter_store(struct device *dev, } static DEVICE_ATTR_WO(unassign_adapter); -static int -vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev, - unsigned long apqi) -{ - int ret; - unsigned long apid; - unsigned long nbits = matrix_mdev->matrix.apm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(NULL, &apqi); - - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - /** * vfio_ap_mdev_qlinks_for_apqi * @@ -947,28 +820,21 @@ static ssize_t assign_domain_store(struct device *dev, if (apqi > max_apqi) return -ENODEV; - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi); - if (ret) - goto done; - memset(aqm, 0, sizeof(aqm)); set_bit_inv(apqi, aqm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, - matrix_mdev->matrix.apm, aqm); - if (ret) - goto done; - + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev, matrix_mdev->matrix.apm, + aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_qlinks_for_apqi(matrix_mdev, apqi); - ret = count; - -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_domain); @@ -1055,11 +921,6 @@ static ssize_t assign_control_domain_store(struct device *dev, if (id > matrix_mdev->matrix.adm_max) return -ENODEV; - /* Set the bit in the ADM (bitmask) corresponding to the AP control - * domain number (id). The bits in the mask, from most significant to - * least significant, correspond to IDs 0 up to the one less than the - * number of control domains that can be assigned. - */ mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); mutex_unlock(&matrix_dev->lock); From patchwork Tue Apr 7 19:20:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478911 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 109CA92A for ; Tue, 7 Apr 2020 19:20:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E260D2076E for ; Tue, 7 Apr 2020 19:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727461AbgDGTUn (ORCPT ); Tue, 7 Apr 2020 15:20:43 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:63168 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727307AbgDGTUk (ORCPT ); Tue, 7 Apr 2020 15:20:40 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J4GTX021051; Tue, 7 Apr 2020 15:20:39 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082k38rxn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:39 -0400 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J4Mfn021745; Tue, 7 Apr 2020 15:20:39 -0400 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082k38rx8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:39 -0400 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKCr9013708; Tue, 7 Apr 2020 19:20:38 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma04wdc.us.ibm.com with ESMTP id 306hv6afgs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:38 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKatO49152352 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:36 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7C39428058; Tue, 7 Apr 2020 19:20:36 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C5E4A28065; Tue, 7 Apr 2020 19:20:35 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:35 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 10/15] s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest Date: Tue, 7 Apr 2020 15:20:10 -0400 Message-Id: <20200407192015.19887-11-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=3 adultscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 clxscore=1015 mlxscore=0 phishscore=0 mlxlogscore=999 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The current support for pass-through crypto adapters does not allow configuration of a matrix mdev when it is in use by a KVM guest. Let's allow AP resources - i.e., adapters, domains and control domains - to be assigned to or unassigned from a matrix mdev while it is in use by a guest. This is in preparation for the introduction of support for dynamic configuration of the AP matrix for a running KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index b5cfa3a967c0..4b16d45b702b 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -665,10 +665,6 @@ static ssize_t assign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -720,10 +716,6 @@ static ssize_t unassign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -810,10 +802,6 @@ static ssize_t assign_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; - /* If the guest is running, disallow assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -865,10 +853,6 @@ static ssize_t unassign_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -910,10 +894,6 @@ static ssize_t assign_control_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &id); if (ret) return ret; @@ -955,10 +935,6 @@ static ssize_t unassign_control_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_domid = matrix_mdev->matrix.adm_max; - /* If the guest is running, disallow un-assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &domid); if (ret) return ret; From patchwork Tue Apr 7 19:20:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478921 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFA8892A for ; Tue, 7 Apr 2020 19:21:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C360C2075E for ; Tue, 7 Apr 2020 19:21:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727901AbgDGTVL (ORCPT ); Tue, 7 Apr 2020 15:21:11 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:38892 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727387AbgDGTUm (ORCPT ); Tue, 7 Apr 2020 15:20:42 -0400 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J7gFD062995; Tue, 7 Apr 2020 15:20:40 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 308eu8m7by-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:40 -0400 Received: from m0098414.ppops.net (m0098414.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J8Smv064685; Tue, 7 Apr 2020 15:20:39 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0b-001b2d01.pphosted.com with ESMTP id 308eu8m7bt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:39 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKF9r027670; Tue, 7 Apr 2020 19:20:39 GMT Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by ppma03wdc.us.ibm.com with ESMTP id 306hv6ae5b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:39 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKb0U52953408 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:37 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4E4E128060; Tue, 7 Apr 2020 19:20:37 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 912182805A; Tue, 7 Apr 2020 19:20:36 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:36 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 11/15] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Date: Tue, 7 Apr 2020 15:20:11 -0400 Message-Id: <20200407192015.19887-12-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 malwarescore=0 clxscore=1015 priorityscore=1501 impostorscore=0 spamscore=0 mlxlogscore=999 suspectscore=3 lowpriorityscore=0 adultscore=0 mlxscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's hot plug/unplug adapters, domains and control domains assigned to or unassigned from an AP matrix mdev device while it is in use by a guest per the following: * When the APID of an adapter is assigned to a matrix mdev in use by a KVM guest, the adapter will be hot plugged into the KVM guest as long as each APQN derived from the Cartesian product of the APID being assigned and the APQIs already assigned to the guest's CRYCB references a queue device bound to the vfio_ap device driver. * When the APID of an adapter is unassigned from a matrix mdev in use by a KVM guest, the adapter will be hot unplugged from the KVM guest. * When the APQI of a domain is assigned to a matrix mdev in use by a KVM guest, the domain will be hot plugged into the KVM guest as long as each APQN derived from the Cartesian product of the APQI being assigned and the APIDs already assigned to the guest's CRYCB references a queue device bound to the vfio_ap device driver. * When the APQI of a domain is unassigned from a matrix mdev in use by a KVM guest, the domain will be hot unplugged from the KVM guest * When the domain number of a control domain is assigned to a matrix mdev in use by a KVM guest, the control domain will be hot plugged into the KVM guest. * When the domain number of a control domain is unassigned from a matrix mdev in use by a KVM guest, the control domain will be hot unplugged from the KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 198 ++++++++++++++++++++++++++++++ 1 file changed, 198 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 4b16d45b702b..88a4aef5193f 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -623,6 +623,56 @@ static void vfio_ap_mdev_qlinks_for_apid(struct ap_matrix_mdev *matrix_mdev, } } +static bool vfio_ap_mdev_assign_apqis_4_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + DECLARE_BITMAP(aqm, AP_DOMAINS); + unsigned long apqi, apqn; + + bitmap_copy(aqm, matrix_mdev->matrix.aqm, AP_DOMAINS); + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { + if (!test_bit_inv(apqi, + (unsigned long *) matrix_dev->info.aqm)) + clear_bit_inv(apqi, aqm); + + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + clear_bit_inv(apqi, aqm); + } + + if (bitmap_empty(aqm, AP_DOMAINS)) + return false; + + set_bit_inv(apid, matrix_mdev->shadow_crycb.apm); + bitmap_copy(matrix_mdev->shadow_crycb.aqm, aqm, AP_DOMAINS); + + return true; +} + +static bool vfio_ap_mdev_assign_guest_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + unsigned long apqi, apqn; + + if (!vfio_ap_mdev_has_crycb(matrix_mdev) || + !test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm)) + return false; + + if (bitmap_empty(matrix_mdev->shadow_crycb.aqm, AP_DOMAINS)) + return vfio_ap_mdev_assign_apqis_4_apid(matrix_mdev, apid); + + for_each_set_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm, AP_DOMAINS) { + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + return false; + } + + set_bit_inv(apid, matrix_mdev->shadow_crycb.apm); + + return true; +} + /** * assign_adapter_store * @@ -684,12 +734,44 @@ static ssize_t assign_adapter_store(struct device *dev, } set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_qlinks_for_apid(matrix_mdev, apid); + + if (vfio_ap_mdev_assign_guest_apid(matrix_mdev, apid)) + vfio_ap_mdev_commit_crycb(matrix_mdev); + mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_adapter); +static bool vfio_ap_mdev_unassign_guest_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(apid, matrix_mdev->shadow_crycb.apm)) { + clear_bit_inv(apid, matrix_mdev->shadow_crycb.apm); + + /* + * If there are no APIDs assigned to the guest, then + * the guest will not have access to any queues, so + * let's also go ahead and unassign the APQIs. Keeping + * them around may yield unpredictable results during + * a probe that is not related to a host AP + * configuration change (i.e., an AP adapter is + * configured online). + */ + if (bitmap_empty(matrix_mdev->shadow_crycb.apm, + AP_DEVICES)) + bitmap_clear(matrix_mdev->shadow_crycb.aqm, 0, + AP_DOMAINS); + + return true; + } + } + + return false; +} + /** * unassign_adapter_store * @@ -726,6 +808,8 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); vfio_ap_mdev_qlinks_for_apid(NULL, apid); + if (vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid)) + vfio_ap_mdev_commit_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -759,6 +843,56 @@ static void vfio_ap_mdev_qlinks_for_apqi(struct ap_matrix_mdev *matrix_mdev, } } +static bool vfio_ap_mdev_assign_apids_4_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + DECLARE_BITMAP(apm, AP_DEVICES); + unsigned long apid, apqn; + + bitmap_copy(apm, matrix_mdev->matrix.apm, AP_DEVICES); + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { + if (!test_bit_inv(apid, + (unsigned long *) matrix_dev->info.apm)) + clear_bit_inv(apqi, apm); + + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + clear_bit_inv(apid, apm); + } + + if (bitmap_empty(apm, AP_DEVICES)) + return false; + + set_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm); + bitmap_copy(matrix_mdev->shadow_crycb.apm, apm, AP_DEVICES); + + return true; +} + +static bool vfio_ap_mdev_assign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + unsigned long apid, apqn; + + if (!vfio_ap_mdev_has_crycb(matrix_mdev) || + !test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) + return false; + + if (bitmap_empty(matrix_mdev->shadow_crycb.apm, AP_DEVICES)) + return vfio_ap_mdev_assign_apids_4_apqi(matrix_mdev, apqi); + + for_each_set_bit_inv(apid, matrix_mdev->shadow_crycb.apm, AP_DEVICES) { + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + return false; + } + + set_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm); + + return true; +} + /** * assign_domain_store * @@ -820,12 +954,41 @@ static ssize_t assign_domain_store(struct device *dev, } set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_qlinks_for_apqi(matrix_mdev, apqi); + if (vfio_ap_mdev_assign_guest_apqi(matrix_mdev, apqi)) + vfio_ap_mdev_commit_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_domain); +static bool vfio_ap_mdev_unassign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm)) { + clear_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm); + + /* + * If there are no APQIs assigned to the guest, then + * the guest will not have access to any queues, so + * let's also go ahead and unassign the APIDs. Keeping + * them around may yield unpredictable results during + * a probe that is not related to a host AP + * configuration change (i.e., an AP adapter is + * configured online). + */ + if (bitmap_empty(matrix_mdev->shadow_crycb.aqm, + AP_DOMAINS)) + bitmap_clear(matrix_mdev->shadow_crycb.apm, 0, + AP_DEVICES); + + return true; + } + } + + return false; +} /** * unassign_domain_store @@ -863,12 +1026,28 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_qlinks_for_apqi(NULL, apqi); + if (vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi)) + vfio_ap_mdev_commit_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(unassign_domain); +static bool vfio_ap_mdev_assign_guest_cdom(struct ap_matrix_mdev *matrix_mdev, + unsigned long domid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (!test_bit_inv(domid, matrix_mdev->shadow_crycb.adm)) { + set_bit_inv(domid, matrix_mdev->shadow_crycb.adm); + + return true; + } + } + + return false; +} + /** * assign_control_domain_store * @@ -903,12 +1082,29 @@ static ssize_t assign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); + if (vfio_ap_mdev_assign_guest_cdom(matrix_mdev, id)) + vfio_ap_mdev_commit_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_control_domain); +static bool +vfio_ap_mdev_unassign_guest_cdom(struct ap_matrix_mdev *matrix_mdev, + unsigned long domid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(domid, matrix_mdev->shadow_crycb.adm)) { + clear_bit_inv(domid, matrix_mdev->shadow_crycb.adm); + + return true; + } + } + + return false; +} + /** * unassign_control_domain_store * @@ -943,6 +1139,8 @@ static ssize_t unassign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv(domid, matrix_mdev->matrix.adm); + if (vfio_ap_mdev_unassign_guest_cdom(matrix_mdev, domid)) + vfio_ap_mdev_commit_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; From patchwork Tue Apr 7 19:20:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478923 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A933692C for ; Tue, 7 Apr 2020 19:21:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7D79F20768 for ; Tue, 7 Apr 2020 19:21:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727881AbgDGTVK (ORCPT ); Tue, 7 Apr 2020 15:21:10 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:57604 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727192AbgDGTUm (ORCPT ); Tue, 7 Apr 2020 15:20:42 -0400 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J3WpH137021; Tue, 7 Apr 2020 15:20:41 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082hxuv3j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:41 -0400 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J3cNx137786; Tue, 7 Apr 2020 15:20:41 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082hxuv30-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:40 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKDl9027655; Tue, 7 Apr 2020 19:20:39 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma03wdc.us.ibm.com with ESMTP id 306hv6ae5d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:39 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKciS53149974 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:38 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0A4852806A; Tue, 7 Apr 2020 19:20:38 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6346728059; Tue, 7 Apr 2020 19:20:37 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:37 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com Subject: [PATCH v7 12/15] s390/zcrypt: Notify driver on config changed and scan complete callbacks Date: Tue, 7 Apr 2020 15:20:12 -0400 Message-Id: <20200407192015.19887-13-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 spamscore=0 clxscore=1015 adultscore=0 bulkscore=0 priorityscore=1501 suspectscore=3 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070153 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Harald Freudenberger This patch intruduces an extension to the ap bus to notify drivers on crypto config changed and bus scan complete events. Two new callbacks are introduced for ap_drivers: void (*on_config_changed)(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); void (*on_scan_complete)(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); Both callbacks are optional. Both callbacks are only triggered when QCI information is available (facility bit 12): * The on_config_changed callback is invoked at the start of the AP bus scan function when it determines that the host AP configuration information has changed since the previous scan. This is done by storing an old and current QCI info struct and comparing them. If there is any difference, the callback is invoked. Note that when the AP bus scan detects that AP adapters or domains have been removed from the host's AP configuration, it will remove the associated devices from the AP bus subsystem's device model. This callback gives the device driver a chance to respond to the removal of the AP devices in bulk rather than one at a time as its remove callback is invoked. It will also allow the device driver to do any any cleanup prior to giving control back to the bus piecemeal. This is particularly important for the vfio_ap driver because there may be guests using the queues at the time. * The on_scan_complete callback is invoked after the ap bus scan is complete if the host AP configuration data has changed. Note that when the AP bus scan detects that adapters or domains have been added to the host's configuration, it will create new devices in the AP bus subsystem's device model. This callback also allows the driver to process all of the new devices in bulk. Please note that changes to the apmask and aqmask do not trigger these two callbacks since the bus scan function is not invoked by changes to those masks. Signed-off-by: Harald Freudenberger --- drivers/s390/crypto/ap_bus.c | 157 +++++++++++++++++++++++++++-------- drivers/s390/crypto/ap_bus.h | 13 +++ 2 files changed, 134 insertions(+), 36 deletions(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index af15c095e76a..9b3fc4ea9c4d 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -71,8 +71,12 @@ struct ap_perms ap_perms; EXPORT_SYMBOL(ap_perms); DEFINE_MUTEX(ap_perms_mutex); EXPORT_SYMBOL(ap_perms_mutex); +DEFINE_MUTEX(ap_config_lock); + +/* current and old qci info structs */ +static struct ap_config_info *ap_config_info; +static struct ap_config_info *ap_old_config_info; -static struct ap_config_info *ap_configuration; static bool initialised; /* @@ -188,8 +192,8 @@ static int ap_apft_available(void) */ static inline int ap_qact_available(void) { - if (ap_configuration) - return ap_configuration->qact; + if (ap_config_info) + return ap_config_info->qact; return 0; } @@ -218,13 +222,15 @@ static void ap_init_configuration(void) if (!ap_configuration_available()) return; - ap_configuration = kzalloc(sizeof(*ap_configuration), GFP_KERNEL); - if (!ap_configuration) - return; - if (ap_query_configuration(ap_configuration) != 0) { - kfree(ap_configuration); - ap_configuration = NULL; + /* allocate current qci info struct */ + ap_config_info = kzalloc(sizeof(*ap_config_info), GFP_KERNEL); + if (!ap_config_info) return; + + /* fetch qci info into the current qci info struct */ + if (ap_query_configuration(ap_config_info)) { + kfree(ap_config_info); + ap_config_info = NULL; } } @@ -247,10 +253,10 @@ static inline int ap_test_config(unsigned int *field, unsigned int nr) */ static inline int ap_test_config_card_id(unsigned int id) { - if (!ap_configuration) /* QCI not supported */ - /* only ids 0...3F may be probed */ + if (!ap_config_info) + /* QCI not available, only ids 0...3F may be probed */ return id < 0x40 ? 1 : 0; - return ap_test_config(ap_configuration->apm, id); + return ap_test_config(ap_config_info->apm, id); } /* @@ -264,9 +270,9 @@ static inline int ap_test_config_card_id(unsigned int id) */ int ap_test_config_usage_domain(unsigned int domain) { - if (!ap_configuration) /* QCI not supported */ + if (!ap_config_info) /* QCI not supported */ return domain < 16; - return ap_test_config(ap_configuration->aqm, domain); + return ap_test_config(ap_config_info->aqm, domain); } EXPORT_SYMBOL(ap_test_config_usage_domain); @@ -280,9 +286,9 @@ EXPORT_SYMBOL(ap_test_config_usage_domain); */ int ap_test_config_ctrl_domain(unsigned int domain) { - if (!ap_configuration) /* QCI not supported */ + if (!ap_config_info) /* QCI not supported */ return 0; - return ap_test_config(ap_configuration->adm, domain); + return ap_test_config(ap_config_info->adm, domain); } EXPORT_SYMBOL(ap_test_config_ctrl_domain); @@ -1052,45 +1058,45 @@ static BUS_ATTR_RW(ap_domain); static ssize_t ap_control_domain_mask_show(struct bus_type *bus, char *buf) { - if (!ap_configuration) /* QCI not supported */ + if (!ap_config_info) /* QCI not supported */ return snprintf(buf, PAGE_SIZE, "not supported\n"); return snprintf(buf, PAGE_SIZE, "0x%08x%08x%08x%08x%08x%08x%08x%08x\n", - ap_configuration->adm[0], ap_configuration->adm[1], - ap_configuration->adm[2], ap_configuration->adm[3], - ap_configuration->adm[4], ap_configuration->adm[5], - ap_configuration->adm[6], ap_configuration->adm[7]); + ap_config_info->adm[0], ap_config_info->adm[1], + ap_config_info->adm[2], ap_config_info->adm[3], + ap_config_info->adm[4], ap_config_info->adm[5], + ap_config_info->adm[6], ap_config_info->adm[7]); } static BUS_ATTR_RO(ap_control_domain_mask); static ssize_t ap_usage_domain_mask_show(struct bus_type *bus, char *buf) { - if (!ap_configuration) /* QCI not supported */ + if (!ap_config_info) /* QCI not supported */ return snprintf(buf, PAGE_SIZE, "not supported\n"); return snprintf(buf, PAGE_SIZE, "0x%08x%08x%08x%08x%08x%08x%08x%08x\n", - ap_configuration->aqm[0], ap_configuration->aqm[1], - ap_configuration->aqm[2], ap_configuration->aqm[3], - ap_configuration->aqm[4], ap_configuration->aqm[5], - ap_configuration->aqm[6], ap_configuration->aqm[7]); + ap_config_info->aqm[0], ap_config_info->aqm[1], + ap_config_info->aqm[2], ap_config_info->aqm[3], + ap_config_info->aqm[4], ap_config_info->aqm[5], + ap_config_info->aqm[6], ap_config_info->aqm[7]); } static BUS_ATTR_RO(ap_usage_domain_mask); static ssize_t ap_adapter_mask_show(struct bus_type *bus, char *buf) { - if (!ap_configuration) /* QCI not supported */ + if (!ap_config_info) /* QCI not supported */ return snprintf(buf, PAGE_SIZE, "not supported\n"); return snprintf(buf, PAGE_SIZE, "0x%08x%08x%08x%08x%08x%08x%08x%08x\n", - ap_configuration->apm[0], ap_configuration->apm[1], - ap_configuration->apm[2], ap_configuration->apm[3], - ap_configuration->apm[4], ap_configuration->apm[5], - ap_configuration->apm[6], ap_configuration->apm[7]); + ap_config_info->apm[0], ap_config_info->apm[1], + ap_config_info->apm[2], ap_config_info->apm[3], + ap_config_info->apm[4], ap_config_info->apm[5], + ap_config_info->apm[6], ap_config_info->apm[7]); } static BUS_ATTR_RO(ap_adapter_mask); @@ -1178,7 +1184,7 @@ static ssize_t ap_max_domain_id_show(struct bus_type *bus, char *buf) { int max_domain_id; - if (ap_configuration) + if (ap_config_info) max_domain_id = ap_max_domain_id ? : -1; else max_domain_id = 15; @@ -1481,6 +1487,50 @@ static int ap_get_compatible_type(ap_qid_t qid, int rawtype, unsigned int func) return comp_type; } +/* Helper function for notify_config_changed */ +static int __drv_notify_config_changed(struct device_driver *drv, void *data) +{ + struct ap_driver *ap_drv = to_ap_drv(drv); + + if (try_module_get(drv->owner)) { + if (ap_drv->on_config_changed) + ap_drv->on_config_changed(ap_config_info, + ap_old_config_info); + module_put(drv->owner); + } + + return 0; +} + +/* Notify all drivers about an qci config change */ +static inline void notify_config_changed(void) +{ + bus_for_each_drv(&ap_bus_type, NULL, NULL, + __drv_notify_config_changed); +} + +/* Helper function for notify_scan_complete */ +static int __drv_notify_scan_complete(struct device_driver *drv, void *data) +{ + struct ap_driver *ap_drv = to_ap_drv(drv); + + if (try_module_get(drv->owner)) { + if (ap_drv->on_scan_complete) + ap_drv->on_scan_complete(ap_config_info, + ap_old_config_info); + module_put(drv->owner); + } + + return 0; +} + +/* Notify all drivers about bus scan complete */ +static inline void notify_scan_complete(void) +{ + bus_for_each_drv(&ap_bus_type, NULL, NULL, + __drv_notify_scan_complete); +} + /* * Helper function to be used with bus_find_dev * matches for the card device with the given id @@ -1663,23 +1713,57 @@ static void _ap_scan_bus_adapter(int id) put_device(&ac->ap_dev.device); } +static int ap_config_changed(void) +{ + int cfg_chg = 0; + + if (ap_config_info) { + if (!ap_old_config_info) { + ap_old_config_info = kzalloc( + sizeof(*ap_old_config_info), GFP_KERNEL); + if (!ap_old_config_info) + return 0; + } else { + memcpy(ap_old_config_info, ap_config_info, + sizeof(struct ap_config_info)); + } + ap_query_configuration(ap_config_info); + cfg_chg = memcmp(ap_config_info, + ap_old_config_info, + sizeof(struct ap_config_info)) != 0; + } + + return cfg_chg; +} + /** * ap_scan_bus(): Scan the AP bus for new devices * Runs periodically, workqueue timer (ap_config_time) */ static void ap_scan_bus(struct work_struct *unused) { - int id; + int id, config_changed = 0; AP_DBF(DBF_DEBUG, "%s running\n", __func__); - ap_query_configuration(ap_configuration); + mutex_lock(&ap_config_lock); + + /* config change notify */ + config_changed = ap_config_changed(); + if (config_changed) + notify_config_changed(); ap_select_domain(); /* loop over all possible adapters */ for (id = 0; id < AP_DEVICES; id++) _ap_scan_bus_adapter(id); + /* scan complete notify */ + if (config_changed) + notify_scan_complete(); + + mutex_unlock(&ap_config_lock); + /* check if there is at least one queue available with default domain */ if (ap_domain_index >= 0) { struct device *dev = @@ -1761,7 +1845,7 @@ static int __init ap_module_init(void) /* Get AP configuration data if available */ ap_init_configuration(); - if (ap_configuration) + if (ap_config_info) max_domain_id = ap_max_domain_id ? ap_max_domain_id : AP_DOMAINS - 1; else @@ -1841,7 +1925,8 @@ static int __init ap_module_init(void) out: if (ap_using_interrupts()) unregister_adapter_interrupt(&ap_airq); - kfree(ap_configuration); + kfree(ap_config_info); + kfree(ap_old_config_info); return rc; } device_initcall(ap_module_init); diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 86bf5e224ba4..c94c97c6ea09 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -139,6 +139,19 @@ struct ap_driver { void (*suspend)(struct ap_device *); void (*resume)(struct ap_device *); bool (*in_use)(unsigned long *apm, unsigned long *aqm); + + /* + * Called at the start of the ap bus scan function when + * the crypto config information (qci) has changed. + */ + void (*on_config_changed)(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); + /* + * Called at the end of the ap bus scan function when + * the crypto config information (qci) has changed. + */ + void (*on_scan_complete)(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) From patchwork Tue Apr 7 19:20:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478919 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55FB492A for ; Tue, 7 Apr 2020 19:21:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 27A3C20768 for ; Tue, 7 Apr 2020 19:21:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727817AbgDGTVA (ORCPT ); Tue, 7 Apr 2020 15:21:00 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:63668 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727435AbgDGTUn (ORCPT ); Tue, 7 Apr 2020 15:20:43 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J50C3000740; Tue, 7 Apr 2020 15:20:42 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082pes0ku-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:42 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J5A4R002062; Tue, 7 Apr 2020 15:20:41 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082pes0kf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:41 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKaWc027396; Tue, 7 Apr 2020 19:20:40 GMT Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by ppma02dal.us.ibm.com with ESMTP id 306hv71392-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:40 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKcfU53936562 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:38 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D3A972805A; Tue, 7 Apr 2020 19:20:38 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 28FE628058; Tue, 7 Apr 2020 19:20:38 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:38 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 13/15] s390/vfio-ap: handle host AP config change notification Date: Tue, 7 Apr 2020 15:20:13 -0400 Message-Id: <20200407192015.19887-14-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 mlxscore=0 clxscore=1015 spamscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 adultscore=0 mlxlogscore=999 suspectscore=3 bulkscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Implements the driver callback invoked by the AP bus when the host AP configuration has changed. Since this callback is invoked prior to unbinding a device from its device driver, the vfio_ap driver will respond by unplugging the AP adapters, domains and control domains removed from the host's AP configuration from the guests using them. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 5 +- drivers/s390/crypto/vfio_ap_ops.c | 136 ++++++++++++++++++++++++-- drivers/s390/crypto/vfio_ap_private.h | 7 +- 3 files changed, 140 insertions(+), 8 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 5197a1fe14d4..9f6c5d82dfb5 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -113,9 +113,11 @@ static int vfio_ap_matrix_dev_create(void) /* Fill in config info via PQAP(QCI), if available */ if (test_facility(12)) { - ret = ap_qci(&matrix_dev->info); + ret = ap_qci(&matrix_dev->config_info); if (ret) goto matrix_alloc_err; + memcpy(&matrix_dev->config_info_prev, &matrix_dev->config_info, + sizeof(struct ap_config_info)); } mutex_init(&matrix_dev->lock); @@ -176,6 +178,7 @@ static int __init vfio_ap_init(void) vfio_ap_drv.remove = vfio_ap_queue_dev_remove; vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; + vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 88a4aef5193f..f1dd77729dd9 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -336,8 +336,9 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) } matrix_mdev->mdev = mdev; - vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); - vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_crycb); + vfio_ap_matrix_init(&matrix_dev->config_info, &matrix_mdev->matrix); + vfio_ap_matrix_init(&matrix_dev->config_info, + &matrix_mdev->shadow_crycb); hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; @@ -485,8 +486,8 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, * If the APID is not assigned to the host AP configuration, * we can not assign it to the guest's AP configuration */ - if (!test_bit_inv(apid, - (unsigned long *)matrix_dev->info.apm)) { + if (!test_bit_inv(apid, (unsigned long *) + matrix_dev->config_info.apm)) { clear_bit_inv(apid, shadow_crycb->apm); continue; } @@ -499,7 +500,7 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, * guest's AP configuration */ if (!test_bit_inv(apqi, (unsigned long *) - matrix_dev->info.aqm)) { + matrix_dev->config_info.aqm)) { clear_bit_inv(apqi, shadow_crycb->aqm); continue; } @@ -1617,7 +1618,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue) void vfio_ap_mdev_remove_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; - int apid, apqi; + unsigned long apid, apqi; mutex_lock(&matrix_dev->lock); q = dev_get_drvdata(&queue->ap_dev.device); @@ -1643,3 +1644,126 @@ bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) return in_use; } + +/** + * vfio_ap_mdev_unassign_apids + * + * @matrix_mdev: The matrix mediated device + * + * @aqm: A bitmap with 256 bits. Each bit in the map represents an APID from 0 + * to 255 (with the leftmost bit corresponding to APID 0). + * + * Unassigns each APID specified in @aqm that is assigned to the shadow CRYCB + * of @matrix_mdev. Returns true if at least one APID is unassigned; otherwise, + * returns false. + */ +bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apm_unassign) +{ + unsigned long apid; + bool unassigned = false; + + /* + * If the matrix mdev is not in use by a KVM guest, return indicating + * that no APIDs have been unassigned. + */ + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return false; + + for_each_set_bit_inv(apid, apm_unassign, AP_DEVICES) { + unassigned |= vfio_ap_mdev_unassign_guest_apid(matrix_mdev, + apid); + } + + return unassigned; +} + +/** + * vfio_ap_mdev_unassign_apqis + * + * @matrix_mdev: The matrix mediated device + * + * @aqm: A bitmap with 256 bits. Each bit in the map represents an APQI from 0 + * to 255 (with the leftmost bit corresponding to APQI 0). + * + * Unassigns each APQI specified in @aqm that is assigned to the shadow CRYCB + * of @matrix_mdev. Returns true if at least one APQI is unassigned; otherwise, + * returns false. + */ +bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev, + unsigned long *aqm_unassign) +{ + unsigned long apqi; + bool unassigned = false; + + /* + * If the matrix mdev is not in use by a KVM guest, return indicating + * that no APQIs have been unassigned. + */ + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return false; + + for_each_set_bit_inv(apqi, aqm_unassign, AP_DOMAINS) { + unassigned |= vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, + apqi); + } + + return unassigned; +} + +void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info) +{ + bool unassigned; + int ap_remove, aq_remove; + struct ap_matrix_mdev *matrix_mdev; + DECLARE_BITMAP(apm_unassign, AP_DEVICES); + DECLARE_BITMAP(aqm_unassign, AP_DOMAINS); + + unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm; + + if (matrix_dev->flags & AP_MATRIX_CFG_CHG) { + WARN_ONCE(1, "AP host configuration change already reported"); + return; + } + + memcpy(&matrix_dev->config_info, new_config_info, + sizeof(struct ap_config_info)); + memcpy(&matrix_dev->config_info_prev, old_config_info, + sizeof(struct ap_config_info)); + + cur_apm = (unsigned long *)matrix_dev->config_info.apm; + cur_aqm = (unsigned long *)matrix_dev->config_info.aqm; + prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm; + prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm; + + ap_remove = bitmap_andnot(apm_unassign, prev_apm, cur_apm, AP_DEVICES); + aq_remove = bitmap_andnot(aqm_unassign, prev_aqm, cur_aqm, AP_DOMAINS); + + mutex_lock(&matrix_dev->lock); + matrix_dev->flags |= AP_MATRIX_CFG_CHG; + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + continue; + + unassigned = false; + + if (ap_remove) + if (bitmap_intersects(matrix_mdev->shadow_crycb.apm, + apm_unassign, AP_DEVICES)) + if (vfio_ap_mdev_unassign_apids(matrix_mdev, + apm_unassign)) + unassigned = true; + if (aq_remove) + if (bitmap_intersects(matrix_mdev->shadow_crycb.aqm, + aqm_unassign, AP_DOMAINS)) + if (vfio_ap_mdev_unassign_apqis(matrix_mdev, + aqm_unassign)) + unassigned = true; + + if (unassigned) + vfio_ap_mdev_commit_crycb(matrix_mdev); + } + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 794c60a767d2..82abbf03781f 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -40,11 +40,14 @@ struct ap_matrix_dev { struct device device; atomic_t available_instances; - struct ap_config_info info; + struct ap_config_info config_info; + struct ap_config_info config_info_prev; struct list_head mdev_list; struct mutex lock; struct ap_driver *vfio_ap_drv; DECLARE_HASHTABLE(qtable, 8); + #define AP_MATRIX_CFG_CHG (1UL << 0) + unsigned long flags; }; extern struct ap_matrix_dev *matrix_dev; @@ -109,5 +112,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue); void vfio_ap_mdev_remove_queue(struct ap_queue *queue); bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); +void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Tue Apr 7 19:20:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478913 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 082D192C for ; Tue, 7 Apr 2020 19:20:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CF46420768 for ; Tue, 7 Apr 2020 19:20:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727548AbgDGTUp (ORCPT ); Tue, 7 Apr 2020 15:20:45 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:55718 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727302AbgDGTUo (ORCPT ); Tue, 7 Apr 2020 15:20:44 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J38KU034732; Tue, 7 Apr 2020 15:20:42 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx343s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:42 -0400 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J3M3E036151; Tue, 7 Apr 2020 15:20:42 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3082nx343e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:42 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKV8l011500; Tue, 7 Apr 2020 19:20:41 GMT Received: from b01cxnp22034.gho.pok.ibm.com (b01cxnp22034.gho.pok.ibm.com [9.57.198.24]) by ppma03dal.us.ibm.com with ESMTP id 306hv6s2dt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:41 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKdAd39256362 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:39 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A520428059; Tue, 7 Apr 2020 19:20:39 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E8B3B28058; Tue, 7 Apr 2020 19:20:38 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:38 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 14/15] s390/vfio-ap: handle AP bus scan completed notification Date: Tue, 7 Apr 2020 15:20:14 -0400 Message-Id: <20200407192015.19887-15-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 phishscore=0 impostorscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 lowpriorityscore=0 suspectscore=3 adultscore=0 clxscore=1015 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070153 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Implements the driver callback invoked by the AP bus when the AP bus scan has completed. Since this callback is invoked after binding the newly added devices to their respective device drivers, the vfio_ap driver will attempt to plug the adapters, domains and control domains into each guest using a matrix mdev to which they are assigned. Keep in mind that an adapter or domain can be plugged in only if each APQN with the APID of the adapter or the APQI of the domain references a queue device bound to the vfio_ap device driver. Consequently, not all newly added adapters and domains will necessarily get hot plugged. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 1 + drivers/s390/crypto/vfio_ap_ops.c | 132 +++++++++++++++++++++++--- drivers/s390/crypto/vfio_ap_private.h | 2 + 3 files changed, 124 insertions(+), 11 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 9f6c5d82dfb5..0ed557634302 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -179,6 +179,7 @@ static int __init vfio_ap_init(void) vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed; + vfio_ap_drv.on_scan_complete = vfio_ap_on_scan_complete; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index f1dd77729dd9..ccc58daf82f6 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -489,6 +489,7 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, if (!test_bit_inv(apid, (unsigned long *) matrix_dev->config_info.apm)) { clear_bit_inv(apid, shadow_crycb->apm); + continue; } @@ -502,6 +503,7 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, if (!test_bit_inv(apqi, (unsigned long *) matrix_dev->config_info.aqm)) { clear_bit_inv(apqi, shadow_crycb->aqm); + continue; } @@ -515,11 +517,12 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, */ apqn = AP_MKQID(apid, apqi); if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) { - if (filter_apids) + if (filter_apids) { clear_bit_inv(apid, shadow_crycb->apm); - else - clear_bit_inv(apqi, shadow_crycb->aqm); - break; + break; + } + + clear_bit_inv(apqi, shadow_crycb->aqm); } } @@ -541,7 +544,7 @@ static bool vfio_ap_mdev_configure_crycb(struct ap_matrix_mdev *matrix_mdev) int napm, naqm; struct ap_matrix shadow_crycb; - vfio_ap_matrix_init(&matrix_dev->info, &shadow_crycb); + vfio_ap_matrix_init(&matrix_dev->config_info, &shadow_crycb); napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES); naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS); /* @@ -569,6 +572,8 @@ static bool vfio_ap_mdev_configure_crycb(struct ap_matrix_mdev *matrix_mdev) * matrix mdev after filtering the APQIs, then no APQNs * shall be assigned to the guest's CRYCB. */ + naqm = vfio_ap_mdev_filter_matrix(matrix_mdev, + &shadow_crycb, false); if (naqm == 0) { bitmap_clear(shadow_crycb.apm, 0, AP_DEVICES); bitmap_clear(shadow_crycb.aqm, 0, AP_DOMAINS); @@ -633,8 +638,8 @@ static bool vfio_ap_mdev_assign_apqis_4_apid(struct ap_matrix_mdev *matrix_mdev, bitmap_copy(aqm, matrix_mdev->matrix.aqm, AP_DOMAINS); for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { - if (!test_bit_inv(apqi, - (unsigned long *) matrix_dev->info.aqm)) + if (!test_bit_inv(apqi, (unsigned long *) + matrix_dev->config_info.aqm)) clear_bit_inv(apqi, aqm); apqn = AP_MKQID(apid, apqi); @@ -657,7 +662,7 @@ static bool vfio_ap_mdev_assign_guest_apid(struct ap_matrix_mdev *matrix_mdev, unsigned long apqi, apqn; if (!vfio_ap_mdev_has_crycb(matrix_mdev) || - !test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm)) + !test_bit_inv(apid, (unsigned long *)matrix_dev->config_info.apm)) return false; if (bitmap_empty(matrix_mdev->shadow_crycb.aqm, AP_DOMAINS)) @@ -853,8 +858,8 @@ static bool vfio_ap_mdev_assign_apids_4_apqi(struct ap_matrix_mdev *matrix_mdev, bitmap_copy(apm, matrix_mdev->matrix.apm, AP_DEVICES); for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { - if (!test_bit_inv(apid, - (unsigned long *) matrix_dev->info.apm)) + if (!test_bit_inv(apid, (unsigned long *) + matrix_dev->config_info.apm)) clear_bit_inv(apqi, apm); apqn = AP_MKQID(apid, apqi); @@ -877,7 +882,7 @@ static bool vfio_ap_mdev_assign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, unsigned long apid, apqn; if (!vfio_ap_mdev_has_crycb(matrix_mdev) || - !test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) + !test_bit_inv(apqi, (unsigned long *)matrix_dev->config_info.aqm)) return false; if (bitmap_empty(matrix_mdev->shadow_crycb.apm, AP_DEVICES)) @@ -1673,6 +1678,16 @@ bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apid, apm_unassign, AP_DEVICES) { unassigned |= vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid); + /* + * If the APID is not assigned to the matrix mdev's shadow + * CRYCB, continue with the next APID. + */ + if (!test_bit_inv(apid, matrix_mdev->shadow_crycb.apm)) + continue; + + /* Unassign the APID from the matrix mdev's shadow CRYCB */ + clear_bit_inv(apid, matrix_mdev->shadow_crycb.apm); + unassigned = true; } return unassigned; @@ -1706,6 +1721,17 @@ bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apqi, aqm_unassign, AP_DOMAINS) { unassigned |= vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi); + + /* + * If the APQI is not assigned to the matrix mdev's shadow + * CRYCB, continue with the next APQI + */ + if (!test_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm)) + continue; + + /* Unassign the APQI from the matrix mdev's shadow CRYCB */ + clear_bit_inv(apqi, matrix_mdev->shadow_crycb.aqm); + unassigned = true; } return unassigned; @@ -1767,3 +1793,87 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, } mutex_unlock(&matrix_dev->lock); } + +bool vfio_ap_mdev_assign_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apm_assign) +{ + unsigned long apid; + bool assigned = false; + + for_each_set_bit_inv(apid, apm_assign, AP_DEVICES) + if (test_bit_inv(apid, matrix_mdev->matrix.apm)) + if (vfio_ap_mdev_assign_guest_apid(matrix_mdev, apid)) + assigned = true; + + return assigned; +} + +bool vfio_ap_mdev_assign_apqis(struct ap_matrix_mdev *matrix_mdev, + unsigned long *aqm_assign) +{ + unsigned long apqi; + bool assigned = false; + + for_each_set_bit_inv(apqi, aqm_assign, AP_DOMAINS) + if (test_bit_inv(apqi, matrix_mdev->matrix.aqm)) + if (vfio_ap_mdev_assign_guest_apqi(matrix_mdev, apqi)) + assigned = true; + + return assigned; +} + +void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info) +{ + struct ap_matrix_mdev *matrix_mdev; + DECLARE_BITMAP(apm_assign, AP_DEVICES); + DECLARE_BITMAP(aqm_assign, AP_DOMAINS); + int ap_add, aq_add; + bool assign; + unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm; + + /* + * If we are not in the middle of a host configuration change scan it is + * likely that the vfio_ap driver was loaded mid-scan, so let's handle + * this scenario by calling the vfio_ap_on_cfg_changed function which + * gets called at the start of an AP bus scan when the host AP + * configuration has changed. + */ + if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG)) + vfio_ap_on_cfg_changed(new_config_info, old_config_info); + + cur_apm = (unsigned long *)matrix_dev->config_info.apm; + cur_aqm = (unsigned long *)matrix_dev->config_info.aqm; + + prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm; + prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm; + + ap_add = bitmap_andnot(apm_assign, cur_apm, prev_apm, AP_DEVICES); + aq_add = bitmap_andnot(aqm_assign, cur_aqm, prev_aqm, AP_DOMAINS); + + mutex_lock(&matrix_dev->lock); + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + continue; + + assign = false; + + if (ap_add) + if (bitmap_intersects(matrix_mdev->matrix.apm, + apm_assign, AP_DEVICES)) + assign |= vfio_ap_mdev_assign_apids(matrix_mdev, + apm_assign); + + if (aq_add) + if (bitmap_intersects(matrix_mdev->matrix.aqm, + aqm_assign, AP_DOMAINS)) + assign |= vfio_ap_mdev_assign_apqis(matrix_mdev, + aqm_assign); + + if (assign) + vfio_ap_mdev_commit_crycb(matrix_mdev); + } + + matrix_dev->flags &= ~AP_MATRIX_CFG_CHG; + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 82abbf03781f..38974c37591a 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -114,5 +114,7 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue); bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); +void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Tue Apr 7 19:20:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11478915 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79900174A for ; Tue, 7 Apr 2020 19:20:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 578A52076E for ; Tue, 7 Apr 2020 19:20:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727302AbgDGTUs (ORCPT ); Tue, 7 Apr 2020 15:20:48 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:12518 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727611AbgDGTUr (ORCPT ); Tue, 7 Apr 2020 15:20:47 -0400 Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 037J4SMi117309; Tue, 7 Apr 2020 15:20:43 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 306n25knud-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:43 -0400 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 037J4YB8117562; Tue, 7 Apr 2020 15:20:43 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 306n25knu0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 15:20:43 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 037JKV8m011500; Tue, 7 Apr 2020 19:20:42 GMT Received: from b01cxnp22035.gho.pok.ibm.com (b01cxnp22035.gho.pok.ibm.com [9.57.198.25]) by ppma03dal.us.ibm.com with ESMTP id 306hv6s2dw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 07 Apr 2020 19:20:42 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 037JKe5S54460704 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 7 Apr 2020 19:20:40 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 753312805A; Tue, 7 Apr 2020 19:20:40 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BA15728058; Tue, 7 Apr 2020 19:20:39 +0000 (GMT) Received: from cpe-172-100-173-215.stny.res.rr.com.com (unknown [9.85.207.206]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 7 Apr 2020 19:20:39 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, jjherne@linux.ibm.com, fiuczy@linux.ibm.com, Tony Krowiak Subject: [PATCH v7 15/15] s390/vfio-ap: handle probe/remove not due to host AP config changes Date: Tue, 7 Apr 2020 15:20:15 -0400 Message-Id: <20200407192015.19887-16-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200407192015.19887-1-akrowiak@linux.ibm.com> References: <20200407192015.19887-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.676 definitions=2020-04-07_08:2020-04-07,2020-04-07 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxlogscore=999 priorityscore=1501 malwarescore=0 bulkscore=0 impostorscore=0 suspectscore=3 lowpriorityscore=0 adultscore=0 mlxscore=0 spamscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2004070151 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org AP queue devices are probed or removed for reasons other than changes to the host AP configuration: * Each queue device associated with a card device will get created and probed when the state of the AP adapter represented by the card device dynamically changes from standby to online. * Each queue device associated with a card device will get removed when the state of the AP adapter to which the queue represented by the queue device dynamically changes from online to standby. * Each queue device associated with a card device will get removed when the type of the AP adapter to which the queue represented by the queue device dynamically changes. * Each queue device associated with a card device will get removed when the status of the queue represented by the queue device changes from operating to check stop. * AP queue devices can be manually bound to or unbound from the vfio_ap device driver by a root user via the sysfs bind/unbind attributes of the driver. In response to a queue device probe or remove that is not the result of a change to the host's AP configuration, if a KVM guest is using the matrix mdev to which the APQN of the queue device is assigned, the vfio_ap device driver must respond accordingly. In an ideal world, the queue corresponding to the queue device being probed would be hot plugged into the guest. Likewise, the queue corresponding to the queue device being removed would be hot unplugged from the guest. Unfortunately, the AP architecture precludes plugging or unplugging individual queues, so let's handle the probe or remove of an AP queue device as follows: Handling Probe -------------- There are two requirements that must be met in order to give a guest access to the queue corresponding to the queue device being probed: * Each APQN derived from the APID of the queue device and the APQIs of the domains already assigned to the guest's AP configuration must reference a queue device bound to the vfio_ap device driver. * Each APQN derived from the APQI of the queue device and the APIDs of the adapters assigned to the guest's AP configuration must reference a queue device bound to the vfio_ap device driver. If the above conditions are met, the APQN will be assigned to the guest's AP configuration and the guest will be given access to the queue. Handling Remove --------------- Since the AP architecture precludes us from taking access to an individual queue from a guest, we are left with the choice of taking access away from either the adapter or the domain to which the queue is connected. Access to the adapter will be taken away because it is likely that most of the time, the remove callback will be invoked because the adapter state has transitioned from online to standby. In such a case, no queue connected to the adapter will be available to access. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 38 +++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index ccc58daf82f6..918b735d5d56 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1601,6 +1601,15 @@ static void vfio_ap_mdev_for_queue(struct vfio_ap_queue *q) } } +void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q) +{ + if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + return; + + if (vfio_ap_mdev_configure_crycb(q->matrix_mdev)) + vfio_ap_mdev_commit_crycb(q->matrix_mdev); +} + int vfio_ap_mdev_probe_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; @@ -1615,11 +1624,35 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue) q->saved_isc = VFIO_AP_ISC_INVALID; vfio_ap_mdev_for_queue(q); hash_add(matrix_dev->qtable, &q->qnode, q->apqn); + /* Make sure we're not in the middle of an AP configuration change. */ + if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG)) + vfio_ap_mdev_hot_plug_queue(q); mutex_unlock(&matrix_dev->lock); return 0; } +void vfio_ap_mdev_hot_unplug_queue(struct vfio_ap_queue *q) +{ + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); + + if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + return; + + /* + * If the APQN is assigned to the guest, then let's + * go ahead and unplug the adapter since the + * architecture does not provide a means to unplug + * an individual queue. + */ + if (test_bit_inv(apid, q->matrix_mdev->shadow_crycb.apm) && + test_bit_inv(apqi, q->matrix_mdev->shadow_crycb.aqm)) { + if (vfio_ap_mdev_unassign_guest_apid(q->matrix_mdev, apid)) + vfio_ap_mdev_commit_crycb(q->matrix_mdev); + } +} + void vfio_ap_mdev_remove_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; @@ -1627,6 +1660,11 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) mutex_lock(&matrix_dev->lock); q = dev_get_drvdata(&queue->ap_dev.device); + + /* Make sure we're not in the middle of an AP configuration change. */ + if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG)) + vfio_ap_mdev_hot_unplug_queue(q); + dev_set_drvdata(&queue->ap_dev.device, NULL); apid = AP_QID_CARD(q->apqn); apqi = AP_QID_QUEUE(q->apqn);