From patchwork Fri Aug 21 19:56:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730503 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94235618 for ; Fri, 21 Aug 2020 20:01:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7E6F820791 for ; Fri, 21 Aug 2020 20:01:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="m/hgqOb6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727003AbgHUT4g (ORCPT ); Fri, 21 Aug 2020 15:56:36 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:3948 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726306AbgHUT4c (ORCPT ); Fri, 21 Aug 2020 15:56:32 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJWxB5023963; Fri, 21 Aug 2020 15:56:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=1PGyV5xOIEUjK9z3Jg4QxXcPpDhZXcE3sZY/Qjg8gwg=; b=m/hgqOb6xYkIo2JBgQ4vwLXlYtOPuBTDm8FJSm5gXVV9YWphk4Tn+cBZcQhbwU+kSQYq dplXEUnQEnSiZCqEVhbLVXA96AUXd3g//V3V14reif3WsA958GtJPaXKa6VYFx4e2v6K hQubKbUzI7Ov5DmYwv2rhwKhc2jk7b1idk7Qj+DX4uIo6kUVPHRiIMr5+2rFdA6+PzIs Mv9P3WkjmgPXCQsYHCqzwtO38cBwiyEQCu/KrBrPRPYicvzeDNvp5vf+GlDre3xubmut i/iJISzCa0EVrpylb+/2j/fSKquUAQIuH6QlPf1CkcDJ+qsBS4KCE4vXy5cX2g9D7VOR 4Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332dw6vavh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:29 -0400 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJYAfO027332; Fri, 21 Aug 2020 15:56:29 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 332dw6vava-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:29 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsM76005387; Fri, 21 Aug 2020 19:56:28 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma03wdc.us.ibm.com with ESMTP id 3304ceprww-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:28 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuPFW47841588 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:25 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C41F778060; Fri, 21 Aug 2020 19:56:25 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 475C27805C; Fri, 21 Aug 2020 19:56:24 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:24 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 01/16] s390/vfio-ap: add version vfio_ap module Date: Fri, 21 Aug 2020 15:56:01 -0400 Message-Id: <20200821195616.13554-2-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 priorityscore=1501 suspectscore=3 mlxlogscore=999 spamscore=0 impostorscore=0 mlxscore=0 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210183 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's set a version for the vfio_ap module so that automated regression tests can determine whether dynamic configuration tests can be run or not. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index be2520cc010b..f4ceb380dd61 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -17,10 +17,12 @@ #define VFIO_AP_ROOT_NAME "vfio_ap" #define VFIO_AP_DEV_NAME "matrix" +#define VFIO_AP_MODULE_VERSION "1.2.0" MODULE_AUTHOR("IBM Corporation"); MODULE_DESCRIPTION("VFIO AP device driver, Copyright IBM Corp. 2018"); MODULE_LICENSE("GPL v2"); +MODULE_VERSION(VFIO_AP_MODULE_VERSION); static struct ap_driver vfio_ap_drv; From patchwork Fri Aug 21 19:56:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730501 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E149D618 for ; Fri, 21 Aug 2020 20:00:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC9922072D for ; Fri, 21 Aug 2020 20:00:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="tiwESppU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726870AbgHUUAy (ORCPT ); Fri, 21 Aug 2020 16:00:54 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:9182 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726716AbgHUT4g (ORCPT ); Fri, 21 Aug 2020 15:56:36 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJVk0Z093455; Fri, 21 Aug 2020 15:56:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=ry+9IDY5zB19YSm8+z1lF5SU9BlVwA+BNZHqaYs5rdY=; b=tiwESppU6yU8NsPK5fO4v1nAQLTXife7mk/v0n+FYwVzvoFz6v7U9k6L3OXm08o4NB5I TfJ+lT3ym/mNt+/3bvLhv4jH4Y6TNEEx4b7MQ54rTGAe+EaBqZ5V2hqZ2i8xLlmkXG1f 7r5uhlfZQvUDw/u0lgaeXzeZ/D3R3ZARMZUUttPVTf25j1Tskc5psc/6znnQrulkdWv/ kjfg/3rnxu1tPVNuDLzaPqzN0NNVNV7sp+QYyIP7E45r213DBmlo66YMiH8sLDOZd7lr 5pc1EOiESw5pQwqZWFC3/2umbdg7owXwxkL0EoAay3z9GkUgOpOJAwPwd9Vs5d8/tl31 qw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332kse9gb6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:32 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJW442095640; Fri, 21 Aug 2020 15:56:32 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 332kse9gau-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:31 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsM2b005384; Fri, 21 Aug 2020 19:56:30 GMT Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by ppma03wdc.us.ibm.com with ESMTP id 3304ceprx3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:30 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuNQv53870954 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:23 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8C7D67805E; Fri, 21 Aug 2020 19:56:27 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EA3D27805C; Fri, 21 Aug 2020 19:56:25 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:25 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , kernel test robot Subject: [PATCH v10 02/16] s390/vfio-ap: use new AP bus interface to search for queue devices Date: Fri, 21 Aug 2020 15:56:02 -0400 Message-Id: <20200821195616.13554-3-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=11 adultscore=0 mlxlogscore=978 phishscore=0 clxscore=1015 malwarescore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This patch refactor's the vfio_ap device driver to use the AP bus's ap_get_qdev() function to retrieve the vfio_ap_queue struct containing information about a queue that is bound to the vfio_ap device driver. The bus's ap_get_qdev() function retrieves the queue device from a hashtable keyed by APQN. This is much more efficient than looping over the list of devices attached to the AP bus by several orders of magnitude. Signed-off-by: Tony Krowiak Reported-by: kernel test robot --- drivers/s390/crypto/vfio_ap_drv.c | 27 ++------- drivers/s390/crypto/vfio_ap_ops.c | 86 +++++++++++++++------------ drivers/s390/crypto/vfio_ap_private.h | 8 ++- 3 files changed, 59 insertions(+), 62 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index f4ceb380dd61..24cdef60039a 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -53,15 +53,9 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids); */ static int vfio_ap_queue_dev_probe(struct ap_device *apdev) { - struct vfio_ap_queue *q; - - q = kzalloc(sizeof(*q), GFP_KERNEL); - if (!q) - return -ENOMEM; - dev_set_drvdata(&apdev->device, q); - q->apqn = to_ap_queue(&apdev->device)->qid; - q->saved_isc = VFIO_AP_ISC_INVALID; - return 0; + struct ap_queue *queue = to_ap_queue(&apdev->device); + + return vfio_ap_mdev_probe_queue(queue); } /** @@ -72,18 +66,9 @@ static int vfio_ap_queue_dev_probe(struct ap_device *apdev) */ static void vfio_ap_queue_dev_remove(struct ap_device *apdev) { - struct vfio_ap_queue *q; - int apid, apqi; - - mutex_lock(&matrix_dev->lock); - q = dev_get_drvdata(&apdev->device); - dev_set_drvdata(&apdev->device, NULL); - apid = AP_QID_CARD(q->apqn); - apqi = AP_QID_QUEUE(q->apqn); - vfio_ap_mdev_reset_queue(apid, apqi, 1); - vfio_ap_irq_disable(q); - kfree(q); - mutex_unlock(&matrix_dev->lock); + struct ap_queue *queue = to_ap_queue(&apdev->device); + + vfio_ap_mdev_remove_queue(queue); } static void vfio_ap_matrix_dev_release(struct device *dev) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index e0bde8518745..ad3925f04f61 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -26,43 +26,26 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev); -static int match_apqn(struct device *dev, const void *data) -{ - struct vfio_ap_queue *q = dev_get_drvdata(dev); - - return (q->apqn == *(int *)(data)) ? 1 : 0; -} - /** - * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list - * @matrix_mdev: the associated mediated matrix + * vfio_ap_get_queue: Retrieve a queue with a specific APQN. * @apqn: The queue APQN * - * Retrieve a queue with a specific APQN from the list of the - * devices of the vfio_ap_drv. - * Verify that the APID and the APQI are set in the matrix. + * Retrieve a queue with a specific APQN from the AP queue devices attached to + * the AP bus. * - * Returns the pointer to the associated vfio_ap_queue + * Returns the pointer to the vfio_ap_queue with the specified APQN, or NULL. */ -static struct vfio_ap_queue *vfio_ap_get_queue( - struct ap_matrix_mdev *matrix_mdev, - int apqn) +static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn) { + struct ap_queue *queue; struct vfio_ap_queue *q; - struct device *dev; - if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm)) - return NULL; - if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm)) + queue = ap_get_qdev(apqn); + if (!queue) return NULL; - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &apqn, match_apqn); - if (!dev) - return NULL; - q = dev_get_drvdata(dev); - q->matrix_mdev = matrix_mdev; - put_device(dev); + q = dev_get_drvdata(&queue->ap_dev.device); + put_device(&queue->ap_dev.device); return q; } @@ -144,7 +127,7 @@ static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q) * Returns if ap_aqic function failed with invalid, deconfigured or * checkstopped AP. */ -struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) +static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) { struct ap_qirq_ctrl aqic_gisa = {}; struct ap_queue_status status; @@ -293,10 +276,11 @@ static int handle_pqap(struct kvm_vcpu *vcpu) matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook, struct ap_matrix_mdev, pqap_hook); - q = vfio_ap_get_queue(matrix_mdev, apqn); + q = vfio_ap_get_queue(apqn); if (!q) goto out_unlock; + q->matrix_mdev = matrix_mdev; status = vcpu->run->s.regs.gprs[1]; /* If IR bit(16) is set we enable the interrupt */ @@ -1116,20 +1100,15 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, static void vfio_ap_irq_disable_apqn(int apqn) { - struct device *dev; struct vfio_ap_queue *q; - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &apqn, match_apqn); - if (dev) { - q = dev_get_drvdata(dev); + q = vfio_ap_get_queue(apqn); + if (q) vfio_ap_irq_disable(q); - put_device(dev); - } } -int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, - unsigned int retry) +static int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, + unsigned int retry) { struct ap_queue_status status; int retry2 = 2; @@ -1302,3 +1281,34 @@ void vfio_ap_mdev_unregister(void) { mdev_unregister_device(&matrix_dev->device); } + +int vfio_ap_mdev_probe_queue(struct ap_queue *queue) +{ + struct vfio_ap_queue *q; + + q = kzalloc(sizeof(*q), GFP_KERNEL); + if (!q) + return -ENOMEM; + + dev_set_drvdata(&queue->ap_dev.device, q); + q->apqn = queue->qid; + q->saved_isc = VFIO_AP_ISC_INVALID; + + return 0; +} + +void vfio_ap_mdev_remove_queue(struct ap_queue *queue) +{ + struct vfio_ap_queue *q; + int apid, apqi; + + mutex_lock(&matrix_dev->lock); + q = dev_get_drvdata(&queue->ap_dev.device); + dev_set_drvdata(&queue->ap_dev.device, NULL); + apid = AP_QID_CARD(q->apqn); + apqi = AP_QID_QUEUE(q->apqn); + vfio_ap_mdev_reset_queue(apid, apqi, 1); + vfio_ap_irq_disable(q); + kfree(q); + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index f46dde56b464..a2aa05bec718 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -18,6 +18,7 @@ #include #include #include +#include #include "ap_bus.h" @@ -90,8 +91,6 @@ struct ap_matrix_mdev { extern int vfio_ap_mdev_register(void); extern void vfio_ap_mdev_unregister(void); -int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, - unsigned int retry); struct vfio_ap_queue { struct ap_matrix_mdev *matrix_mdev; @@ -100,5 +99,8 @@ struct vfio_ap_queue { #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; }; -struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q); + +int vfio_ap_mdev_probe_queue(struct ap_queue *queue); +void vfio_ap_mdev_remove_queue(struct ap_queue *queue); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Fri Aug 21 19:56:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730485 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 214B4618 for ; Fri, 21 Aug 2020 19:59:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E729120738 for ; Fri, 21 Aug 2020 19:59:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="pAWD3kVd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727041AbgHUT4o (ORCPT ); Fri, 21 Aug 2020 15:56:44 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:33268 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726846AbgHUT4h (ORCPT ); Fri, 21 Aug 2020 15:56:37 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJW0AJ086100; Fri, 21 Aug 2020 15:56:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Q83ZffGYATxIZXUMd+md0kXm5JAcARVD+UGuULz5O7U=; b=pAWD3kVdZuY8j4g33+wlgcqqlNRFQrrzYi98Msr4jq67aV6gX5HWnnaexo4VhI8j2DYi 8cZlDFC6oXUm9Z+fZ1Oyxf2c83qezoQ8uz7Dl2rfrStFJyGaBpLstSXd0pCvSu3slCPI HRFKEN2Rl0rc24sHGrAPxGvlSPERMtSoCootMaM7g8ZGzFTS7lae2Pmb4hOu0saEv1Dz YljHFHi6qp7kspZQJBlfaZoCbRI35EfdrRSkttaInB3cvDtH+x8MiodwIQE14gDaIuJV J7RB78NrnDsgIQy0+J+CZyBIcZWj1jc+nCJDXGS69r3KUI8OuK9bH1XJjaZ5sK4YvfwV Xw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332jhakdm2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:34 -0400 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJXj86090839; Fri, 21 Aug 2020 15:56:33 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 332jhakdkn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:33 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsqXM013481; Fri, 21 Aug 2020 19:56:32 GMT Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by ppma01wdc.us.ibm.com with ESMTP id 3304tm6m7u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:32 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuTQH60096978 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:29 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4D0F07805F; Fri, 21 Aug 2020 19:56:29 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C96857805C; Fri, 21 Aug 2020 19:56:27 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:27 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 03/16] s390/vfio-ap: manage link between queue struct and matrix mdev Date: Fri, 21 Aug 2020 15:56:03 -0400 Message-Id: <20200821195616.13554-4-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 malwarescore=0 phishscore=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 impostorscore=0 suspectscore=11 bulkscore=0 clxscore=1015 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's create links between each queue device bound to the vfio_ap device driver and the matrix mdev to which the queue is assigned. The idea is to facilitate efficient retrieval of the objects representing the queue devices and matrix mdevs as well as to verify that a queue assigned to a matrix mdev is bound to the driver. The links will be created as follows: * When the queue device is probed, if its APQN is assigned to a matrix mdev, the structures representing the queue device and the matrix mdev will be linked. * When an adapter or domain is assigned to a matrix mdev, for each new APQN assigned that references a queue device bound to the vfio_ap device driver, the structures representing the queue device and the matrix mdev will be linked. The links will be removed as follows: * When the queue device is removed, if its APQN is assigned to a matrix mdev, the structures representing the queue device and the matrix mdev will be unlinked. * When an adapter or domain is unassigned from a matrix mdev, for each APQN unassigned that references a queue device bound to the vfio_ap device driver, the structures representing the queue device and the matrix mdev will be unlinked. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 132 +++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 2 + 2 files changed, 129 insertions(+), 5 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index ad3925f04f61..2e37ee82e422 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -50,6 +50,19 @@ static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn) return q; } +static struct vfio_ap_queue *vfio_ap_get_mdev_queue(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqn) +{ + struct vfio_ap_queue *q; + + hash_for_each_possible(matrix_mdev->qtable, q, mdev_qnode, apqn) { + if (q && (q->apqn == apqn)) + return q; + } + + return NULL; +} + /** * vfio_ap_wait_for_irqclear * @apqn: The AP Queue number @@ -160,7 +173,6 @@ static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) status.response_code); end_free: vfio_ap_free_aqic_resources(q); - q->matrix_mdev = NULL; return status; } @@ -262,7 +274,6 @@ static int handle_pqap(struct kvm_vcpu *vcpu) struct vfio_ap_queue *q; struct ap_queue_status qstatus = { .response_code = AP_RESPONSE_Q_NOT_AVAIL, }; - struct ap_matrix_mdev *matrix_mdev; /* If we do not use the AIV facility just go to userland */ if (!(vcpu->arch.sie_block->eca & ECA_AIV)) @@ -273,14 +284,11 @@ static int handle_pqap(struct kvm_vcpu *vcpu) if (!vcpu->kvm->arch.crypto.pqap_hook) goto out_unlock; - matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook, - struct ap_matrix_mdev, pqap_hook); q = vfio_ap_get_queue(apqn); if (!q) goto out_unlock; - q->matrix_mdev = matrix_mdev; status = vcpu->run->s.regs.gprs[1]; /* If IR bit(16) is set we enable the interrupt */ @@ -320,6 +328,7 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) matrix_mdev->mdev = mdev; vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); + hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; matrix_mdev->pqap_hook.owner = THIS_MODULE; @@ -548,6 +557,87 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) return 0; } +enum qlink_type { + LINK_APID, + LINK_APQI, + UNLINK_APID, + UNLINK_APQI, +}; + +static void vfio_ap_mdev_link_queue(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid, unsigned long apqi) +{ + struct vfio_ap_queue *q; + + q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); + if (q) { + q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, + &q->mdev_qnode, q->apqn); + } +} + +static void vfio_ap_mdev_unlink_queue(unsigned long apid, unsigned long apqi) +{ + struct vfio_ap_queue *q; + + q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); + if (q) { + q->matrix_mdev = NULL; + hash_del(&q->mdev_qnode); + } +} + +/** + * vfio_ap_mdev_link_queues + * + * @matrix_mdev: The matrix mdev to link. + * @type: The type of @qlink_id. + * @qlink_id: The APID or APQI of the queues to link. + * + * Sets or clears the links between the queues with the specified @qlink_id + * and the @matrix_mdev: + * @type == LINK_APID: Set the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APID) + * @type == LINK_APQI: Set the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APQI) + * @type == UNLINK_APID: Clear the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APID) + * @type == UNLINK_APQI: Clear the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APQI) + */ +static void vfio_ap_mdev_link_queues(struct ap_matrix_mdev *matrix_mdev, + enum qlink_type type, + unsigned long qlink_id) +{ + unsigned long id; + + switch (type) { + case LINK_APID: + for_each_set_bit_inv(id, matrix_mdev->matrix.aqm, + matrix_mdev->matrix.aqm_max + 1) + vfio_ap_mdev_link_queue(matrix_mdev, qlink_id, id); + break; + case UNLINK_APID: + for_each_set_bit_inv(id, matrix_mdev->matrix.aqm, + matrix_mdev->matrix.aqm_max + 1) + vfio_ap_mdev_unlink_queue(qlink_id, id); + break; + case LINK_APQI: + for_each_set_bit_inv(id, matrix_mdev->matrix.apm, + matrix_mdev->matrix.apm_max + 1) + vfio_ap_mdev_link_queue(matrix_mdev, id, qlink_id); + break; + case UNLINK_APQI: + for_each_set_bit_inv(id, matrix_mdev->matrix.apm, + matrix_mdev->matrix.apm_max + 1) + vfio_ap_mdev_link_queue(matrix_mdev, id, qlink_id); + break; + default: + WARN_ON_ONCE(1); + } +} + /** * assign_adapter_store * @@ -617,6 +707,7 @@ static ssize_t assign_adapter_store(struct device *dev, if (ret) goto share_err; + vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); ret = count; goto done; @@ -668,6 +759,7 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); + vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid); mutex_unlock(&matrix_dev->lock); return count; @@ -758,6 +850,7 @@ static ssize_t assign_domain_store(struct device *dev, if (ret) goto share_err; + vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); ret = count; goto done; @@ -810,6 +903,7 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); + vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi); mutex_unlock(&matrix_dev->lock); return count; @@ -1282,6 +1376,29 @@ void vfio_ap_mdev_unregister(void) mdev_unregister_device(&matrix_dev->device); } +/** + * vfio_ap_queue_link_mdev + * + * @q: The queue to link with the matrix mdev. + * + * Links @q with the matrix mdev to which the queue's APQN is assigned. + */ +static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q) +{ + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); + struct ap_matrix_mdev *matrix_mdev; + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (test_bit_inv(apid, matrix_mdev->matrix.apm) && + test_bit_inv(apqi, matrix_mdev->matrix.aqm)) { + q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, &q->mdev_qnode, q->apqn); + break; + } + } +} + int vfio_ap_mdev_probe_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; @@ -1290,9 +1407,12 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue) if (!q) return -ENOMEM; + mutex_lock(&matrix_dev->lock); dev_set_drvdata(&queue->ap_dev.device, q); q->apqn = queue->qid; q->saved_isc = VFIO_AP_ISC_INVALID; + vfio_ap_queue_link_mdev(q); + mutex_unlock(&matrix_dev->lock); return 0; } @@ -1309,6 +1429,8 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) apqi = AP_QID_QUEUE(q->apqn); vfio_ap_mdev_reset_queue(apid, apqi, 1); vfio_ap_irq_disable(q); + if (q->matrix_mdev) + hash_del(&q->mdev_qnode); kfree(q); mutex_unlock(&matrix_dev->lock); } diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index a2aa05bec718..57da703b549a 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -87,6 +87,7 @@ struct ap_matrix_mdev { struct kvm *kvm; struct kvm_s390_module_hook pqap_hook; struct mdev_device *mdev; + DECLARE_HASHTABLE(qtable, 8); }; extern int vfio_ap_mdev_register(void); @@ -98,6 +99,7 @@ struct vfio_ap_queue { int apqn; #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; + struct hlist_node mdev_qnode; }; int vfio_ap_mdev_probe_queue(struct ap_queue *queue); From patchwork Fri Aug 21 19:56:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79845618 for ; Fri, 21 Aug 2020 19:59:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5AEDB20738 for ; Fri, 21 Aug 2020 19:59:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="p9RaS9ke" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727078AbgHUT4p (ORCPT ); Fri, 21 Aug 2020 15:56:45 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:46524 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727013AbgHUT4j (ORCPT ); Fri, 21 Aug 2020 15:56:39 -0400 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJg53V124299; Fri, 21 Aug 2020 15:56:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=S+hOrLycsYb04yYf+Uy0gRqS48y+/ZaYdhmy2oTqGu4=; b=p9RaS9kenxviAH1g50qgyPC+mSWkXo6HmkThJHl1IteRZ5gYZEt+8RtXHlG9dxtJTi3b 6aDjdFRugqFR3AOC+FLJj9bwqMZ4AzmQL0qk6QiD7clPUHB5+ahX2hdtXHYWd5ME+Gv7 Y2PlSC8rf7jPVYaFhIZ86SIe02InoVj9JivX+jGS6yUyhzZrDnQrJhjjg7KpQGdIntNG 0h55PuT79mfTSrVKtKkZDKvmSL3uUJzpGx2eTMKAubSYvgF6bpfl9dc4ynPsqYPt/R0j urTcfiBonMuhM4FbYngQPoVaZBYGiSkgwSMlFcK6xouhfGBnkv8WL9+INs2Fmw52bmY5 Bg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332mna88xv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:35 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJgU94124988; Fri, 21 Aug 2020 15:56:34 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 332mna88xm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:34 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJshAM010544; Fri, 21 Aug 2020 19:56:34 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma05wdc.us.ibm.com with ESMTP id 3304themg5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:34 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuVvT62915068 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:31 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 21AB578066; Fri, 21 Aug 2020 19:56:31 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7EBDF7805C; Fri, 21 Aug 2020 19:56:29 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:29 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , kernel test robot Subject: [PATCH v10 04/16] s390/zcrypt: driver callback to indicate resource in use Date: Fri, 21 Aug 2020 15:56:04 -0400 Message-Id: <20200821195616.13554-5-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 priorityscore=1501 suspectscore=3 malwarescore=0 lowpriorityscore=0 impostorscore=0 mlxlogscore=999 clxscore=1015 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduces a new driver callback to prevent a root user from unbinding an AP queue from its device driver if the queue is in use. The intent of this callback is to provide a driver with the means to prevent a root user from inadvertently taking a queue away from a matrix mdev and giving it to the host while it is assigned to the matrix mdev. The callback will be invoked whenever a change to the AP bus's sysfs apmask or aqmask attributes would result in one or more AP queues being removed from its driver. If the callback responds in the affirmative for any driver queried, the change to the apmask or aqmask will be rejected with a device in use error. For this patch, only non-default drivers will be queried. Currently, there is only one non-default driver, the vfio_ap device driver. The vfio_ap device driver facilitates pass-through of an AP queue to a guest. The idea here is that a guest may be administered by a different sysadmin than the host and we don't want AP resources to unexpectedly disappear from a guest's AP configuration (i.e., adapters, domains and control domains assigned to the matrix mdev). This will enforce the proper procedure for removing AP resources intended for guest usage which is to first unassign them from the matrix mdev, then unbind them from the vfio_ap device driver. Signed-off-by: Tony Krowiak Reported-by: kernel test robot --- drivers/s390/crypto/ap_bus.c | 148 ++++++++++++++++++++++++++++++++--- drivers/s390/crypto/ap_bus.h | 4 + 2 files changed, 142 insertions(+), 10 deletions(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index 24a1940b829e..db27bd931308 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -35,6 +35,7 @@ #include #include #include +#include #include "ap_bus.h" #include "ap_debug.h" @@ -889,6 +890,23 @@ static int modify_bitmap(const char *str, unsigned long *bitmap, int bits) return 0; } +static int ap_parse_bitmap_str(const char *str, unsigned long *bitmap, int bits, + unsigned long *newmap) +{ + unsigned long size; + int rc; + + size = BITS_TO_LONGS(bits)*sizeof(unsigned long); + if (*str == '+' || *str == '-') { + memcpy(newmap, bitmap, size); + rc = modify_bitmap(str, newmap, bits); + } else { + memset(newmap, 0, size); + rc = hex2bitmap(str, newmap, bits); + } + return rc; +} + int ap_parse_mask_str(const char *str, unsigned long *bitmap, int bits, struct mutex *lock) @@ -908,14 +926,7 @@ int ap_parse_mask_str(const char *str, kfree(newmap); return -ERESTARTSYS; } - - if (*str == '+' || *str == '-') { - memcpy(newmap, bitmap, size); - rc = modify_bitmap(str, newmap, bits); - } else { - memset(newmap, 0, size); - rc = hex2bitmap(str, newmap, bits); - } + rc = ap_parse_bitmap_str(str, bitmap, bits, newmap); if (rc == 0) memcpy(bitmap, newmap, size); mutex_unlock(lock); @@ -1107,12 +1118,70 @@ static ssize_t apmask_show(struct bus_type *bus, char *buf) return rc; } +static int __verify_card_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newapm = (unsigned long *)data; + + /* + * No need to verify whether the driver is using the queues if it is the + * default driver. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(newapm, ap_perms.aqm)) + rc = -EADDRINUSE; + + module_put(drv->owner); + + return rc; +} + +static int apmask_commit(unsigned long *newapm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DEVICES)]; + + /* + * Check if any bits in the apmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newapm, ap_perms.apm, AP_DEVICES)) { + rc = bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_card_reservations); + if (rc) + return rc; + } + + memcpy(ap_perms.apm, newapm, APMASKSIZE); + + return 0; +} + static ssize_t apmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + DECLARE_BITMAP(newapm, AP_DEVICES); + + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + rc = ap_parse_bitmap_str(buf, ap_perms.apm, AP_DEVICES, newapm); + if (rc) + goto done; - rc = ap_parse_mask_str(buf, ap_perms.apm, AP_DEVICES, &ap_perms_mutex); + rc = apmask_commit(newapm); + +done: + mutex_unlock(&ap_perms_mutex); if (rc) return rc; @@ -1138,12 +1207,71 @@ static ssize_t aqmask_show(struct bus_type *bus, char *buf) return rc; } +static int __verify_queue_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newaqm = (unsigned long *)data; + + /* + * If the reserved bits do not identify queues reserved for use by the + * non-default driver, there is no need to verify the driver is using + * the queues. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(ap_perms.apm, newaqm)) + rc = -EADDRINUSE; + + module_put(drv->owner); + + return rc; +} + +static int aqmask_commit(unsigned long *newaqm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DOMAINS)]; + + /* + * Check if any bits in the aqmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newaqm, ap_perms.aqm, AP_DOMAINS)) { + rc = bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_queue_reservations); + if (rc) + return rc; + } + + memcpy(ap_perms.aqm, newaqm, AQMASKSIZE); + + return 0; +} + static ssize_t aqmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + DECLARE_BITMAP(newaqm, AP_DOMAINS); - rc = ap_parse_mask_str(buf, ap_perms.aqm, AP_DOMAINS, &ap_perms_mutex); + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + rc = ap_parse_bitmap_str(buf, ap_perms.aqm, AP_DOMAINS, newaqm); + if (rc) + goto done; + + rc = aqmask_commit(newaqm); + +done: + mutex_unlock(&ap_perms_mutex); if (rc) return rc; diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 1ea046324e8f..48c57b3d53a0 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -136,6 +136,7 @@ struct ap_driver { int (*probe)(struct ap_device *); void (*remove)(struct ap_device *); + bool (*in_use)(unsigned long *apm, unsigned long *aqm); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) @@ -255,6 +256,9 @@ void ap_queue_init_state(struct ap_queue *aq); struct ap_card *ap_card_create(int id, int queue_depth, int raw_device_type, int comp_device_type, unsigned int functions); +#define APMASKSIZE (BITS_TO_LONGS(AP_DEVICES) * sizeof(unsigned long)) +#define AQMASKSIZE (BITS_TO_LONGS(AP_DOMAINS) * sizeof(unsigned long)) + struct ap_perms { unsigned long ioctlm[BITS_TO_LONGS(AP_IOCTLS)]; unsigned long apm[BITS_TO_LONGS(AP_DEVICES)]; From patchwork Fri Aug 21 19:56:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730491 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 537D2618 for ; Fri, 21 Aug 2020 19:59:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2AE9720724 for ; Fri, 21 Aug 2020 19:59:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="VKFiuYIz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726886AbgHUT7n (ORCPT ); Fri, 21 Aug 2020 15:59:43 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:10604 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726887AbgHUT4n (ORCPT ); Fri, 21 Aug 2020 15:56:43 -0400 Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJXTYJ065328; Fri, 21 Aug 2020 15:56:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=3TQOtABLu9jM2ycJgktASokO4ui8ZL08vYc1+Eax2i8=; b=VKFiuYIzUSx9HJnKwoH7Q1IG5cB1rbUw3LH7gcfRSA5AtYZu9wAcbPctmkSmnMK2F6c1 HE02iKAtkLGirI3hF+jtcwlFwa1a56NqlozvLs271DRGwCeScCbNJopWtCOymTQTbQBZ V+zx42JcbairfSt4utaae9tiGIgO7rbgBR1VwaYcOlOa9sQ9yHLuEXO+U2u8SZco9IWv KbUT8DgjFwkkvmskJI6EbUu56NyK3cWZ4e9kHMdTwUZbRxTy8gW+0oE9+cp4lSnm/uef IC6UEs1fTsBtcPgtgTIDn/d33XU+EBRJlOWBNcnNyPtnTZdlPQkVA9Z/VJxCjqT+A8eu QA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3322nnud9r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:39 -0400 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJXZvx065636; Fri, 21 Aug 2020 15:56:38 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 3322nnud90-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:37 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJtg9V004830; Fri, 21 Aug 2020 19:56:36 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma02dal.us.ibm.com with ESMTP id 3304cdedq0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:36 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuU0t65012172 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:30 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BCB327805E; Fri, 21 Aug 2020 19:56:32 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 49A9D7805C; Fri, 21 Aug 2020 19:56:31 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:31 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 05/16] s390/vfio-ap: implement in-use callback for vfio_ap driver Date: Fri, 21 Aug 2020 15:56:05 -0400 Message-Id: <20200821195616.13554-6-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=3 lowpriorityscore=0 impostorscore=0 phishscore=0 priorityscore=1501 clxscore=1015 spamscore=0 mlxlogscore=999 malwarescore=0 bulkscore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210183 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's implement the callback to indicate when an APQN is in use by the vfio_ap device driver. The callback is invoked whenever a change to the apmask or aqmask would result in one or more queue devices being removed from the driver. The vfio_ap device driver will indicate a resource is in use if the APQN of any of the queue devices to be removed are assigned to any of the matrix mdevs under the driver's control. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 1 + drivers/s390/crypto/vfio_ap_ops.c | 68 ++++++++++++++++++++------- drivers/s390/crypto/vfio_ap_private.h | 2 + 3 files changed, 53 insertions(+), 18 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 24cdef60039a..aae5b3d8e3fa 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -175,6 +175,7 @@ static int __init vfio_ap_init(void) memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv)); vfio_ap_drv.probe = vfio_ap_queue_dev_probe; vfio_ap_drv.remove = vfio_ap_queue_dev_remove; + vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 2e37ee82e422..fc1aa6f947eb 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -515,18 +515,36 @@ vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, return 0; } +#define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \ + "already assigned to %s" + +static void vfio_ap_mdev_log_sharing_err(const char *mdev_name, + unsigned long *apm, + unsigned long *aqm) +{ + unsigned long apid, apqi; + + for_each_set_bit_inv(apid, apm, AP_DEVICES) + for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) + pr_err(MDEV_SHARING_ERR, apid, apqi, mdev_name); +} + /** * vfio_ap_mdev_verify_no_sharing * * Verifies that the APQNs derived from the cross product of the AP adapter IDs - * and AP queue indexes comprising the AP matrix are not configured for another + * and AP queue indexes comprising an AP matrix are not assigned to another * mediated device. AP queue sharing is not allowed. * * @matrix_mdev: the mediated matrix device + * @mdev_apm: mask indicating the APIDs of the APQNs to be verified + * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified * * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE. */ -static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, + unsigned long *mdev_apm, + unsigned long *mdev_aqm) { struct ap_matrix_mdev *lstdev; DECLARE_BITMAP(apm, AP_DEVICES); @@ -543,14 +561,15 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) * We work on full longs, as we can only exclude the leftover * bits in non-inverse order. The leftover is all zeros. */ - if (!bitmap_and(apm, matrix_mdev->matrix.apm, - lstdev->matrix.apm, AP_DEVICES)) + if (!bitmap_and(apm, mdev_apm, lstdev->matrix.apm, AP_DEVICES)) continue; - if (!bitmap_and(aqm, matrix_mdev->matrix.aqm, - lstdev->matrix.aqm, AP_DOMAINS)) + if (!bitmap_and(aqm, mdev_aqm, lstdev->matrix.aqm, AP_DOMAINS)) continue; + vfio_ap_mdev_log_sharing_err(dev_name(mdev_dev(lstdev->mdev)), + apm, aqm); + return -EADDRINUSE; } @@ -676,6 +695,7 @@ static ssize_t assign_adapter_store(struct device *dev, { int ret; unsigned long apid; + DECLARE_BITMAP(apm, AP_DEVICES); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); @@ -701,18 +721,18 @@ static ssize_t assign_adapter_store(struct device *dev, if (ret) goto done; - set_bit_inv(apid, matrix_mdev->matrix.apm); + memset(apm, 0, sizeof(apm)); + set_bit_inv(apid, apm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm, + matrix_mdev->matrix.aqm); if (ret) - goto share_err; + goto done; + set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); ret = count; - goto done; -share_err: - clear_bit_inv(apid, matrix_mdev->matrix.apm); done: mutex_unlock(&matrix_dev->lock); @@ -824,6 +844,7 @@ static ssize_t assign_domain_store(struct device *dev, { int ret; unsigned long apqi; + DECLARE_BITMAP(aqm, AP_DOMAINS); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; @@ -844,18 +865,18 @@ static ssize_t assign_domain_store(struct device *dev, if (ret) goto done; - set_bit_inv(apqi, matrix_mdev->matrix.aqm); + memset(aqm, 0, sizeof(aqm)); + set_bit_inv(apqi, aqm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, + matrix_mdev->matrix.apm, aqm); if (ret) - goto share_err; + goto done; + set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); ret = count; - goto done; -share_err: - clear_bit_inv(apqi, matrix_mdev->matrix.aqm); done: mutex_unlock(&matrix_dev->lock); @@ -1434,3 +1455,14 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) kfree(q); mutex_unlock(&matrix_dev->lock); } + +bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) +{ + bool in_use; + + mutex_lock(&matrix_dev->lock); + in_use = !!vfio_ap_mdev_verify_no_sharing(NULL, apm, aqm); + mutex_unlock(&matrix_dev->lock); + + return in_use; +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 57da703b549a..0c796ef11426 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -105,4 +105,6 @@ struct vfio_ap_queue { int vfio_ap_mdev_probe_queue(struct ap_queue *queue); void vfio_ap_mdev_remove_queue(struct ap_queue *queue); +bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Fri Aug 21 19:56:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BA585618 for ; Fri, 21 Aug 2020 19:59:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B6D7207BB for ; Fri, 21 Aug 2020 19:59:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="NkVAiYWF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727062AbgHUT4o (ORCPT ); Fri, 21 Aug 2020 15:56:44 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:50214 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726799AbgHUT4k (ORCPT ); Fri, 21 Aug 2020 15:56:40 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJX7uo154579; Fri, 21 Aug 2020 15:56:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=YeMv20HgYFdDqTqlofBZiWTx2uk6+q+V/Ippzz8KwoI=; b=NkVAiYWFTQEwJTPwikeHACRWRG0dhEZzraqEJpbp+Ue+QYExT2fbpnKleqB3IWkDvu8V XW6P0ubUINDNO40jOOD5kzD6nvd5NhceOAGRnlxzwC4LiJikHo/fCFOmoxFtyK08uvG/ NBKEw4BVUERCAIyOKOEvJRoZkfa11zfn64SKHEUHd7FqqrRtgxXLdPe6aqoYC0CKx2+/ isjHNInr+i0QGV/2xWQx0OqSYXdiupWet+EoYOmU3N+ggpGzaBW2rCn1QRJnQ0qEEw8K Ke1kJiB3k52DUI3l+dbJOYJ8uuvVXWhpKakbU8TrueG0clUBXLMMcwbiOqzL/YvuZaAr tA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3328e9n41s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:39 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJeYm3176961; Fri, 21 Aug 2020 15:56:38 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 3328e9n414-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:38 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsPiw005410; Fri, 21 Aug 2020 19:56:37 GMT Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by ppma03wdc.us.ibm.com with ESMTP id 3304ceprxq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:37 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuVlF18416330 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:31 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5AC877805F; Fri, 21 Aug 2020 19:56:34 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E419D7805C; Fri, 21 Aug 2020 19:56:32 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:32 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 06/16] s390/vfio-ap: introduce shadow APCB Date: Fri, 21 Aug 2020 15:56:06 -0400 Message-Id: <20200821195616.13554-7-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 spamscore=0 malwarescore=0 mlxscore=0 lowpriorityscore=0 suspectscore=3 adultscore=0 priorityscore=1501 impostorscore=0 phishscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210183 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The APCB is a field within the CRYCB that provides the AP configuration to a KVM guest. Let's introduce a shadow copy of the KVM guest's APCB and maintain it for the lifespan of the guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 32 ++++++++++++++++++++++----- drivers/s390/crypto/vfio_ap_private.h | 2 ++ 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index fc1aa6f947eb..efb229033f9e 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -305,14 +305,35 @@ static int handle_pqap(struct kvm_vcpu *vcpu) return 0; } +static void vfio_ap_matrix_clear_masks(struct ap_matrix *matrix) +{ + bitmap_clear(matrix->apm, 0, AP_DEVICES); + bitmap_clear(matrix->aqm, 0, AP_DOMAINS); + bitmap_clear(matrix->adm, 0, AP_DOMAINS); +} + static void vfio_ap_matrix_init(struct ap_config_info *info, struct ap_matrix *matrix) { + vfio_ap_matrix_clear_masks(matrix); matrix->apm_max = info->apxa ? info->Na : 63; matrix->aqm_max = info->apxa ? info->Nd : 15; matrix->adm_max = info->apxa ? info->Nd : 15; } +static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd); +} + +static void vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + kvm_arch_crypto_set_masks(matrix_mdev->kvm, + matrix_mdev->shadow_apcb.apm, + matrix_mdev->shadow_apcb.aqm, + matrix_mdev->shadow_apcb.adm); +} + static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) { struct ap_matrix_mdev *matrix_mdev; @@ -1202,13 +1223,12 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (ret) return NOTIFY_DONE; - /* If there is no CRYCB pointer, then we can't copy the masks */ - if (!matrix_mdev->kvm->arch.crypto.crycbd) + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) return NOTIFY_DONE; - kvm_arch_crypto_set_masks(matrix_mdev->kvm, matrix_mdev->matrix.apm, - matrix_mdev->matrix.aqm, - matrix_mdev->matrix.adm); + memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix, + sizeof(matrix_mdev->shadow_apcb)); + vfio_ap_mdev_commit_crycb(matrix_mdev); return NOTIFY_OK; } @@ -1323,6 +1343,8 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev) kvm_put_kvm(matrix_mdev->kvm); matrix_mdev->kvm = NULL; } + + vfio_ap_matrix_clear_masks(&matrix_mdev->shadow_apcb); mutex_unlock(&matrix_dev->lock); vfio_unregister_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY, diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 0c796ef11426..055bce6d45db 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -75,6 +75,7 @@ struct ap_matrix { * @list: allows the ap_matrix_mdev struct to be added to a list * @matrix: the adapters, usage domains and control domains assigned to the * mediated matrix device. + * @shadow_apcb: the shadow copy of the APCB field of the KVM guest's CRYCB * @group_notifier: notifier block used for specifying callback function for * handling the VFIO_GROUP_NOTIFY_SET_KVM event * @kvm: the struct holding guest's state @@ -82,6 +83,7 @@ struct ap_matrix { struct ap_matrix_mdev { struct list_head node; struct ap_matrix matrix; + struct ap_matrix shadow_apcb; struct notifier_block group_notifier; struct notifier_block iommu_notifier; struct kvm *kvm; From patchwork Fri Aug 21 19:56:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C5A51618 for ; Fri, 21 Aug 2020 20:00:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A7723207DA for ; Fri, 21 Aug 2020 20:00:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="knWEIyf7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727900AbgHUUAD (ORCPT ); Fri, 21 Aug 2020 16:00:03 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:6268 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727018AbgHUT4m (ORCPT ); Fri, 21 Aug 2020 15:56:42 -0400 Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJXQkd065199; Fri, 21 Aug 2020 15:56:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=o9D+ibd9iW4J8eEFqd99w505aNq/EjzI4JCfJIlJN/E=; b=knWEIyf7gKKd6Lb4d8xZXxV4OSKRSaFcIp05jvClqw6gBRyewA5+NLr+AN2i7sdyUh+I L6ytYOPLtYOGSZQ+5n9HCXnt47mNFaMO7mszGQBwCv5L99oVAS//xU3MpOXurtSD05W9 7FqSdLNhMHzwLLg2NzdyOQJtjeyB0DNN+g+O66WD11AOSdIzPfA7XCK1yUDVF3ppYrty SzyW4TGJ0Ikutpby5cCzA5YU4Ly2nkguPnLN5y++QoFX0orz1rqlMO1fjulhZjO8AWAr XYJjOKoDvEj6WMBI5i1uoxngmzTD9iGa3z1Nv763qpAVxVaynxxZayi2qjlVHKZtQ29q nQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3322nnuda4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:40 -0400 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJXZw1065636; Fri, 21 Aug 2020 15:56:39 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 3322nnud9v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:39 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsqXS013481; Fri, 21 Aug 2020 19:56:39 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma01wdc.us.ibm.com with ESMTP id 3304tm6m8d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:39 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuarO41484708 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:36 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1092A78063; Fri, 21 Aug 2020 19:56:36 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8ABB47805F; Fri, 21 Aug 2020 19:56:34 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:34 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 07/16] s390/vfio-ap: sysfs attribute to display the guest's matrix Date: Fri, 21 Aug 2020 15:56:07 -0400 Message-Id: <20200821195616.13554-8-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=3 lowpriorityscore=0 impostorscore=0 phishscore=0 priorityscore=1501 clxscore=1015 spamscore=0 mlxlogscore=999 malwarescore=0 bulkscore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210183 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The matrix of adapters and domains configured in a guest's CRYCB may differ from the matrix of adapters and domains assigned to the matrix mdev, so this patch introduces a sysfs attribute to display the matrix of a guest using the matrix mdev. For a matrix mdev denoted by $uuid, the crycb for a guest using the matrix mdev can be displayed as follows: cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix If a guest is not using the matrix mdev at the time the crycb is displayed, an error (ENODEV) will be returned. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 58 +++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index efb229033f9e..30bf23734af6 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1119,6 +1119,63 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr, } static DEVICE_ATTR_RO(matrix); +static ssize_t guest_matrix_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + char *bufpos = buf; + unsigned long apid; + unsigned long apqi; + unsigned long apid1; + unsigned long apqi1; + unsigned long napm_bits = matrix_mdev->shadow_apcb.apm_max + 1; + unsigned long naqm_bits = matrix_mdev->shadow_apcb.aqm_max + 1; + int nchars = 0; + int n; + + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return -ENODEV; + + apid1 = find_first_bit_inv(matrix_mdev->shadow_apcb.apm, napm_bits); + apqi1 = find_first_bit_inv(matrix_mdev->shadow_apcb.aqm, naqm_bits); + + mutex_lock(&matrix_dev->lock); + + if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) { + for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm, + napm_bits) { + for_each_set_bit_inv(apqi, + matrix_mdev->shadow_apcb.aqm, + naqm_bits) { + n = sprintf(bufpos, "%02lx.%04lx\n", apid, + apqi); + bufpos += n; + nchars += n; + } + } + } else if (apid1 < napm_bits) { + for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm, + napm_bits) { + n = sprintf(bufpos, "%02lx.\n", apid); + bufpos += n; + nchars += n; + } + } else if (apqi1 < naqm_bits) { + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, + naqm_bits) { + n = sprintf(bufpos, ".%04lx\n", apqi); + bufpos += n; + nchars += n; + } + } + + mutex_unlock(&matrix_dev->lock); + + return nchars; +} +static DEVICE_ATTR_RO(guest_matrix); + static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_assign_adapter.attr, &dev_attr_unassign_adapter.attr, @@ -1128,6 +1185,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_unassign_control_domain.attr, &dev_attr_control_domains.attr, &dev_attr_matrix.attr, + &dev_attr_guest_matrix.attr, NULL, }; From patchwork Fri Aug 21 19:56:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730477 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61F00618 for ; Fri, 21 Aug 2020 19:58:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A73B2072D for ; Fri, 21 Aug 2020 19:58:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ZNVaE7+y" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726760AbgHUT6w (ORCPT ); Fri, 21 Aug 2020 15:58:52 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:46072 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726996AbgHUT4p (ORCPT ); Fri, 21 Aug 2020 15:56:45 -0400 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJgJqe124541; Fri, 21 Aug 2020 15:56:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=/0XcHRYrFIAXLMZwZiNPHhNJdWzjT08xDt3mNNG0pkw=; b=ZNVaE7+ylLhdZGJ+N613gFk5EYcKMH+KzmQavINBtNPrxwUmLByWIxTqiSY5hscngx8M bviVxj0ycjkR8FXNVGDPg/aN5SQIoVDe02ZgJEgWdm146N2ITyczsw5r7hE5qBju7Oy1 XZtAf9SaNfvHlbTRUWC9vs4qxPVRZgNc4vuwknqKvcu1nXSuanW6GaRWYqql7u12WMnC nBUampFNtZX30WZj3Ms4R7OOFQ2jTeoQSDhMY+kzqEPnGJwhVeInvlUKCXGLrwU1nELW fOCKtZ+TQj9gSApojwFrpOIa0AnFnf8mdeZDQ6UCQnPIc4WzFae4GfmAjUCtE87HfDW4 Sw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332mna8916-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:41 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJggXM125127; Fri, 21 Aug 2020 15:56:41 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 332mna890r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:41 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsqXU013481; Fri, 21 Aug 2020 19:56:40 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma01wdc.us.ibm.com with ESMTP id 3304tm6m8n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:40 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJubiU524902 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:37 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AC2BC7805E; Fri, 21 Aug 2020 19:56:37 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3610078060; Fri, 21 Aug 2020 19:56:36 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:36 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 08/16] s390/vfio-ap: filter matrix for unavailable queue devices Date: Fri, 21 Aug 2020 15:56:08 -0400 Message-Id: <20200821195616.13554-9-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 priorityscore=1501 suspectscore=3 malwarescore=0 lowpriorityscore=0 impostorscore=0 mlxlogscore=999 clxscore=1015 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Even though APQNs for queues that are not in the host's AP configuration may be assigned to a matrix mdev, we do not want to set bits in the guest's APCB for APQNs that do not reference AP queue devices bound to the vfio_ap device driver. Ideally, it would be great if such APQNs could be filtered out before setting the bits in the guest's APCB; however, the architecture precludes filtering individual APQNs. Consequently, either the APID or APQI must be filtered. This patch introduces code to filter the APIDs or APQIs assigned to the matrix mdev's AP configuration before assigning them to the guest's AP configuration (i.e., APCB). We'll start by filtering the APIDs: If an APQN assigned to the matrix mdev's AP configuration does not reference a queue device bound to the vfio_ap device driver, the APID will be filtered out (i.e., not assigned to the guest's APCB). If every APID assigned to the matrix mdev is filtered out, then we'll try filtering the APQI's: If an APQN assigned to the matrix mdev's AP configuration does not reference a queue device bound to the vfio_ap device driver, the APQI will be filtered out (i.e., not assigned to the guest's APCB). In any case, if after filtering either the APIDs or APQIs there are any APQNs that can be assigned to the guest's APCB, they will be assigned and the CRYCB will be hot plugged into the guest. Example ======= APQNs bound to vfio_ap device driver: 04.0004 04.0047 04.0054 05.0005 05.0047 05.0054 Assignments to matrix mdev: APIDs APQIs -> APQNs 04 0004 04.0004 05 0005 04.0005 0047 04.0047 0054 04.0054 05.0004 05.0005 05.0047 04.0054 Filter APIDs: APID 04 will be filtered because APQN 04.0005 is not bound. APID 05 will be filtered because APQN 05.0004 is not bound. APQNs remaining: None Filter APQIs: APQI 04 will be filtered because APQN 05.0004 is not bound. APQI 05 will be filtered because APQN 04.0005 is not bound. APQNs remaining: 04.0047, 04.0054, 05.0047, 05.0054 APQNs 04.0047, 04.0054, 05.0047, 05.0054 will be assigned to the CRYCB and hot plugged into the KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 159 +++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 4 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 30bf23734af6..eaf4e9eab6cb 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -326,7 +326,7 @@ static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev) return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd); } -static void vfio_ap_mdev_commit_crycb(struct ap_matrix_mdev *matrix_mdev) +static void vfio_ap_mdev_commit_shadow_apcb(struct ap_matrix_mdev *matrix_mdev) { kvm_arch_crypto_set_masks(matrix_mdev->kvm, matrix_mdev->shadow_apcb.apm, @@ -597,6 +597,157 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, return 0; } +/** + * vfio_ap_mdev_filter_matrix + * + * Filter APQNs assigned to the matrix mdev that do not reference an AP queue + * device bound to the vfio_ap device driver. + * + * @matrix_mdev: the matrix mdev whose AP configuration is to be filtered + * @shadow_apcb: the shadow of the KVM guest's APCB (contains AP configuration + * for guest) + * @filter_apids: boolean value indicating whether the APQNs shall be filtered + * by APID (true) or by APQI (false). + * + * Returns the number of APQNs remaining after filtering is complete. + */ +static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, + struct ap_matrix *shadow_apcb, + bool filter_apids) +{ + unsigned long apid, apqi, apqn; + + memcpy(shadow_apcb, &matrix_mdev->matrix, sizeof(*shadow_apcb)); + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { + /* + * If the APID is not assigned to the host AP configuration, + * we can not assign it to the guest's AP configuration + */ + if (!test_bit_inv(apid, + (unsigned long *)matrix_dev->info.apm)) { + clear_bit_inv(apid, shadow_apcb->apm); + continue; + } + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, + AP_DOMAINS) { + /* + * If the APQI is not assigned to the host AP + * configuration, then it can not be assigned to the + * guest's AP configuration + */ + if (!test_bit_inv(apqi, (unsigned long *) + matrix_dev->info.aqm)) { + clear_bit_inv(apqi, shadow_apcb->aqm); + continue; + } + + /* + * If the APQN is not bound to the vfio_ap device + * driver, then we can't assign it to the guest's + * AP configuration. The AP architecture won't + * allow filtering of a single APQN, so if we're + * filtering APIDs, then filter the APID; otherwise, + * filter the APQI. + */ + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_queue(apqn)) { + if (filter_apids) + clear_bit_inv(apid, shadow_apcb->apm); + else + clear_bit_inv(apqi, shadow_apcb->aqm); + break; + } + } + + /* + * If we're filtering APQIs and all of them have been filtered, + * there's no need to continue filtering. + */ + if (!filter_apids) + if (bitmap_empty(shadow_apcb->aqm, AP_DOMAINS)) + break; + } + + return bitmap_weight(shadow_apcb->apm, AP_DEVICES) * + bitmap_weight(shadow_apcb->aqm, AP_DOMAINS); +} + +/** + * vfio_ap_mdev_config_shadow_apcb + * + * Configure the shadow of a KVM guest's APCB specifying the adapters, domains + * and control domains to be assigned to the guest. The shadow APCB will be + * configured after filtering the APQNs assigned to the matrix mdev that do not + * reference a queue device bound to the vfio_ap device driver. + * + * @matrix_mdev: the matrix mdev whose shadow APCB is to be configured. + * + * Returns true if the shadow APCB contents have been changed; otherwise, + * returns false. + */ +static bool vfio_ap_mdev_config_shadow_apcb(struct ap_matrix_mdev *matrix_mdev) +{ + int napm, naqm; + struct ap_matrix shadow_apcb; + + vfio_ap_matrix_init(&matrix_dev->info, &shadow_apcb); + napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES); + naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS); + + /* + * If there are no APIDs or no APQIs assigned to the matrix mdev, + * then no APQNs shall be assigned to the guest CRYCB. + */ + if ((napm != 0) || (naqm != 0)) { + /* + * Filter the APIDs assigned to the matrix mdev for APQNs that + * do not reference an AP queue device bound to the driver. + */ + napm = vfio_ap_mdev_filter_matrix(matrix_mdev, &shadow_apcb, + true); + /* + * If there are no APQNs that can be assigned to the guest's + * CRYCB after filtering, then try filtering the APQIs. + */ + if (napm == 0) { + naqm = vfio_ap_mdev_filter_matrix(matrix_mdev, + &shadow_apcb, false); + + /* + * If there are no APQNs that can be assigned to the + * matrix mdev after filtering the APQIs, then no APQNs + * shall be assigned to the guest's CRYCB. + */ + if (naqm == 0) { + bitmap_clear(shadow_apcb.apm, 0, AP_DEVICES); + bitmap_clear(shadow_apcb.aqm, 0, AP_DOMAINS); + } + } + } + + /* + * If the guest's AP configuration has not changed, then return + * indicating such. + */ + if (bitmap_equal(matrix_mdev->shadow_apcb.apm, shadow_apcb.apm, + AP_DEVICES) && + bitmap_equal(matrix_mdev->shadow_apcb.aqm, shadow_apcb.aqm, + AP_DOMAINS) && + bitmap_equal(matrix_mdev->shadow_apcb.adm, shadow_apcb.adm, + AP_DOMAINS)) + return false; + + /* + * Copy the changes to the guest's CRYCB, then return indicating that + * the guest's AP configuration has changed. + */ + memcpy(&matrix_mdev->shadow_apcb, &shadow_apcb, sizeof(shadow_apcb)); + + return true; +} + enum qlink_type { LINK_APID, LINK_APQI, @@ -1284,9 +1435,8 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (!vfio_ap_mdev_has_crycb(matrix_mdev)) return NOTIFY_DONE; - memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix, - sizeof(matrix_mdev->shadow_apcb)); - vfio_ap_mdev_commit_crycb(matrix_mdev); + if (vfio_ap_mdev_config_shadow_apcb(matrix_mdev)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); return NOTIFY_OK; } @@ -1396,6 +1546,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev) mutex_lock(&matrix_dev->lock); if (matrix_mdev->kvm) { kvm_arch_crypto_clear_masks(matrix_mdev->kvm); + vfio_ap_matrix_clear_masks(&matrix_mdev->shadow_apcb); matrix_mdev->kvm->arch.crypto.pqap_hook = NULL; vfio_ap_mdev_reset_queues(mdev); kvm_put_kvm(matrix_mdev->kvm); From patchwork Fri Aug 21 19:56:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0AAD8138C for ; Fri, 21 Aug 2020 19:58:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D93DE20738 for ; Fri, 21 Aug 2020 19:58:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="CKn9XGjf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727870AbgHUT60 (ORCPT ); Fri, 21 Aug 2020 15:58:26 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:46522 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727069AbgHUT4q (ORCPT ); Fri, 21 Aug 2020 15:56:46 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJnAv5108213; Fri, 21 Aug 2020 15:56:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=0kEUffz7tDVSapzHSNvJklQE4Zoi0fRfR0fAqmptAY8=; b=CKn9XGjfkUMR7xx8AyMP2M3+AXegf0B3wydoqZJWFD0BJMa0w0ynBDuTiJbawfEg7k7r Sv9Sy5+xIVqPsSveWLbuEOkKJ022IN8ZAvsn77CxEYZntSK6bnEdGnSpm6Dn1t6P8DJr LeZGxvifnd3s/JBro8AtvJrQzSeRsRKHy5/X6S4AE9hqZq3mPHC9IMHpRFM8X3SEpvmc DsA24EBCW5HgPYECe4+gFpVI4p8ynj5MYgdNlyK8LEDzI+XHPWi64iA5DNqxKJu67SYL saHXb/yi3U7QwWJocbrv3ajhQfoNF6AsKpHFC+apl9B/dAkFBLkv4tJjlqC95N/2tSF4 OA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332b8b074c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:44 -0400 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJoc8k115706; Fri, 21 Aug 2020 15:56:43 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 332b8b0743-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:43 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJte6t004808; Fri, 21 Aug 2020 19:56:42 GMT Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by ppma02dal.us.ibm.com with ESMTP id 3304cdedqp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:42 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuZMp28901824 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:35 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 505E878060; Fri, 21 Aug 2020 19:56:39 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D21D67805C; Fri, 21 Aug 2020 19:56:37 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:37 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 09/16] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Date: Fri, 21 Aug 2020 15:56:09 -0400 Message-Id: <20200821195616.13554-10-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 clxscore=1015 suspectscore=3 mlxscore=0 bulkscore=0 mlxlogscore=999 lowpriorityscore=0 priorityscore=1501 impostorscore=0 adultscore=0 spamscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210183 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The current implementation does not allow assignment of an AP adapter or domain to an mdev device if the APQNs resulting from the assignment do not reference AP queue devices that are bound to the vfio_ap device driver. This patch allows assignment of AP resources to the matrix mdev as long as the APQNs resulting from the assignment: 1. Are not reserved by the AP BUS for use by the zcrypt device drivers. 2. Are not assigned to another matrix mdev. The rationale behind this is twofold: 1. The AP architecture does not preclude assignment of APQNs to an AP configuration that are not available to the system. 2. APQNs that do not reference a queue device bound to the vfio_ap device driver will not be assigned to the guest's CRYCB, so the guest will not get access to queues not bound to the vfio_ap driver. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 212 +++++------------------------- 1 file changed, 35 insertions(+), 177 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index eaf4e9eab6cb..24fd47e43b80 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1,4 +1,3 @@ -// SPDX-License-Identifier: GPL-2.0+ /* * Adjunct processor matrix VFIO device driver callbacks. * @@ -420,122 +419,6 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = { NULL, }; -struct vfio_ap_queue_reserved { - unsigned long *apid; - unsigned long *apqi; - bool reserved; -}; - -/** - * vfio_ap_has_queue - * - * @dev: an AP queue device - * @data: a struct vfio_ap_queue_reserved reference - * - * Flags whether the AP queue device (@dev) has a queue ID containing the APQN, - * apid or apqi specified in @data: - * - * - If @data contains both an apid and apqi value, then @data will be flagged - * as reserved if the APID and APQI fields for the AP queue device matches - * - * - If @data contains only an apid value, @data will be flagged as - * reserved if the APID field in the AP queue device matches - * - * - If @data contains only an apqi value, @data will be flagged as - * reserved if the APQI field in the AP queue device matches - * - * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if - * @data does not contain either an apid or apqi. - */ -static int vfio_ap_has_queue(struct device *dev, void *data) -{ - struct vfio_ap_queue_reserved *qres = data; - struct ap_queue *ap_queue = to_ap_queue(dev); - ap_qid_t qid; - unsigned long id; - - if (qres->apid && qres->apqi) { - qid = AP_MKQID(*qres->apid, *qres->apqi); - if (qid == ap_queue->qid) - qres->reserved = true; - } else if (qres->apid && !qres->apqi) { - id = AP_QID_CARD(ap_queue->qid); - if (id == *qres->apid) - qres->reserved = true; - } else if (!qres->apid && qres->apqi) { - id = AP_QID_QUEUE(ap_queue->qid); - if (id == *qres->apqi) - qres->reserved = true; - } else { - return -EINVAL; - } - - return 0; -} - -/** - * vfio_ap_verify_queue_reserved - * - * @matrix_dev: a mediated matrix device - * @apid: an AP adapter ID - * @apqi: an AP queue index - * - * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device - * driver according to the following rules: - * - * - If both @apid and @apqi are not NULL, then there must be an AP queue - * device bound to the vfio_ap driver with the APQN identified by @apid and - * @apqi - * - * - If only @apid is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apid - * - * - If only @apqi is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apqi - * - * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL. - */ -static int vfio_ap_verify_queue_reserved(unsigned long *apid, - unsigned long *apqi) -{ - int ret; - struct vfio_ap_queue_reserved qres; - - qres.apid = apid; - qres.apqi = apqi; - qres.reserved = false; - - ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &qres, vfio_ap_has_queue); - if (ret) - return ret; - - if (qres.reserved) - return 0; - - return -EADDRNOTAVAIL; -} - -static int -vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, - unsigned long apid) -{ - int ret; - unsigned long apqi; - unsigned long nbits = matrix_mdev->matrix.aqm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(&apid, NULL); - - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - #define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \ "already assigned to %s" @@ -572,6 +455,11 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, DECLARE_BITMAP(aqm, AP_DOMAINS); list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) { + /* + * If either of the input masks belongs to the mdev to which an + * AP resource is being assigned, then we don't need to verify + * that mdev's masks. + */ if (matrix_mdev == lstdev) continue; @@ -597,6 +485,20 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, return 0; } +static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev, + unsigned long *mdev_apm, + unsigned long *mdev_aqm) +{ + DECLARE_BITMAP(apm, AP_DEVICES); + DECLARE_BITMAP(aqm, AP_DOMAINS); + + if (bitmap_and(apm, mdev_apm, ap_perms.apm, AP_DEVICES) && + bitmap_and(aqm, mdev_aqm, ap_perms.aqm, AP_DOMAINS)) + return -EADDRNOTAVAIL; + + return vfio_ap_mdev_verify_no_sharing(matrix_mdev, mdev_apm, mdev_aqm); +} + /** * vfio_ap_mdev_filter_matrix * @@ -882,33 +784,21 @@ static ssize_t assign_adapter_store(struct device *dev, if (apid > matrix_mdev->matrix.apm_max) return -ENODEV; - /* - * Set the bit in the AP mask (APM) corresponding to the AP adapter - * number (APID). The bits in the mask, from most significant to least - * significant bit, correspond to APIDs 0-255. - */ - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid); - if (ret) - goto done; - memset(apm, 0, sizeof(apm)); set_bit_inv(apid, apm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm, - matrix_mdev->matrix.aqm); - if (ret) - goto done; - + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev, apm, + matrix_mdev->matrix.aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); - ret = count; - -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_adapter); @@ -958,26 +848,6 @@ static ssize_t unassign_adapter_store(struct device *dev, } static DEVICE_ATTR_WO(unassign_adapter); -static int -vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev, - unsigned long apqi) -{ - int ret; - unsigned long apid; - unsigned long nbits = matrix_mdev->matrix.apm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(NULL, &apqi); - - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - /** * assign_domain_store * @@ -1031,28 +901,21 @@ static ssize_t assign_domain_store(struct device *dev, if (apqi > max_apqi) return -ENODEV; - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi); - if (ret) - goto done; - memset(aqm, 0, sizeof(aqm)); set_bit_inv(apqi, aqm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, - matrix_mdev->matrix.apm, aqm); - if (ret) - goto done; - + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev, matrix_mdev->matrix.apm, + aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); - ret = count; - -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_domain); @@ -1139,11 +1002,6 @@ static ssize_t assign_control_domain_store(struct device *dev, if (id > matrix_mdev->matrix.adm_max) return -ENODEV; - /* Set the bit in the ADM (bitmask) corresponding to the AP control - * domain number (id). The bits in the mask, from most significant to - * least significant, correspond to IDs 0 up to the one less than the - * number of control domains that can be assigned. - */ mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); mutex_unlock(&matrix_dev->lock); From patchwork Fri Aug 21 19:56:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730469 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E77C0618 for ; Fri, 21 Aug 2020 19:58:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C73C220738 for ; Fri, 21 Aug 2020 19:58:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="V06dXdir" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727116AbgHUT6Z (ORCPT ); Fri, 21 Aug 2020 15:58:25 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:50334 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726792AbgHUT4r (ORCPT ); Fri, 21 Aug 2020 15:56:47 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJV5mf160830; Fri, 21 Aug 2020 15:56:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=HxH8vAmUqzp7ZUnJmyOzBL08DszKTQ98e7Di1M7GFKQ=; b=V06dXdir/djvFLUb8P55wdVmM05mcrFDBiLmt+WSB3Wqj+BrHfz/JQLvY7tx6E6J149c U9gOerCOREvM9HiY31IJcYlwXDj59aRwK++M++ConkezJCXBZQ6xa4cyLKGs05V3CDSm PFuypSuf6OPssBFS8Tz3/FmgaxZgYv1nUeBSrT9ASTDv64Xf8kZhT5+PfutK3eW0wxFm oTI9S6/7Q1aKZxuMxNbgpaKxADzdNowkNnTFhL80hiBk4ugvwai01IIyCdVh3Vjv8OeI MKq2zeIlmykE5ItkvvRVGxGuE+izaBU3Cyh9+tJiN6UqpuEsZt5J5i+PWf6AQPHa/yGw Wg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 332j0vvbm0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:45 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJlPYG008045; Fri, 21 Aug 2020 15:56:45 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 332j0vvbkj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:45 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJtLls004542; Fri, 21 Aug 2020 19:56:44 GMT Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by ppma04dal.us.ibm.com with ESMTP id 3304ure9j7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:44 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJufZL34603382 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:41 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E1C1C7806A; Fri, 21 Aug 2020 19:56:40 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7651B7805E; Fri, 21 Aug 2020 19:56:39 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:39 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 10/16] s390/vfio-ap: allow configuration of matrix mdev in use by a KVM guest Date: Fri, 21 Aug 2020 15:56:10 -0400 Message-Id: <20200821195616.13554-11-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 spamscore=0 mlxscore=0 clxscore=1015 suspectscore=3 impostorscore=0 adultscore=0 mlxlogscore=999 lowpriorityscore=0 priorityscore=1501 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The current support for pass-through crypto adapters does not allow configuration of a matrix mdev when it is in use by a KVM guest. Let's allow AP resources - i.e., adapters, domains and control domains - to be assigned to or unassigned from a matrix mdev while it is in use by a guest. This is in preparation for the introduction of support for dynamic configuration of the AP matrix for a running KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 24fd47e43b80..cf3321eb239b 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -773,10 +773,6 @@ static ssize_t assign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -828,10 +824,6 @@ static ssize_t unassign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -891,10 +883,6 @@ static ssize_t assign_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; - /* If the guest is running, disallow assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -946,10 +934,6 @@ static ssize_t unassign_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -991,10 +975,6 @@ static ssize_t assign_control_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &id); if (ret) return ret; @@ -1036,10 +1016,6 @@ static ssize_t unassign_control_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_domid = matrix_mdev->matrix.adm_max; - /* If the guest is running, disallow un-assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &domid); if (ret) return ret; From patchwork Fri Aug 21 19:56:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730443 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 70CCE618 for ; Fri, 21 Aug 2020 19:57:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 487372076E for ; Fri, 21 Aug 2020 19:57:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="SVg2PvQj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726875AbgHUT45 (ORCPT ); Fri, 21 Aug 2020 15:56:57 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:61070 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726799AbgHUT4t (ORCPT ); Fri, 21 Aug 2020 15:56:49 -0400 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJVstG184128; Fri, 21 Aug 2020 15:56:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=7CalM92CIWFdGJt7DiCsFWIv1m0UY/uexaDEyJqq9oc=; b=SVg2PvQjkKyHoVdslKQqUrigEt9mYLN3YXeDk8P4kAt8FvRQuDxVpOt9sf2PPbZwaKou Sk2snGom0HIxwjzxOzHDxAtQBPbPerE+cl8Cm6VZndOGJnmNOkX7l9RObNeLlCXV0JjM pL11b0UrRDYAqTw8auUXVAD+g39apjp27ygc57TYJLvoaazJNryHI9mnmLNId/Td4wl6 8IZDbDgKerTk8ICBoq/8mQ6SiDHGkJgkm0Fh9Qbd6Egn5Ddw2K0kjWp73mfP5aV7QLuS 99+xyA2K7qJxV/Z2yfp623uFqnHFzz06ZqeMzTmG++ylSoJ7uAcugKYWRG5uQBgEdWEs ew== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 332jf9kn0w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:46 -0400 Received: from m0098414.ppops.net (m0098414.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJWVQh185744; Fri, 21 Aug 2020 15:56:46 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0b-001b2d01.pphosted.com with ESMTP id 332jf9kn03-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:46 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJscFM010511; Fri, 21 Aug 2020 19:56:45 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma05wdc.us.ibm.com with ESMTP id 3304themh6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:45 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJugIx29491502 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:42 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 863EE7805C; Fri, 21 Aug 2020 19:56:42 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1C3EE78069; Fri, 21 Aug 2020 19:56:41 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:40 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 11/16] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Date: Fri, 21 Aug 2020 15:56:11 -0400 Message-Id: <20200821195616.13554-12-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 clxscore=1015 impostorscore=0 adultscore=0 suspectscore=3 priorityscore=1501 lowpriorityscore=0 mlxscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's hot plug/unplug adapters, domains and control domains assigned to or unassigned from an AP matrix mdev device while it is in use by a guest per the following: * When the APID of an adapter is assigned to a matrix mdev in use by a KVM guest, the adapter will be hot plugged into the KVM guest as long as each APQN derived from the Cartesian product of the APID being assigned and the APQIs already assigned to the guest's CRYCB references a queue device bound to the vfio_ap device driver. * When the APID of an adapter is unassigned from a matrix mdev in use by a KVM guest, the adapter will be hot unplugged from the KVM guest. * When the APQI of a domain is assigned to a matrix mdev in use by a KVM guest, the domain will be hot plugged into the KVM guest as long as each APQN derived from the Cartesian product of the APQI being assigned and the APIDs already assigned to the guest's CRYCB references a queue device bound to the vfio_ap device driver. * When the APQI of a domain is unassigned from a matrix mdev in use by a KVM guest, the domain will be hot unplugged from the KVM guest * When the domain number of a control domain is assigned to a matrix mdev in use by a KVM guest, the control domain will be hot plugged into the KVM guest. * When the domain number of a control domain is unassigned from a matrix mdev in use by a KVM guest, the control domain will be hot unplugged from the KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 196 ++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index cf3321eb239b..2b01a8eb6ee7 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -731,6 +731,56 @@ static void vfio_ap_mdev_link_queues(struct ap_matrix_mdev *matrix_mdev, } } +static bool vfio_ap_mdev_assign_apqis_4_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + DECLARE_BITMAP(aqm, AP_DOMAINS); + unsigned long apqi, apqn; + + bitmap_copy(aqm, matrix_mdev->matrix.aqm, AP_DOMAINS); + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { + if (!test_bit_inv(apqi, + (unsigned long *) matrix_dev->info.aqm)) + clear_bit_inv(apqi, aqm); + + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + clear_bit_inv(apqi, aqm); + } + + if (bitmap_empty(aqm, AP_DOMAINS)) + return false; + + set_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + bitmap_copy(matrix_mdev->shadow_apcb.aqm, aqm, AP_DOMAINS); + + return true; +} + +static bool vfio_ap_mdev_assign_guest_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + unsigned long apqi, apqn; + + if (!vfio_ap_mdev_has_crycb(matrix_mdev) || + !test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm)) + return false; + + if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, AP_DOMAINS)) + return vfio_ap_mdev_assign_apqis_4_apid(matrix_mdev, apid); + + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS) { + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + return false; + } + + set_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + + return true; +} + /** * assign_adapter_store * @@ -792,12 +842,42 @@ static ssize_t assign_adapter_store(struct device *dev, } set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); + if (vfio_ap_mdev_assign_guest_apid(matrix_mdev, apid)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_adapter); +static bool vfio_ap_mdev_unassign_guest_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) { + clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + + /* + * If there are no APIDs assigned to the guest, then + * the guest will not have access to any queues, so + * let's also go ahead and unassign the APQIs. Keeping + * them around may yield unpredictable results during + * a probe that is not related to a host AP + * configuration change (i.e., an AP adapter is + * configured online). + */ + if (bitmap_empty(matrix_mdev->shadow_apcb.apm, + AP_DEVICES)) + bitmap_clear(matrix_mdev->shadow_apcb.aqm, 0, + AP_DOMAINS); + + return true; + } + } + + return false; +} + /** * unassign_adapter_store * @@ -834,12 +914,64 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid); + if (vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(unassign_adapter); +static bool vfio_ap_mdev_assign_apids_4_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + DECLARE_BITMAP(apm, AP_DEVICES); + unsigned long apid, apqn; + + bitmap_copy(apm, matrix_mdev->matrix.apm, AP_DEVICES); + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { + if (!test_bit_inv(apid, + (unsigned long *) matrix_dev->info.apm)) + clear_bit_inv(apqi, apm); + + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + clear_bit_inv(apid, apm); + } + + if (bitmap_empty(apm, AP_DEVICES)) + return false; + + set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); + bitmap_copy(matrix_mdev->shadow_apcb.apm, apm, AP_DEVICES); + + return true; +} + +static bool vfio_ap_mdev_assign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + unsigned long apid, apqn; + + if (!vfio_ap_mdev_has_crycb(matrix_mdev) || + !test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) + return false; + + if (bitmap_empty(matrix_mdev->shadow_apcb.apm, AP_DEVICES)) + return vfio_ap_mdev_assign_apids_4_apqi(matrix_mdev, apqi); + + for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm, AP_DEVICES) { + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_get_mdev_queue(matrix_mdev, apqn)) + return false; + } + + set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); + + return true; +} + /** * assign_domain_store * @@ -901,12 +1033,41 @@ static ssize_t assign_domain_store(struct device *dev, } set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); + if (vfio_ap_mdev_assign_guest_apqi(matrix_mdev, apqi)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_domain); +static bool vfio_ap_mdev_unassign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) { + clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); + + /* + * If there are no APQIs assigned to the guest, then + * the guest will not have access to any queues, so + * let's also go ahead and unassign the APIDs. Keeping + * them around may yield unpredictable results during + * a probe that is not related to a host AP + * configuration change (i.e., an AP adapter is + * configured online). + */ + if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, + AP_DOMAINS)) + bitmap_clear(matrix_mdev->shadow_apcb.apm, 0, + AP_DEVICES); + + return true; + } + } + + return false; +} /** * unassign_domain_store @@ -944,12 +1105,28 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi); + if (vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(unassign_domain); +static bool vfio_ap_mdev_assign_guest_cdom(struct ap_matrix_mdev *matrix_mdev, + unsigned long domid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (!test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) { + set_bit_inv(domid, matrix_mdev->shadow_apcb.adm); + + return true; + } + } + + return false; +} + /** * assign_control_domain_store * @@ -984,12 +1161,29 @@ static ssize_t assign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); + if (vfio_ap_mdev_assign_guest_cdom(matrix_mdev, id)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_control_domain); +static bool +vfio_ap_mdev_unassign_guest_cdom(struct ap_matrix_mdev *matrix_mdev, + unsigned long domid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) { + clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm); + + return true; + } + } + + return false; +} + /** * unassign_control_domain_store * @@ -1024,6 +1218,8 @@ static ssize_t unassign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv(domid, matrix_mdev->matrix.adm); + if (vfio_ap_mdev_unassign_guest_cdom(matrix_mdev, domid)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; From patchwork Fri Aug 21 19:56:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730457 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 808BC138C for ; Fri, 21 Aug 2020 19:57:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54D172072D for ; Fri, 21 Aug 2020 19:57:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="sYA0symB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727785AbgHUT47 (ORCPT ); Fri, 21 Aug 2020 15:56:59 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:53470 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726828AbgHUT4t (ORCPT ); Fri, 21 Aug 2020 15:56:49 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJX7if154616; Fri, 21 Aug 2020 15:56:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=swpEaVYxiwQ1DDluF6gAxR7mvE6r5gDPnjm29zDFU7U=; b=sYA0symBOnCCD53ZAbzjfUmLiu9QX42c7XIqqXfjkEDbR+A9vBfUrzgued6lNL3O9JrH vFGp/UvzFvAQXNXvMmFGd0DziyOUOsd5R0HWMPZQdf+Px5aIFfI7V5OMY0EC8CGzMxZs xoWRIL2rmOFXRBOKD7usAJPvwzOMhc97FKALfXY/M/KSmnSZDk+qqDzm3+qIMR49SLe7 ouECgSrU4y1bEZrkEKOq3GgADgD+FI+a+Oj48wGp6ubkNJoRkCywLELmincFhouqtz4B BP/c0G4yvaJCpH0WX5YO0JJ4zc7hNq87NWbantFrDOFGinyGszDhkDGmCiX5lWut+RmH Zw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3328e9n49d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:48 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJsjwj010652; Fri, 21 Aug 2020 15:56:48 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 3328e9n48v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:48 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJscD8010508; Fri, 21 Aug 2020 19:56:47 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma05wdc.us.ibm.com with ESMTP id 3304themha-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:47 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJufvS14745966 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:41 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 31A8278068; Fri, 21 Aug 2020 19:56:44 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B560678066; Fri, 21 Aug 2020 19:56:42 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:42 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 12/16] s390/zcrypt: Notify driver on config changed and scan complete callbacks Date: Fri, 21 Aug 2020 15:56:12 -0400 Message-Id: <20200821195616.13554-13-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 spamscore=0 malwarescore=0 mlxscore=0 lowpriorityscore=0 suspectscore=3 adultscore=0 priorityscore=1501 impostorscore=0 phishscore=0 clxscore=1015 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210183 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This patch intruduces an extension to the ap bus to notify drivers on crypto config changed and bus scan complete events. Two new callbacks are introduced for ap_drivers: void (*on_config_changed)(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); void (*on_scan_complete)(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); Both callbacks are optional. Both callbacks are only triggered when QCI information is available (facility bit 12): * The on_config_changed callback is invoked at the start of the AP bus scan function when it determines that the host AP configuration information has changed since the previous scan. This is done by storing an old and current QCI info struct and comparing them. If there is any difference, the callback is invoked. Note that when the AP bus scan detects that AP adapters or domains have been removed from the host's AP configuration, it will remove the associated devices from the AP bus subsystem's device model. This callback gives the device driver a chance to respond to the removal of the AP devices in bulk rather than one at a time as its remove callback is invoked. It will also allow the device driver to do any any cleanup prior to giving control back to the bus piecemeal. This is particularly important for the vfio_ap driver because there may be guests using the queues at the time. * The on_scan_complete callback is invoked after the ap bus scan is complete if the host AP configuration data has changed. Note that when the AP bus scan detects that adapters or domains have been added to the host's configuration, it will create new devices in the AP bus subsystem's device model. This callback also allows the driver to process all of the new devices in bulk. Please note that changes to the apmask and aqmask do not trigger these two callbacks since the bus scan function is not invoked by changes to those masks. Signed-off-by: Harald Freudenberger Signed-off-by: Tony Krowiak --- drivers/s390/crypto/ap_bus.c | 85 +++++++++++++++++++++++++++++++++++- drivers/s390/crypto/ap_bus.h | 12 +++++ 2 files changed, 96 insertions(+), 1 deletion(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index db27bd931308..cbf4c4d2e573 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -73,8 +73,10 @@ struct ap_perms ap_perms; EXPORT_SYMBOL(ap_perms); DEFINE_MUTEX(ap_perms_mutex); EXPORT_SYMBOL(ap_perms_mutex); +DEFINE_MUTEX(ap_config_lock); static struct ap_config_info *ap_qci_info; +static struct ap_config_info *ap_qci_info_old; /* * AP bus related debug feature things. @@ -1412,6 +1414,52 @@ static int __match_queue_device_with_queue_id(struct device *dev, const void *da && AP_QID_QUEUE(to_ap_queue(dev)->qid) == (int)(long) data; } +/* Helper function for notify_config_changed */ +static int __drv_notify_config_changed(struct device_driver *drv, void *data) +{ + struct ap_driver *ap_drv = to_ap_drv(drv); + + if (try_module_get(drv->owner)) { + if (ap_drv->on_config_changed) + ap_drv->on_config_changed(ap_qci_info, + ap_qci_info_old); + module_put(drv->owner); + } + + return 0; +} + +/* Notify all drivers about an qci config change */ +static inline void notify_config_changed(void) +{ + bus_for_each_drv(&ap_bus_type, NULL, NULL, + __drv_notify_config_changed); +} + +/* Helper function for notify_scan_complete */ +static int __drv_notify_scan_complete(struct device_driver *drv, void *data) +{ + struct ap_driver *ap_drv = to_ap_drv(drv); + + if (try_module_get(drv->owner)) { + if (ap_drv->on_scan_complete) + ap_drv->on_scan_complete(ap_qci_info, + ap_qci_info_old); + module_put(drv->owner); + } + + return 0; +} + +/* Notify all drivers about bus scan complete */ +static inline void notify_scan_complete(void) +{ + bus_for_each_drv(&ap_bus_type, NULL, NULL, + __drv_notify_scan_complete); +} + + + /* * Helper function for ap_scan_bus(). * Does the scan bus job for the given adapter id. @@ -1565,15 +1613,44 @@ static void _ap_scan_bus_adapter(int id) put_device(&ac->ap_dev.device); } +static int ap_config_changed(void) +{ + int cfg_chg = 0; + + if (ap_qci_info) { + if (!ap_qci_info_old) { + ap_qci_info_old = kzalloc(sizeof(*ap_qci_info_old), + GFP_KERNEL); + if (!ap_qci_info_old) + return 0; + } else { + memcpy(ap_qci_info_old, ap_qci_info, + sizeof(struct ap_config_info)); + } + ap_fetch_qci_info(ap_qci_info); + cfg_chg = memcmp(ap_qci_info, + ap_qci_info_old, + sizeof(struct ap_config_info)) != 0; + } + + return cfg_chg; +} + /** * ap_scan_bus(): Scan the AP bus for new devices * Runs periodically, workqueue timer (ap_config_time) */ static void ap_scan_bus(struct work_struct *unused) { - int id; + int id, config_changed = 0; ap_fetch_qci_info(ap_qci_info); + mutex_lock(&ap_config_lock); + + /* config change notify */ + config_changed = ap_config_changed(); + if (config_changed) + notify_config_changed(); ap_select_domain(); AP_DBF(DBF_DEBUG, "%s running\n", __func__); @@ -1582,6 +1659,12 @@ static void ap_scan_bus(struct work_struct *unused) for (id = 0; id < AP_DEVICES; id++) _ap_scan_bus_adapter(id); + /* scan complete notify */ + if (config_changed) + notify_scan_complete(); + + mutex_unlock(&ap_config_lock); + /* check if there is at least one queue available with default domain */ if (ap_domain_index >= 0) { struct device *dev = diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 48c57b3d53a0..3fc743ac549c 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -137,6 +137,18 @@ struct ap_driver { int (*probe)(struct ap_device *); void (*remove)(struct ap_device *); bool (*in_use)(unsigned long *apm, unsigned long *aqm); + /* + * Called at the start of the ap bus scan function when + * the crypto config information (qci) has changed. + */ + void (*on_config_changed)(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); + /* + * Called at the end of the ap bus scan function when + * the crypto config information (qci) has changed. + */ + void (*on_scan_complete)(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) From patchwork Fri Aug 21 19:56:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730445 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F075618 for ; Fri, 21 Aug 2020 19:57:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BD37207DA for ; Fri, 21 Aug 2020 19:57:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="LNk4Mjkp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726670AbgHUT5A (ORCPT ); Fri, 21 Aug 2020 15:57:00 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:42352 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727114AbgHUT4y (ORCPT ); Fri, 21 Aug 2020 15:56:54 -0400 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJW0xh072501; Fri, 21 Aug 2020 15:56:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=zUKrPLCkB5FaxwcHmeszOQg0x2nYAn5+lXouE59uvkk=; b=LNk4MjkpE2AcGT4dkEmjRLY+ogfTklAjHdU4D0k580eSIxu8RK5D31PuANYlKuBO1Z/f InbWLVHxgJK7RR94FlJJWeOWFQr37+CQLJ2y3lOZVcDe9/CSaqLglTSMmTp0ah9HvKic qxq1Cg9ZAuxPd1Jg/HAGbkR6mijEmsF5rTjQJV4Tofy1dvt9DwDwJliKwIZ3nN/5GKza VQ0MwyrsMoa4otSaIstzk7592eu4BO1yQ4BipGQQQ8oLGgEJtrBe2ZQabFRwBNXfcN6R Sk9n64oN0o3NkbHl/MIsfj2qQLc3LqTrmzrkV7PTusY34AZu7FHUWbmJZnZmi9yEjmGc jw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 332b8a93xv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:50 -0400 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJk6pc109957; Fri, 21 Aug 2020 15:56:49 -0400 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0b-001b2d01.pphosted.com with ESMTP id 332b8a93xm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:49 -0400 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsLTk025060; Fri, 21 Aug 2020 19:56:49 GMT Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by ppma02wdc.us.ibm.com with ESMTP id 3304sdepkb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:49 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJuhmM15532414 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:43 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E07737806B; Fri, 21 Aug 2020 19:56:45 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 61A3B7805C; Fri, 21 Aug 2020 19:56:44 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:44 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak , kernel test robot Subject: [PATCH v10 13/16] s390/vfio-ap: handle host AP config change notification Date: Fri, 21 Aug 2020 15:56:13 -0400 Message-Id: <20200821195616.13554-14-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=3 spamscore=0 priorityscore=1501 bulkscore=0 impostorscore=0 adultscore=0 malwarescore=0 mlxlogscore=999 clxscore=1015 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Implements the driver callback invoked by the AP bus when the host AP configuration has changed. Since this callback is invoked prior to unbinding a device from its device driver, the vfio_ap driver will respond by unplugging the AP adapters, domains and control domains removed from the host's AP configuration from the guests using them. Signed-off-by: Tony Krowiak Reported-by: kernel test robot --- drivers/s390/crypto/vfio_ap_drv.c | 5 +- drivers/s390/crypto/vfio_ap_ops.c | 147 ++++++++++++++++++++++++-- drivers/s390/crypto/vfio_ap_private.h | 7 +- 3 files changed, 146 insertions(+), 13 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index aae5b3d8e3fa..ea0a7603e886 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -115,9 +115,11 @@ static int vfio_ap_matrix_dev_create(void) /* Fill in config info via PQAP(QCI), if available */ if (test_facility(12)) { - ret = ap_qci(&matrix_dev->info); + ret = ap_qci(&matrix_dev->config_info); if (ret) goto matrix_alloc_err; + memcpy(&matrix_dev->config_info_prev, &matrix_dev->config_info, + sizeof(struct ap_config_info)); } mutex_init(&matrix_dev->lock); @@ -177,6 +179,7 @@ static int __init vfio_ap_init(void) vfio_ap_drv.remove = vfio_ap_queue_dev_remove; vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; + vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 2b01a8eb6ee7..e002d556abab 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -347,7 +347,9 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) } matrix_mdev->mdev = mdev; - vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); + vfio_ap_matrix_init(&matrix_dev->config_info, &matrix_mdev->matrix); + vfio_ap_matrix_init(&matrix_dev->config_info, + &matrix_mdev->shadow_apcb); hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; @@ -526,8 +528,8 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, * If the APID is not assigned to the host AP configuration, * we can not assign it to the guest's AP configuration */ - if (!test_bit_inv(apid, - (unsigned long *)matrix_dev->info.apm)) { + if (!test_bit_inv(apid, (unsigned long *) + matrix_dev->config_info.apm)) { clear_bit_inv(apid, shadow_apcb->apm); continue; } @@ -540,7 +542,7 @@ static int vfio_ap_mdev_filter_matrix(struct ap_matrix_mdev *matrix_mdev, * guest's AP configuration */ if (!test_bit_inv(apqi, (unsigned long *) - matrix_dev->info.aqm)) { + matrix_dev->config_info.aqm)) { clear_bit_inv(apqi, shadow_apcb->aqm); continue; } @@ -594,7 +596,7 @@ static bool vfio_ap_mdev_config_shadow_apcb(struct ap_matrix_mdev *matrix_mdev) int napm, naqm; struct ap_matrix shadow_apcb; - vfio_ap_matrix_init(&matrix_dev->info, &shadow_apcb); + vfio_ap_matrix_init(&matrix_dev->config_info, &shadow_apcb); napm = bitmap_weight(matrix_mdev->matrix.apm, AP_DEVICES); naqm = bitmap_weight(matrix_mdev->matrix.aqm, AP_DOMAINS); @@ -741,7 +743,7 @@ static bool vfio_ap_mdev_assign_apqis_4_apid(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { if (!test_bit_inv(apqi, - (unsigned long *) matrix_dev->info.aqm)) + (unsigned long *)matrix_dev->config_info.aqm)) clear_bit_inv(apqi, aqm); apqn = AP_MKQID(apid, apqi); @@ -764,7 +766,7 @@ static bool vfio_ap_mdev_assign_guest_apid(struct ap_matrix_mdev *matrix_mdev, unsigned long apqi, apqn; if (!vfio_ap_mdev_has_crycb(matrix_mdev) || - !test_bit_inv(apid, (unsigned long *)matrix_dev->info.apm)) + !test_bit_inv(apid, (unsigned long *)matrix_dev->config_info.apm)) return false; if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, AP_DOMAINS)) @@ -931,8 +933,8 @@ static bool vfio_ap_mdev_assign_apids_4_apqi(struct ap_matrix_mdev *matrix_mdev, bitmap_copy(apm, matrix_mdev->matrix.apm, AP_DEVICES); for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { - if (!test_bit_inv(apid, - (unsigned long *) matrix_dev->info.apm)) + if (!test_bit_inv(apid, (unsigned long *) + matrix_dev->config_info.apm)) clear_bit_inv(apqi, apm); apqn = AP_MKQID(apid, apqi); @@ -955,7 +957,7 @@ static bool vfio_ap_mdev_assign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, unsigned long apid, apqn; if (!vfio_ap_mdev_has_crycb(matrix_mdev) || - !test_bit_inv(apqi, (unsigned long *)matrix_dev->info.aqm)) + !test_bit_inv(apqi, (unsigned long *)matrix_dev->config_info.aqm)) return false; if (bitmap_empty(matrix_mdev->shadow_apcb.apm, AP_DEVICES)) @@ -1702,7 +1704,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue) void vfio_ap_mdev_remove_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; - int apid, apqi; + unsigned long apid, apqi; mutex_lock(&matrix_dev->lock); q = dev_get_drvdata(&queue->ap_dev.device); @@ -1727,3 +1729,126 @@ bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) return in_use; } + +/** + * vfio_ap_mdev_unassign_apids + * + * @matrix_mdev: The matrix mediated device + * + * @aqm: A bitmap with 256 bits. Each bit in the map represents an APID from 0 + * to 255 (with the leftmost bit corresponding to APID 0). + * + * Unassigns each APID specified in @aqm that is assigned to the shadow CRYCB + * of @matrix_mdev. Returns true if at least one APID is unassigned; otherwise, + * returns false. + */ +static bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apm_unassign) +{ + unsigned long apid; + bool unassigned = false; + + /* + * If the matrix mdev is not in use by a KVM guest, return indicating + * that no APIDs have been unassigned. + */ + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return false; + + for_each_set_bit_inv(apid, apm_unassign, AP_DEVICES) { + unassigned |= vfio_ap_mdev_unassign_guest_apid(matrix_mdev, + apid); + } + + return unassigned; +} + +/** + * vfio_ap_mdev_unassign_apqis + * + * @matrix_mdev: The matrix mediated device + * + * @aqm: A bitmap with 256 bits. Each bit in the map represents an APQI from 0 + * to 255 (with the leftmost bit corresponding to APQI 0). + * + * Unassigns each APQI specified in @aqm that is assigned to the shadow CRYCB + * of @matrix_mdev. Returns true if at least one APQI is unassigned; otherwise, + * returns false. + */ +static bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev, + unsigned long *aqm_unassign) +{ + unsigned long apqi; + bool unassigned = false; + + /* + * If the matrix mdev is not in use by a KVM guest, return indicating + * that no APQIs have been unassigned. + */ + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return false; + + for_each_set_bit_inv(apqi, aqm_unassign, AP_DOMAINS) { + unassigned |= vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, + apqi); + } + + return unassigned; +} + +void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info) +{ + bool unassigned; + int ap_remove, aq_remove; + struct ap_matrix_mdev *matrix_mdev; + DECLARE_BITMAP(apm_unassign, AP_DEVICES); + DECLARE_BITMAP(aqm_unassign, AP_DOMAINS); + + unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm; + + if (matrix_dev->flags & AP_MATRIX_CFG_CHG) { + WARN_ONCE(1, "AP host configuration change already reported"); + return; + } + + memcpy(&matrix_dev->config_info, new_config_info, + sizeof(struct ap_config_info)); + memcpy(&matrix_dev->config_info_prev, old_config_info, + sizeof(struct ap_config_info)); + + cur_apm = (unsigned long *)matrix_dev->config_info.apm; + cur_aqm = (unsigned long *)matrix_dev->config_info.aqm; + prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm; + prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm; + + ap_remove = bitmap_andnot(apm_unassign, prev_apm, cur_apm, AP_DEVICES); + aq_remove = bitmap_andnot(aqm_unassign, prev_aqm, cur_aqm, AP_DOMAINS); + + mutex_lock(&matrix_dev->lock); + matrix_dev->flags |= AP_MATRIX_CFG_CHG; + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + continue; + + unassigned = false; + + if (ap_remove) + if (bitmap_intersects(matrix_mdev->shadow_apcb.apm, + apm_unassign, AP_DEVICES)) + if (vfio_ap_mdev_unassign_apids(matrix_mdev, + apm_unassign)) + unassigned = true; + if (aq_remove) + if (bitmap_intersects(matrix_mdev->shadow_apcb.aqm, + aqm_unassign, AP_DOMAINS)) + if (vfio_ap_mdev_unassign_apqis(matrix_mdev, + aqm_unassign)) + unassigned = true; + + if (unassigned) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); + } + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 055bce6d45db..fc8629e28ad3 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -40,10 +40,13 @@ struct ap_matrix_dev { struct device device; atomic_t available_instances; - struct ap_config_info info; + struct ap_config_info config_info; + struct ap_config_info config_info_prev; struct list_head mdev_list; struct mutex lock; struct ap_driver *vfio_ap_drv; + #define AP_MATRIX_CFG_CHG (1UL << 0) + unsigned long flags; }; extern struct ap_matrix_dev *matrix_dev; @@ -108,5 +111,7 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue); void vfio_ap_mdev_remove_queue(struct ap_queue *queue); bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); +void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Fri Aug 21 19:56:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730461 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 60BFA618 for ; Fri, 21 Aug 2020 19:58:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A1192072D for ; Fri, 21 Aug 2020 19:58:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="N7BfQ02a" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727840AbgHUT55 (ORCPT ); Fri, 21 Aug 2020 15:57:57 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:48848 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727115AbgHUT4y (ORCPT ); Fri, 21 Aug 2020 15:56:54 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJV5u9160871; Fri, 21 Aug 2020 15:56:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=cPZ05bXzaPw8Ei780BDr6olkJ4CuSCeO6RCm9ad9JHo=; b=N7BfQ02auMn4wWSliBjxLfyeH/qYn+yh/Gu82nBn6G6QzNxC/thzIVipZ04v+HXlSzQI EpzwM7T1M7Su1Kt/u6wM8UgMdOYBbh94dt9F8vYXmmPto10YYBnku60Swg0kaBGlO9WA eiZQ9VnpFUhdoGA8QsYGcZubDEG5IPrv1PWOJkPca5Pwi2dW67m7lIaoKIhhKes65Ytm 3BjQWrgO0c6Wqgqj8J5awxKvOLP0B8ZEukELVew3Q1wzdq5LVZXfbXieiHkXORYK/Wzn wy/tNMbV8qexujoA5RchUZ/RsLa112S9es4FduDWkS9FAcPF6GOhC1+aDSievrSIooV3 pw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 332j0vvbq5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:51 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJVr0E163302; Fri, 21 Aug 2020 15:56:51 -0400 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com with ESMTP id 332j0vvbps-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:51 -0400 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsr6L019738; Fri, 21 Aug 2020 19:56:51 GMT Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by ppma04wdc.us.ibm.com with ESMTP id 3304uf6ne2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:50 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJulMu66716072 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:47 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9A8CE7805C; Fri, 21 Aug 2020 19:56:47 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1C03178064; Fri, 21 Aug 2020 19:56:46 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:45 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 14/16] s390/vfio-ap: handle AP bus scan completed notification Date: Fri, 21 Aug 2020 15:56:14 -0400 Message-Id: <20200821195616.13554-15-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_08:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 spamscore=0 mlxscore=0 clxscore=1015 suspectscore=3 impostorscore=0 adultscore=0 mlxlogscore=999 lowpriorityscore=0 priorityscore=1501 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Implements the driver callback invoked by the AP bus when the AP bus scan has completed. Since this callback is invoked after binding the newly added devices to their respective device drivers, the vfio_ap driver will attempt to plug the adapters, domains and control domains into each guest using a matrix mdev to which they are assigned. Keep in mind that an adapter or domain can be plugged in only if each APQN with the APID of the adapter or the APQI of the domain references a queue device bound to the vfio_ap device driver. Consequently, not all newly added adapters and domains will necessarily get hot plugged. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 1 + drivers/s390/crypto/vfio_ap_ops.c | 110 +++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 2 + 3 files changed, 110 insertions(+), 3 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index ea0a7603e886..21bfae928be5 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -180,6 +180,7 @@ static int __init vfio_ap_init(void) vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed; + vfio_ap_drv.on_scan_complete = vfio_ap_on_scan_complete; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index e002d556abab..e6480f31a42b 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -616,14 +616,13 @@ static bool vfio_ap_mdev_config_shadow_apcb(struct ap_matrix_mdev *matrix_mdev) * CRYCB after filtering, then try filtering the APQIs. */ if (napm == 0) { - naqm = vfio_ap_mdev_filter_matrix(matrix_mdev, - &shadow_apcb, false); - /* * If there are no APQNs that can be assigned to the * matrix mdev after filtering the APQIs, then no APQNs * shall be assigned to the guest's CRYCB. */ + naqm = vfio_ap_mdev_filter_matrix(matrix_mdev, + &shadow_apcb, false); if (naqm == 0) { bitmap_clear(shadow_apcb.apm, 0, AP_DEVICES); bitmap_clear(shadow_apcb.aqm, 0, AP_DOMAINS); @@ -1758,6 +1757,16 @@ static bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apid, apm_unassign, AP_DEVICES) { unassigned |= vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid); + /* + * If the APID is not assigned to the matrix mdev's shadow + * CRYCB, continue with the next APID. + */ + if (!test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) + continue; + + /* Unassign the APID from the matrix mdev's shadow CRYCB */ + clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + unassigned = true; } return unassigned; @@ -1791,6 +1800,17 @@ static bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev, for_each_set_bit_inv(apqi, aqm_unassign, AP_DOMAINS) { unassigned |= vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi); + + /* + * If the APQI is not assigned to the matrix mdev's shadow + * CRYCB, continue with the next APQI + */ + if (!test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) + continue; + + /* Unassign the APQI from the matrix mdev's shadow CRYCB */ + clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); + unassigned = true; } return unassigned; @@ -1852,3 +1872,87 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, } mutex_unlock(&matrix_dev->lock); } + +bool vfio_ap_mdev_assign_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apm_assign) +{ + unsigned long apid; + bool assigned = false; + + for_each_set_bit_inv(apid, apm_assign, AP_DEVICES) + if (test_bit_inv(apid, matrix_mdev->matrix.apm)) + if (vfio_ap_mdev_assign_guest_apid(matrix_mdev, apid)) + assigned = true; + + return assigned; +} + +bool vfio_ap_mdev_assign_apqis(struct ap_matrix_mdev *matrix_mdev, + unsigned long *aqm_assign) +{ + unsigned long apqi; + bool assigned = false; + + for_each_set_bit_inv(apqi, aqm_assign, AP_DOMAINS) + if (test_bit_inv(apqi, matrix_mdev->matrix.aqm)) + if (vfio_ap_mdev_assign_guest_apqi(matrix_mdev, apqi)) + assigned = true; + + return assigned; +} + +void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info) +{ + struct ap_matrix_mdev *matrix_mdev; + DECLARE_BITMAP(apm_assign, AP_DEVICES); + DECLARE_BITMAP(aqm_assign, AP_DOMAINS); + int ap_add, aq_add; + bool assign; + unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm; + + /* + * If we are not in the middle of a host configuration change scan it is + * likely that the vfio_ap driver was loaded mid-scan, so let's handle + * this scenario by calling the vfio_ap_on_cfg_changed function which + * gets called at the start of an AP bus scan when the host AP + * configuration has changed. + */ + if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG)) + vfio_ap_on_cfg_changed(new_config_info, old_config_info); + + cur_apm = (unsigned long *)matrix_dev->config_info.apm; + cur_aqm = (unsigned long *)matrix_dev->config_info.aqm; + + prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm; + prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm; + + ap_add = bitmap_andnot(apm_assign, cur_apm, prev_apm, AP_DEVICES); + aq_add = bitmap_andnot(aqm_assign, cur_aqm, prev_aqm, AP_DOMAINS); + + mutex_lock(&matrix_dev->lock); + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + continue; + + assign = false; + + if (ap_add) + if (bitmap_intersects(matrix_mdev->matrix.apm, + apm_assign, AP_DEVICES)) + assign |= vfio_ap_mdev_assign_apids(matrix_mdev, + apm_assign); + + if (aq_add) + if (bitmap_intersects(matrix_mdev->matrix.aqm, + aqm_assign, AP_DOMAINS)) + assign |= vfio_ap_mdev_assign_apqis(matrix_mdev, + aqm_assign); + + if (assign) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); + } + + matrix_dev->flags &= ~AP_MATRIX_CFG_CHG; + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index fc8629e28ad3..da1754fd4f66 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -113,5 +113,7 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue); bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); +void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Fri Aug 21 19:56:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730463 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAA9D618 for ; Fri, 21 Aug 2020 19:58:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87005207BB for ; Fri, 21 Aug 2020 19:58:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Fs84d+aU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727019AbgHUT54 (ORCPT ); Fri, 21 Aug 2020 15:57:56 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58800 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727120AbgHUT4z (ORCPT ); Fri, 21 Aug 2020 15:56:55 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJVjWW093266; Fri, 21 Aug 2020 15:56:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=sXk2PXkPC1GfrnqnUAtOL+4rS0fUxLSGoIxQuRimknU=; b=Fs84d+aUXGaEnMcxD5qrabSl73CTTyxUo+Axr7lOuyIzrhQI8MlQ0ovNOSVHr5WdIMvY 3ieE5+s7JJj/5sWb8CCIxpttUBiVfT6fs7HkdAJfr4+aRWTVlNt4SxXtJX8RaOihFNwv 9MFXHNj3e5U92wfzuliP3iu98g4UDdRC7ayxSJI++iPkoLOROpPlXsm7cK0170QSxjsj 2dgZZ6jnnvSOQuMNrtVyD/5P0Ov1GjlFlNyn9brY/op7mDR5ZKXawHVmwQsVKjCU5zDo RoqW8HBHTDYw0rOTRj6DwdfK+WShbk43/U8RFf68kLkVIQbB+9ikQmeK1hOxMCoWdcRc tw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332kse9gjb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:53 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJZSP0114205; Fri, 21 Aug 2020 15:56:53 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0a-001b2d01.pphosted.com with ESMTP id 332kse9gj2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:53 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsPj5005410; Fri, 21 Aug 2020 19:56:52 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma03wdc.us.ibm.com with ESMTP id 3304ceps01-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:52 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJunlF58458448 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:49 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 42B9C7805E; Fri, 21 Aug 2020 19:56:49 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C0C897805F; Fri, 21 Aug 2020 19:56:47 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:47 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 15/16] s390/vfio-ap: handle probe/remove not due to host AP config changes Date: Fri, 21 Aug 2020 15:56:15 -0400 Message-Id: <20200821195616.13554-16-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=3 adultscore=0 mlxlogscore=999 phishscore=0 clxscore=1015 malwarescore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org AP queue devices are probed or removed for reasons other than changes to the host AP configuration. For example: * The state of an AP adapter can be dynamically changed from standby to online via the SE or by execution of the SCLP Configure AP command. When the state changes, each queue device associated with the card device representing the adapter will get created and probed. * The state of an AP adapter can be dynamically changed from online to standby via the SE or by execution of the SCLP Deconfigure AP command. When the state changes, each queue device associated with the card device representing the adapter will get removed. * Each queue device associated with a card device will get removed when the type of the AP adapter represented by the card device dynamically changes. * Each queue device associated with a card device will get removed when the status of the queue represented by the queue device changes from operating to check stop. * AP queue devices can be manually bound to or unbound from the vfio_ap device driver by a root user via the sysfs bind/unbind attributes of the driver. In response to a queue device probe or remove that is not the result of a change to the host's AP configuration, if a KVM guest is using the matrix mdev to which the APQN of the queue device is assigned, the vfio_ap device driver must respond accordingly. In an ideal world, the queue device being probed would be hot plugged into the guest. Likewise, the queue corresponding to the queue device being removed would be hot unplugged from the guest. Unfortunately, the AP architecture precludes plugging or unplugging individual queues, so let's handle the probe or remove of an AP queue device as follows: Handling Probe -------------- There are two requirements that must be met in order to give a guest access to the queue corresponding to the queue device being probed: * Each APQN derived from the APID of the queue device and the APQIs of the domains already assigned to the guest's AP configuration must reference a queue device bound to the vfio_ap device driver. * Each APQN derived from the APQI of the queue device and the APIDs of the adapters assigned to the guest's AP configuration must reference a queue device bound to the vfio_ap device driver. If the above conditions are met, the APQN will be assigned to the guest's AP configuration and the guest will be given access to the queue. Handling Remove --------------- Since the AP architecture precludes us from taking access to an individual queue from a guest, we are left with the choice of taking access away from either the adapter or the domain to which the queue is connected. Access to the adapter will be taken away because it is likely that most of the time, the remove callback will be invoked because the adapter state has transitioned from online to standby. In such a case, no queue connected to the adapter will be available to access. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 84 +++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index e6480f31a42b..b6a1e280991d 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1682,6 +1682,61 @@ static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q) } } +static bool vfio_ap_mdev_assign_shadow_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + unsigned long apqi; + + for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, + matrix_mdev->shadow_apcb.aqm_max + 1) { + if (!vfio_ap_get_queue(AP_MKQID(apid, apqi))) + return false; + } + + set_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + + return true; +} + +static bool vfio_ap_mdev_assign_shadow_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + unsigned long apid; + + for_each_set_bit_inv(apid, matrix_mdev->shadow_apcb.apm, + matrix_mdev->shadow_apcb.apm_max + 1) { + if (!vfio_ap_get_queue(AP_MKQID(apid, apqi))) + return false; + } + + set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); + + return true; +} + +static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q) +{ + bool commit = false; + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); + + if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + return; + + if (!test_bit_inv(apid, q->matrix_mdev->matrix.apm) || + !test_bit_inv(apqi, q->matrix_mdev->matrix.aqm)) + return; + + if (!test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm)) + commit |= vfio_ap_mdev_assign_shadow_apid(q->matrix_mdev, apid); + + if (!test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm)) + commit |= vfio_ap_mdev_assign_shadow_apqi(q->matrix_mdev, apqi); + + if (commit) + vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev); +} + int vfio_ap_mdev_probe_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; @@ -1695,11 +1750,35 @@ int vfio_ap_mdev_probe_queue(struct ap_queue *queue) q->apqn = queue->qid; q->saved_isc = VFIO_AP_ISC_INVALID; vfio_ap_queue_link_mdev(q); + /* Make sure we're not in the middle of an AP configuration change. */ + if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG)) + vfio_ap_mdev_hot_plug_queue(q); mutex_unlock(&matrix_dev->lock); return 0; } +void vfio_ap_mdev_hot_unplug_queue(struct vfio_ap_queue *q) +{ + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); + + if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + return; + + /* + * If the APQN is assigned to the guest, then let's + * go ahead and unplug the adapter since the + * architecture does not provide a means to unplug + * an individual queue. + */ + if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm) && + test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm)) { + if (vfio_ap_mdev_unassign_guest_apid(q->matrix_mdev, apid)) + vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev); + } +} + void vfio_ap_mdev_remove_queue(struct ap_queue *queue) { struct vfio_ap_queue *q; @@ -1707,6 +1786,11 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) mutex_lock(&matrix_dev->lock); q = dev_get_drvdata(&queue->ap_dev.device); + + /* Make sure we're not in the middle of an AP configuration change. */ + if (!(matrix_dev->flags & AP_MATRIX_CFG_CHG)) + vfio_ap_mdev_hot_unplug_queue(q); + dev_set_drvdata(&queue->ap_dev.device, NULL); apid = AP_QID_CARD(q->apqn); apqi = AP_QID_QUEUE(q->apqn); From patchwork Fri Aug 21 19:56:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11730455 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6B41E138C for ; Fri, 21 Aug 2020 19:57:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1DA8F207DA for ; Fri, 21 Aug 2020 19:57:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ipHl5L1z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726904AbgHUT5l (ORCPT ); Fri, 21 Aug 2020 15:57:41 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:50286 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727770AbgHUT5C (ORCPT ); Fri, 21 Aug 2020 15:57:02 -0400 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07LJg8Ed124330; Fri, 21 Aug 2020 15:56:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=BLK4ydrEhHjUNAahYUJV/TK9BaunSDrHi1EGPEBJ9x4=; b=ipHl5L1z2OpDqIR8crrRSBzpjgh9L4PCtcWCen1V2nnDWHICr/6E+aTbe7EBjwthKY1f EVXSYWvPy1NKWcevaxhovmlYjm8VjwNmwde5WqgogX14ekRi/her+ssbbieP1uVpXLKf Uz31TU4ykrdiEt86VX4ff+872P7IxxqvFNH5xC0boi/L+1k0WQHQj/Epqo1G8/68xRK3 4VN+Lhn5ZOsnyoVODkCjT6uwZjUHk/bZCptiVP7I11gfyMwV6XV5+dZmJWmy/m/th1PI J1drm3mKjLED7U7JrkV4hHQAz2SrMAWeHmxTGHm8UKASDxlRAPGrmXvy7yW9M5ES+2QR 4g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 332mna898b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:55 -0400 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 07LJgvU7125691; Fri, 21 Aug 2020 15:56:54 -0400 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0a-001b2d01.pphosted.com with ESMTP id 332mna8980-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 15:56:54 -0400 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07LJsjfc019696; Fri, 21 Aug 2020 19:56:54 GMT Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by ppma04wdc.us.ibm.com with ESMTP id 3304uf6ne8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Aug 2020 19:56:54 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07LJukFN53871042 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 21 Aug 2020 19:56:46 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 068307805E; Fri, 21 Aug 2020 19:56:51 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 698F27805F; Fri, 21 Aug 2020 19:56:49 +0000 (GMT) Received: from cpe-172-100-175-116.stny.res.rr.com.com (unknown [9.85.191.76]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Fri, 21 Aug 2020 19:56:49 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, imbrenda@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v10 16/16] s390/vfio-ap: update docs to include dynamic config support Date: Fri, 21 Aug 2020 15:56:16 -0400 Message-Id: <20200821195616.13554-17-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200821195616.13554-1-akrowiak@linux.ibm.com> References: <20200821195616.13554-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-21_09:2020-08-21,2020-08-21 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 priorityscore=1501 suspectscore=3 malwarescore=0 lowpriorityscore=0 impostorscore=0 mlxlogscore=999 clxscore=1015 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008210178 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Update the documentation in vfio-ap.rst to include information about the AP dynamic configuration support (i.e., hot plug of adapters, domains and control domains via the matrix mediated device's sysfs assignment attributes). Signed-off-by: Tony Krowiak --- Documentation/s390/vfio-ap.rst | 362 ++++++++++++++++++++++++++------- 1 file changed, 285 insertions(+), 77 deletions(-) diff --git a/Documentation/s390/vfio-ap.rst b/Documentation/s390/vfio-ap.rst index e15436599086..8907aeca8fb7 100644 --- a/Documentation/s390/vfio-ap.rst +++ b/Documentation/s390/vfio-ap.rst @@ -253,7 +253,7 @@ The process for reserving an AP queue for use by a KVM guest is: 1. The administrator loads the vfio_ap device driver 2. The vfio-ap driver during its initialization will register a single 'matrix' device with the device core. This will serve as the parent device for - all mediated matrix devices used to configure an AP matrix for a guest. + all matrix mediated devices used to configure an AP matrix for a guest. 3. The /sys/devices/vfio_ap/matrix device is created by the device core 4. The vfio_ap device driver will register with the AP bus for AP queue devices of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap @@ -269,7 +269,7 @@ The process for reserving an AP queue for use by a KVM guest is: default zcrypt cex4queue driver. 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for it. -9. The administrator creates a passthrough type mediated matrix device to be +9. The administrator creates a passthrough type matrix mediated device to be used by a guest 10. The administrator assigns the adapters, usage domains and control domains to be exclusively used by a guest. @@ -279,14 +279,14 @@ Set up the VFIO mediated device interfaces The VFIO AP device driver utilizes the common interface of the VFIO mediated device core driver to: -* Register an AP mediated bus driver to add a mediated matrix device to and +* Register an AP mediated bus driver to add a matrix mediated device to and remove it from a VFIO group. -* Create and destroy a mediated matrix device -* Add a mediated matrix device to and remove it from the AP mediated bus driver -* Add a mediated matrix device to and remove it from an IOMMU group +* Create and destroy a matrix mediated device +* Add a matrix mediated device to and remove it from the AP mediated bus driver +* Add a matrix mediated device to and remove it from an IOMMU group The following high-level block diagram shows the main components and interfaces -of the VFIO AP mediated matrix device driver:: +of the VFIO AP matrix mediated device driver:: +-------------+ | | @@ -351,29 +351,37 @@ matrix device. This attribute group identifies the user-defined sysfs attributes of the mediated device. When a device is registered with the VFIO mediated device framework, the sysfs attribute files identified in the 'mdev_attr_groups' - structure will be created in the mediated matrix device's directory. The - sysfs attributes for a mediated matrix device are: + structure will be created in the matrix mediated device's directory. The + sysfs attributes for a matrix mediated device are: assign_adapter / unassign_adapter: Write-only attributes for assigning/unassigning an AP adapter to/from the - mediated matrix device. To assign/unassign an adapter, the APID of the + matrix mediated device. To assign/unassign an adapter, the APID of the adapter is echoed to the respective attribute file. assign_domain / unassign_domain: Write-only attributes for assigning/unassigning an AP usage domain to/from - the mediated matrix device. To assign/unassign a domain, the domain + the matrix mediated device. To assign/unassign a domain, the domain number of the usage domain is echoed to the respective attribute file. matrix: - A read-only file for displaying the APQNs derived from the cross product - of the adapter and domain numbers assigned to the mediated matrix device. + A read-only file for displaying the APQNs derived from the Cartesian + product of the adapter and domain numbers assigned to the mediated matrix + device. + guest_matrix: + A read-only file for displaying the APQNs derived from the Cartesian + product of the adapter and domain numbers assigned to the APM and AQM + fields respectively of the KVM guest's CRYCB. This will differ from the + matrix if any APQNs assigned to the matrix mediated device do not + reference a queue device bound to the vfio_ap device driver (i.e., the + queue is not in the AP configuration). assign_control_domain / unassign_control_domain: Write-only attributes for assigning/unassigning an AP control domain - to/from the mediated matrix device. To assign/unassign a control domain, + to/from the matrix mediated device. To assign/unassign a control domain, the ID of the domain to be assigned/unassigned is echoed to the respective attribute file. control_domains: A read-only file for displaying the control domain numbers assigned to the - mediated matrix device. + matrix mediated device. * functions: @@ -385,7 +393,7 @@ matrix device. domains assigned via the corresponding sysfs attributes files remove: - deallocates the mediated matrix device's ap_matrix_mdev structure. This will + deallocates the matrix mediated device's ap_matrix_mdev structure. This will be allowed only if a running guest is not using the mdev. * callback interfaces @@ -397,7 +405,7 @@ matrix device. for the mdev matrix device to the MDEV bus. Access to the KVM structure used to configure the KVM guest is provided via this callback. The KVM structure, is used to configure the guest's access to the AP matrix defined via the - mediated matrix device's sysfs attribute files. + matrix mediated device's sysfs attribute files. release: unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix device and deconfigures the guest's AP matrix. @@ -410,11 +418,49 @@ function is called when QEMU connects to KVM. The guest's AP matrix is configured via it's CRYCB by: * Setting the bits in the APM corresponding to the APIDs assigned to the - mediated matrix device via its 'assign_adapter' interface. + matrix mediated device via its 'assign_adapter' interface. * Setting the bits in the AQM corresponding to the domains assigned to the - mediated matrix device via its 'assign_domain' interface. + matrix mediated device via its 'assign_domain' interface. * Setting the bits in the ADM corresponding to the domain dIDs assigned to the - mediated matrix device via its 'assign_control_domains' interface. + matrix mediated device via its 'assign_control_domains' interface. + +The linux device model precludes passing a device through to a KVM guest that +is not bound to the device driver facilitating its pass-through. Consequently, +an APQN that does not reference a queue device bound to the vfio_ap device +driver will not be assigned to a KVM guest's CRYCB. The AP architecture, +however, does not provide a means to filter individual APQNs from the guest's +CRYCB, so the following logic is employed to filter them: + +* Filter the APQNs assigned to the matrix mediated device by APID. + + To filter APQNs by APID, each APQN derived from the Cartesian product of the + adapter numbers (APID) and domain numbers (APQI) assigned to the mdev is + examined and if any one of them does not reference a queue device bound to the + vfio_ap device driver, the adapter will not be plugged into the guest (i.e., + the bit corresponding to its APID will not be set in the APM of the guest's + CRYCB). + + If at least one adapter is plugged into the guest, then all domains assigned + to the mdev will also be plugged into the guest (i.e., the bits corresponding + to the APQIs of the domains assigned to the mdev will be set in the AQM field + of the guest's CRYCB). + +* Filter the APQNs assigned to the matrix mediated device by APQI. + + The APQNs will be filtered by APQI if filtering by APID does not result in any + adapters or domains getting plugged into the guest. + + To filter APQNs by APQI, each APQN derived from the Cartesian product of the + adapter numbers (APID) and domain numbers (APQI) assigned to the mdev is + examined and if any one of them does not reference a queue device bound to the + vfio_ap device driver, the domain will not be plugged into the guest (i.e., + the bit corresponding to its APQI will not be set in the AQM of the guest's + CRYCB). + + If at least one domain is plugged into the guest, then all adapters assigned + to the mdev will also be plugged into the guest (i.e., the bits corresponding + to the APIDs of the adapters assigned to the mdev will be set in the APM field + of the guest's CRYCB). The CPU model features for AP ----------------------------- @@ -435,6 +481,10 @@ available to a KVM guest via the following CPU model features: can be made available to the guest only if it is available on the host (i.e., facility bit 12 is set). +4. apqi: Indicates AP queue interrupts are available on the guest. This facility + can be made available to the guest only if it is available on the host (i.e., + facility bit 65 is set). + Note: If the user chooses to specify a CPU model different than the 'host' model to QEMU, the CPU model features and facilities need to be turned on explicitly; for example:: @@ -444,7 +494,7 @@ explicitly; for example:: A guest can be precluded from using AP features/facilities by turning them off explicitly; for example:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off + /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off Note: If the APFT facility is turned off (apft=off) for the guest, the guest will not see any AP devices. The zcrypt device drivers that register for type 10 @@ -530,40 +580,56 @@ These are the steps: 2. Secure the AP queues to be used by the three guests so that the host can not access them. To secure them, there are two sysfs files that specify - bitmasks marking a subset of the APQN range as 'usable by the default AP - queue device drivers' or 'not usable by the default device drivers' and thus - available for use by the vfio_ap device driver'. The location of the sysfs - files containing the masks are:: + bitmasks marking a subset of the APQN range as usable only by the default AP + queue device drivers. All remaining APQNs are available available for use by + any other device driver. The vfio_ap device driver is currently the only + non-default device driver. The location of the sysfs files containing the + masks are:: /sys/bus/ap/apmask /sys/bus/ap/aqmask The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs - (APID). Each bit in the mask, from left to right (i.e., from most significant - to least significant bit in big endian order), corresponds to an APID from - 0-255. If a bit is set, the APID is marked as usable only by the default AP - queue device drivers; otherwise, the APID is usable by the vfio_ap - device driver. + (APID). Each bit in the mask, from left to right corresponds to an APID from + 0-255. If a bit is set, the APID is marked as available to the default AP + queue device drivers. The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes - (APQI). Each bit in the mask, from left to right (i.e., from most significant - to least significant bit in big endian order), corresponds to an APQI from - 0-255. If a bit is set, the APQI is marked as usable only by the default AP - queue device drivers; otherwise, the APQI is usable by the vfio_ap device - driver. + (APQI). Each bit in the mask, from left to right corresponds to an APQI from + 0-255. If a bit is set, the APQI is marked as available to the default AP + queue device drivers. + + The Cartesian product of the APIDs corresponding to the bits set in the + apmask and the APQIs corresponding to the bits set in the aqmask comprise + the subset of APQNs that can be used only by the host default device drivers. + All other APQNs are available to the non-default device drivers such as the + vfio_ap driver. + + Take, for example, the following masks:: + + apmask: + 0x7d00000000000000000000000000000000000000000000000000000000000000 + + aqmask: + 0x8000000000000000000000000000000000000000000000000000000000000000 + + The masks indicate: - Take, for example, the following mask:: + * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default + device drivers. - 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff + * Domain 0 is available for use by the host default device drivers - It indicates: + * The subset of APQNs available for use only by the default host device + drivers are: - 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6 - belong to the vfio_ap device driver's pool. + (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0) + + * All other APQNs are available for use by the non-default device drivers. The APQN of each AP queue device assigned to the linux host is checked by the - AP bus against the set of APQNs derived from the cross product of APIDs - and APQIs marked as usable only by the default AP queue device drivers. If a + AP bus against the set of APQNs derived from the Cartesian product of APIDs + and APQIs marked as available to the default AP queue device drivers. If a match is detected, only the default AP queue device drivers will be probed; otherwise, the vfio_ap device driver will be probed. @@ -627,6 +693,16 @@ These are the steps: default drivers pool: adapter 0-15, domain 1 alternate drivers pool: adapter 16-255, domains 0, 2-255 + Note ***: + Changing a mask such that one or more APQNs will be taken from a matrix + mediated device (see below) will fail with an error (EADDRINUSE). The error + is logged to the kernel ring buffer which can be viewed with the 'dmesg' + command. The output identifies each APQN flagged as 'in use' and the matrix + mediated device to which it is assigned; for example: + + Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804 + Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4 + Securing the APQNs for our example ---------------------------------- To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, @@ -684,7 +760,7 @@ Securing the APQNs for our example /sys/devices/vfio_ap/matrix/ --- [mdev_supported_types] - ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) + ------ [vfio_ap-passthrough] (passthrough matrix mediated device type) --------- create --------- [devices] @@ -775,17 +851,18 @@ Securing the APQNs for our example higher than the maximum is specified, the operation will terminate with an error (ENODEV). - * All APQNs that can be derived from the adapter ID and the IDs of - the previously assigned domains must be bound to the vfio_ap device - driver. If no domains have yet been assigned, then there must be at least - one APQN with the specified APID bound to the vfio_ap driver. If no such - APQNs are bound to the driver, the operation will terminate with an - error (EADDRNOTAVAIL). + * All APQNs that can be derived from the Cartesian product of the APID of the + adapter being assigned and the APQIs of the previously assigned domains + must be available to the vfio_ap device driver as specified in the sysfs + /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even one APQN + is reserved for use by the host device driver, the operation will terminate + with an error (EADDRNOTAVAIL). - No APQN that can be derived from the adapter ID and the IDs of the - previously assigned domains can be assigned to another mediated matrix - device. If an APQN is assigned to another mediated matrix device, the - operation will terminate with an error (EADDRINUSE). + * No APQN that can be derived from the Cartesian product of the APID of the + adapter being assigned and the APQIs of the previously assigned domains can + be assigned to another matrix mediated device. If even one APQN is assigned + to another matrix mediated device, the operation will terminate with an + error (EADDRINUSE). In order to successfully assign a domain: @@ -794,17 +871,18 @@ Securing the APQNs for our example higher than the maximum is specified, the operation will terminate with an error (ENODEV). - * All APQNs that can be derived from the domain ID and the IDs of - the previously assigned adapters must be bound to the vfio_ap device - driver. If no domains have yet been assigned, then there must be at least - one APQN with the specified APQI bound to the vfio_ap driver. If no such - APQNs are bound to the driver, the operation will terminate with an - error (EADDRNOTAVAIL). + * All APQNs that can be derived from the Cartesian product of the APQI of the + domain being assigned and the APIDs of the previously assigned adapters + must be available to the vfio_ap device driver as specified in the sysfs + /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even one APQN + is reserved for use by the host device driver, the operation will terminate + with an error (EADDRNOTAVAIL). - No APQN that can be derived from the domain ID and the IDs of the - previously assigned adapters can be assigned to another mediated matrix - device. If an APQN is assigned to another mediated matrix device, the - operation will terminate with an error (EADDRINUSE). + * No APQN that can be derived from the Cartesian product of the APQI of the + domain being assigned and the APIDs of the previously assigned adapters can + be assigned to another matrix mediated device. If even one APQN is assigned + to another matrix mediated device, the operation will terminate with an + error (EADDRINUSE). In order to successfully assign a control domain, the domain number specified must represent a value from 0 up to the maximum domain number @@ -813,22 +891,22 @@ Securing the APQNs for our example 5. Start Guest1:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ... 7. Start Guest2:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ... 7. Start Guest3:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ... -When the guest is shut down, the mediated matrix devices may be removed. +When the guest is shut down, the matrix mediated devices may be removed. -Using our example again, to remove the mediated matrix device $uuid1:: +Using our example again, to remove the matrix mediated device $uuid1:: /sys/devices/vfio_ap/matrix/ --- [mdev_supported_types] @@ -851,16 +929,146 @@ remove it if no guest will use it during the remaining lifetime of the linux host. If the mdev matrix device is removed, one may want to also reconfigure the pool of adapters and queues reserved for use by the default drivers. +Hot plug support: +================ +An adapter, domain or control domain may be hot plugged into a running KVM +guest by assigning it to the matrix mediated device being used by the guest. +Control domains will always be hot plugged; however, an adapter or domain will +be hot plugged only if each new APQN resulting from its assignment +references a queue device bound to the vfio_ap device driver as described +below. + +When an adapter is assigned to a matrix mediated device in use by a KVM guest: + +* If no domains have yet been plugged into the KVM guest: + + Hot plug the adapter and every domain previously assigned to the mdev if each + APQN derived from the Cartesian product of the APID of the adapter being + assigned and the APQIs of the domains previously assigned references a queue + device bound to the vfio_ap device driver. + +* If one or more domains have previously been plugged into the guest: + + Hot plug the adapter if each APQN derived from the Cartesian product of the + APID of the adapter being assigned and the APQIs of the domains already + plugged into the guest references a queue device bound to the vfio_ap device + driver. + +When a domain is assigned to a matrix mediated device in use by a KVM guest: + +* If no adapters have yet been plugged into the KVM guest: + + Hot plug the domain and every adapter previously assigned to the mdev if each + APQN derived from the Cartesian product of the APIDs of the adapters + previously assigned and the APQI of the domain being assigned references a + queue device bound to the vfio_ap device driver. + +* If one or more adapters have previously been plugged into the guest: + + Hot plug the domain if each APQN derived from the Cartesian product of the + APIDs of the adapters already plugged into the guest and the APQI of the + domain being assigned references a queue device bound to the vfio_ap device + driver. + +Over-provisioning of AP queues for a KVM guest: +============================================== +Over-provisioning is defined herein as the assignment of adapters or domains to +a matrix mediated device that do not reference AP devices in the host's AP +configuration. The idea here is that when the adapter or domain becomes +available, it will be automatically hot-plugged into the KVM guest using +the matrix mediated device to which it is assigned as long as each new APQN +resulting from plugging it in references a queue device bound to the vfio_ap +device driver. + Limitations =========== -* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN - to the default drivers pool of a queue that is still assigned to a mediated - device in use by a guest. It is incumbent upon the administrator to - ensure there is no mediated device in use by a guest to which the APQN is - assigned lest the host be given access to the private data of the AP queue - device such as a private key configured specifically for the guest. +Live guest migration is not supported for guests using AP devices without +intervention by a system administrator. Before a KVM guest can be migrated, +the matrix mediated device must be removed. Unfortunately, it can not be +removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while +the mdev is in use by a KVM guest. If the guest is being emulated by QEMU, +its mdev can be hot unplugged from the guest in one of two ways: + +1. If the KVM guest was started with libvirt, you can hot unplug the mdev via + the following commands: + + virsh detach-device + + For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from + the guest named 'my-guest': + + virsh detach-device my-guest ~/config/my-guest-hostdev.xml + + The contents of my-guest-hostdev.xml: + + + +
+ + + + + virsh qemu-monitor-command --hmp "device-del " + + For example, to hot unplug the matrix mediated device identified on the + qemu command line with 'id=hostdev0' from the guest named 'my-guest': + + virsh qemu-monitor-command my-guest --hmp "device_del hostdev0" + +2. A matrix mediated device can be hot unplugged by attaching the qemu monitor + to the guest and using the following qemu monitor command: + + (QEMU) device-del id= + + For example, to hot unplug the matrix mediated device that was specified + on the qemu command line with 'id=hostdev0' when the guest was started: + + (QEMU) device-del id=hostdev0 + +After live migration of the KVM guest completes, an AP configuration can be +restored to the KVM guest by hot plugging a matrix mediated device on the target +system into the guest in one of two ways: + +1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated + device into the guest via the following virsh commands: + + virsh attach-device + + For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into + the guest named 'my-guest': + + virsh attach-device my-guest ~/config/my-guest-hostdev.xml + + The contents of my-guest-hostdev.xml: + + + +
+ + + + + virsh qemu-monitor-command --hmp \ + "device_add vfio-ap,sysfsdev=,id=" + + For example, to hot plug the matrix mediated device + 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with + device-id hostdev0: + + virsh qemu-monitor-command my-guest --hmp \ + "device_add vfio-ap,\ + sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ + id=hostdev0" + +2. A matrix mediated device can be hot plugged by attaching the qemu monitor + to the guest and using the following qemu monitor command: + + (qemu) device_add "vfio-ap,sysfsdev=,id=" -* Dynamically modifying the AP matrix for a running guest (which would amount to - hot(un)plug of AP devices for the guest) is currently not supported + For example, to plug the matrix mediated device + 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id + hostdev0: -* Live guest migration is not supported for guests using AP devices. + (QEMU) device-add "vfio-ap,\ + sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ + id=hostdev0" \ No newline at end of file