From patchwork Thu Oct 22 17:11:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61770C388F7 for ; Thu, 22 Oct 2020 17:12:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB1BB24630 for ; Thu, 22 Oct 2020 17:12:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="o/kz1s2I" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2898702AbgJVRM2 (ORCPT ); Thu, 22 Oct 2020 13:12:28 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:56276 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2506063AbgJVRM1 (ORCPT ); Thu, 22 Oct 2020 13:12:27 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH2Mah030283; Thu, 22 Oct 2020 13:12:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=9iz41qkKcI5UFyaLbfBJOVav71tdPjHy80191QGlZaE=; b=o/kz1s2Ig5M8/fQTmipfV5LItIYzun7+q8tt8Zn/1zw00ahvzXHgHHdvi5GsoW9PWINV /yMDlSoRMo9Ce62YH7WDKUbU56X59G2nub1bbCL8YlqGnXaZ+FRbmVWqWLd7VRn50JGT 4iC+6wtV10cUAqgpgtGi4MpMhybz+SfiXr4aOTf4K/Ck6J1eZIIib1mR5mtW+8aZnX8/ i/KYiwlB9cx19V0iibaA+SUzLLjz7ZZQVzMnTxp2l6jVxDf+xTSAKugVtQBNbqR5c5y0 8na6p8dIK1CtZI/kduw1mtebIuSQV14Q8wxQTpRoO8W27WXD1Z38i/mCjVSNx7rsgXyX iw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b6n8r6k1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:25 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH2Sch030989; Thu, 22 Oct 2020 13:12:25 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b6n8r6jn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:25 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCLav015337; Thu, 22 Oct 2020 17:12:24 GMT Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by ppma01dal.us.ibm.com with ESMTP id 347r89vb9q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:24 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCKCn7864646 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:21 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D7A2178064; Thu, 22 Oct 2020 17:12:20 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3A9E37805C; Thu, 22 Oct 2020 17:12:19 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:19 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 01/14] s390/vfio-ap: No need to disable IRQ after queue reset Date: Thu, 22 Oct 2020 13:11:56 -0400 Message-Id: <20201022171209.19494-2-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 adultscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 impostorscore=0 spamscore=0 mlxlogscore=973 malwarescore=0 phishscore=0 mlxscore=0 suspectscore=2 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The queues assigned to a matrix mediated device are currently reset when: * The VFIO_DEVICE_RESET ioctl is invoked * The mdev fd is closed by userspace (QEMU) * The mdev is removed from sysfs. Immediately after the reset of a queue, a call is made to disable interrupts for the queue. This is entirely unnecessary because the reset of a queue disables interrupts, so this will be removed. Since interrupt processing may have been enabled by the guest, it may also be necessary to clean up the resources used for interrupt processing. Part of the cleanup operation requires a reference to KVM, so a check is also being added to ensure the reference to KVM exists. The reason is because the release callback - invoked when userspace closes the mdev fd - removes the reference to KVM. When the remove callback - called when the mdev is removed from sysfs - is subsequently invoked, there will be no reference to KVM when the cleanup is performed. This patch will also do a bit of refactoring due to the fact that the remove callback, implemented in vfio_ap_drv.c, disables the queue after resetting it. Instead of the remove callback making a call into the vfio_ap_ops.c to clean up the resources used for interrupt processing, let's move the probe and remove callbacks into the vfio_ap_ops.c file keep all code related to managing queues in a single file. Signed-off-by: Tony Krowiak Reported-by: kernel test robot --- drivers/s390/crypto/vfio_ap_drv.c | 45 +------------------ drivers/s390/crypto/vfio_ap_ops.c | 63 +++++++++++++++++++-------- drivers/s390/crypto/vfio_ap_private.h | 7 +-- 3 files changed, 52 insertions(+), 63 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index be2520cc010b..73bd073fd5d3 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -43,47 +43,6 @@ static struct ap_device_id ap_queue_ids[] = { MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids); -/** - * vfio_ap_queue_dev_probe: - * - * Allocate a vfio_ap_queue structure and associate it - * with the device as driver_data. - */ -static int vfio_ap_queue_dev_probe(struct ap_device *apdev) -{ - struct vfio_ap_queue *q; - - q = kzalloc(sizeof(*q), GFP_KERNEL); - if (!q) - return -ENOMEM; - dev_set_drvdata(&apdev->device, q); - q->apqn = to_ap_queue(&apdev->device)->qid; - q->saved_isc = VFIO_AP_ISC_INVALID; - return 0; -} - -/** - * vfio_ap_queue_dev_remove: - * - * Takes the matrix lock to avoid actions on this device while removing - * Free the associated vfio_ap_queue structure - */ -static void vfio_ap_queue_dev_remove(struct ap_device *apdev) -{ - struct vfio_ap_queue *q; - int apid, apqi; - - mutex_lock(&matrix_dev->lock); - q = dev_get_drvdata(&apdev->device); - dev_set_drvdata(&apdev->device, NULL); - apid = AP_QID_CARD(q->apqn); - apqi = AP_QID_QUEUE(q->apqn); - vfio_ap_mdev_reset_queue(apid, apqi, 1); - vfio_ap_irq_disable(q); - kfree(q); - mutex_unlock(&matrix_dev->lock); -} - static void vfio_ap_matrix_dev_release(struct device *dev) { struct ap_matrix_dev *matrix_dev = dev_get_drvdata(dev); @@ -186,8 +145,8 @@ static int __init vfio_ap_init(void) return ret; memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv)); - vfio_ap_drv.probe = vfio_ap_queue_dev_probe; - vfio_ap_drv.remove = vfio_ap_queue_dev_remove; + vfio_ap_drv.probe = vfio_ap_mdev_probe_queue; + vfio_ap_drv.remove = vfio_ap_mdev_remove_queue; vfio_ap_drv.ids = ap_queue_ids; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index e0bde8518745..c471832f0a30 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -119,7 +119,8 @@ static void vfio_ap_wait_for_irqclear(int apqn) */ static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q) { - if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev) + if (q->saved_isc != VFIO_AP_ISC_INVALID && q->matrix_mdev && + q->matrix_mdev->kvm) kvm_s390_gisc_unregister(q->matrix_mdev->kvm, q->saved_isc); if (q->saved_pfn && q->matrix_mdev) vfio_unpin_pages(mdev_dev(q->matrix_mdev->mdev), @@ -144,7 +145,7 @@ static void vfio_ap_free_aqic_resources(struct vfio_ap_queue *q) * Returns if ap_aqic function failed with invalid, deconfigured or * checkstopped AP. */ -struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) +static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) { struct ap_qirq_ctrl aqic_gisa = {}; struct ap_queue_status status; @@ -297,6 +298,7 @@ static int handle_pqap(struct kvm_vcpu *vcpu) if (!q) goto out_unlock; + q->matrix_mdev = matrix_mdev; status = vcpu->run->s.regs.gprs[1]; /* If IR bit(16) is set we enable the interrupt */ @@ -1114,20 +1116,6 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, return NOTIFY_OK; } -static void vfio_ap_irq_disable_apqn(int apqn) -{ - struct device *dev; - struct vfio_ap_queue *q; - - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &apqn, match_apqn); - if (dev) { - q = dev_get_drvdata(dev); - vfio_ap_irq_disable(q); - put_device(dev); - } -} - int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, unsigned int retry) { @@ -1162,6 +1150,7 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev) { int ret; int rc = 0; + struct vfio_ap_queue *q; unsigned long apid, apqi; struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); @@ -1177,7 +1166,10 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev) */ if (ret) rc = ret; - vfio_ap_irq_disable_apqn(AP_MKQID(apid, apqi)); + q = vfio_ap_get_queue(matrix_mdev, + AP_MKQID(apid, apqi)); + if (q) + vfio_ap_free_aqic_resources(q); } } @@ -1302,3 +1294,40 @@ void vfio_ap_mdev_unregister(void) { mdev_unregister_device(&matrix_dev->device); } + +int vfio_ap_mdev_probe_queue(struct ap_device *apdev) +{ + struct vfio_ap_queue *q; + struct ap_queue *queue; + + queue = to_ap_queue(&apdev->device); + + q = kzalloc(sizeof(*q), GFP_KERNEL); + if (!q) + return -ENOMEM; + + dev_set_drvdata(&queue->ap_dev.device, q); + q->apqn = queue->qid; + q->saved_isc = VFIO_AP_ISC_INVALID; + + return 0; +} + +void vfio_ap_mdev_remove_queue(struct ap_device *apdev) +{ + struct vfio_ap_queue *q; + struct ap_queue *queue; + int apid, apqi; + + queue = to_ap_queue(&apdev->device); + + mutex_lock(&matrix_dev->lock); + q = dev_get_drvdata(&queue->ap_dev.device); + dev_set_drvdata(&queue->ap_dev.device, NULL); + apid = AP_QID_CARD(q->apqn); + apqi = AP_QID_QUEUE(q->apqn); + vfio_ap_mdev_reset_queue(apid, apqi, 1); + vfio_ap_free_aqic_resources(q); + kfree(q); + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index f46dde56b464..d9003de4fbad 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -90,8 +90,6 @@ struct ap_matrix_mdev { extern int vfio_ap_mdev_register(void); extern void vfio_ap_mdev_unregister(void); -int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, - unsigned int retry); struct vfio_ap_queue { struct ap_matrix_mdev *matrix_mdev; @@ -100,5 +98,8 @@ struct vfio_ap_queue { #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; }; -struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q); + +int vfio_ap_mdev_probe_queue(struct ap_device *queue); +void vfio_ap_mdev_remove_queue(struct ap_device *queue); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Thu Oct 22 17:11:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 084F6C388F9 for ; Thu, 22 Oct 2020 17:12:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 943F624630 for ; Thu, 22 Oct 2020 17:12:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="WfSC+7CE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902334AbgJVRMb (ORCPT ); Thu, 22 Oct 2020 13:12:31 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:28978 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2506063AbgJVRMa (ORCPT ); Thu, 22 Oct 2020 13:12:30 -0400 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH2LfH145306; Thu, 22 Oct 2020 13:12:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=p+4VwF18wVgZ9LefIkeRRTJLWxgu0aKc0xXTLqp2BEM=; b=WfSC+7CEEYXqR/asZeB46ihJWGJNwopUe4ZZFkFrpcloueHqJbcl23TjIR+ddSazHufG Z/ilGDNfyyZB5Vn1G8lykPqeMX2YKKnaQrErozNYUmcawGtesGviTMrz7I7DBVyUll8w Y9eo3A35ct485IlSkLHypNDd2pcn/ypOfQSyLIlNZwF6UzzQnVci6RN56febGwdtmhFP ydz1oGq/Z3UFM2JdhQzJjgetlXlufIrZdOsvRAsnTzWGVwK3CJ7f/9LkYXQHc9VkxUmB wT1uhx8u6+AXn4BkT2zAvtoS1odQt33fMkcyG95FTQUamk7TXmcXO7ywSgSIAMgRKMOl vg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 34bd0rjkkj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:27 -0400 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH9W6Z190009; Thu, 22 Oct 2020 13:12:27 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0b-001b2d01.pphosted.com with ESMTP id 34bd0rjkkd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:27 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCD3w000582; Thu, 22 Oct 2020 17:12:26 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma02dal.us.ibm.com with ESMTP id 347r89vdqc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:26 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCHQo40108436 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:17 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C8F6978067; Thu, 22 Oct 2020 17:12:22 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 108CB7805F; Thu, 22 Oct 2020 17:12:21 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:20 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 02/14] 390/vfio-ap: use new AP bus interface to search for queue devices Date: Thu, 22 Oct 2020 13:11:57 -0400 Message-Id: <20201022171209.19494-3-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 phishscore=0 suspectscore=0 spamscore=0 impostorscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 priorityscore=1501 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This patch refactors the vfio_ap device driver to use the AP bus's ap_get_qdev() function to retrieve the vfio_ap_queue struct containing information about a queue that is bound to the vfio_ap device driver. The bus's ap_get_qdev() function retrieves the queue device from a hashtable keyed by APQN. This is much more efficient than looping over the list of devices attached to the AP bus by several orders of magnitude. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- drivers/s390/crypto/vfio_ap_ops.c | 35 +++++++++++++------------------ 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index c471832f0a30..049b97d7444c 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -26,43 +26,36 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev); -static int match_apqn(struct device *dev, const void *data) -{ - struct vfio_ap_queue *q = dev_get_drvdata(dev); - - return (q->apqn == *(int *)(data)) ? 1 : 0; -} - /** - * vfio_ap_get_queue: Retrieve a queue with a specific APQN from a list + * vfio_ap_get_queue: Retrieve a queue with a specific APQN. * @matrix_mdev: the associated mediated matrix * @apqn: The queue APQN * - * Retrieve a queue with a specific APQN from the list of the - * devices of the vfio_ap_drv. - * Verify that the APID and the APQI are set in the matrix. + * Retrieve a queue with a specific APQN from the AP queue devices attached to + * the AP bus. * - * Returns the pointer to the associated vfio_ap_queue + * Returns the pointer to the vfio_ap_queue with the specified APQN, or NULL. */ static struct vfio_ap_queue *vfio_ap_get_queue( struct ap_matrix_mdev *matrix_mdev, - int apqn) + unsigned long apqn) { - struct vfio_ap_queue *q; - struct device *dev; + struct ap_queue *queue; + struct vfio_ap_queue *q = NULL; if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm)) return NULL; if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm)) return NULL; - dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &apqn, match_apqn); - if (!dev) + queue = ap_get_qdev(apqn); + if (!queue) return NULL; - q = dev_get_drvdata(dev); - q->matrix_mdev = matrix_mdev; - put_device(dev); + + if (queue->ap_dev.device.driver == &matrix_dev->vfio_ap_drv->driver) + q = dev_get_drvdata(&queue->ap_dev.device); + + put_device(&queue->ap_dev.device); return q; } From patchwork Thu Oct 22 17:11:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B7D3C388F7 for ; Thu, 22 Oct 2020 17:13:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D241B2464B for ; Thu, 22 Oct 2020 17:13:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="h/inS62k" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902356AbgJVRMg (ORCPT ); Thu, 22 Oct 2020 13:12:36 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:49556 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902344AbgJVRMe (ORCPT ); Thu, 22 Oct 2020 13:12:34 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH3vlu014797; Thu, 22 Oct 2020 13:12:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=8Y2ajxvyZdj2VyJLFNx4qWZFy7JzUO+mwMN5N1MVZeA=; b=h/inS62kELthjwdEcdaOFoYDTR6DdLYo2KKNHFhVD/6L3SFUt17c4h+i9+or81HNNzWx CYiQz5+NJba2TQJM4fUqurX9bQHQmXZduuPrPkfNJs4nZaxgQu1/NdJAovY0YpAUptpz lgc+C4j+nrySZlanbKTs2f7o8NoaUIJkUpHV2au1E80QJCyWdjvYcYqqBOC6yn+QXi/O uXw3+TKvBa6kZvPI8gpa+8UzsO1nHCJudksrF6x3eozJrrvC9p2ClI0rONa2OKG/JYkW 7sQYIFmM+zWicVD0bbyVVcmKL+eoQBBtOLzay/a7/Xw2S90WEaumeASHdM4mt127+SyS uw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b0vs0y20-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:30 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH47ul015735; Thu, 22 Oct 2020 13:12:29 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b0vs0y1m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:29 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHC7pV002139; Thu, 22 Oct 2020 17:12:28 GMT Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by ppma03dal.us.ibm.com with ESMTP id 347r89vbr2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:28 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCKnP30933612 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:20 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 39AED78060; Thu, 22 Oct 2020 17:12:25 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0D0AA78067; Thu, 22 Oct 2020 17:12:22 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:22 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 03/14] s390/vfio-ap: manage link between queue struct and matrix mdev Date: Thu, 22 Oct 2020 13:11:58 -0400 Message-Id: <20201022171209.19494-4-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 clxscore=1015 lowpriorityscore=0 suspectscore=2 impostorscore=0 malwarescore=0 mlxscore=0 adultscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220111 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's create links between each queue device bound to the vfio_ap device driver and the matrix mdev to which the queue is assigned. The idea is to facilitate efficient retrieval of the objects representing the queue devices and matrix mdevs as well as to verify that a queue assigned to a matrix mdev is bound to the driver. The links will be created as follows: * When the queue device is probed, if its APQN is assigned to a matrix mdev, the structures representing the queue device and the matrix mdev will be linked. * When an adapter or domain is assigned to a matrix mdev, for each new APQN assigned that references a queue device bound to the vfio_ap device driver, the structures representing the queue device and the matrix mdev will be linked. The links will be removed as follows: * When the queue device is removed, if its APQN is assigned to a matrix mdev, the structures representing the queue device and the matrix mdev will be unlinked. * When an adapter or domain is unassigned from a matrix mdev, for each APQN unassigned that references a queue device bound to the vfio_ap device driver, the structures representing the queue device and the matrix mdev will be unlinked. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 146 +++++++++++++++++++++++--- drivers/s390/crypto/vfio_ap_private.h | 3 + 2 files changed, 135 insertions(+), 14 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 049b97d7444c..1357f8f8b7e4 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -28,7 +28,6 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev); /** * vfio_ap_get_queue: Retrieve a queue with a specific APQN. - * @matrix_mdev: the associated mediated matrix * @apqn: The queue APQN * * Retrieve a queue with a specific APQN from the AP queue devices attached to @@ -36,18 +35,11 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev); * * Returns the pointer to the vfio_ap_queue with the specified APQN, or NULL. */ -static struct vfio_ap_queue *vfio_ap_get_queue( - struct ap_matrix_mdev *matrix_mdev, - unsigned long apqn) +static struct vfio_ap_queue *vfio_ap_get_queue(unsigned long apqn) { struct ap_queue *queue; struct vfio_ap_queue *q = NULL; - if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm)) - return NULL; - if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm)) - return NULL; - queue = ap_get_qdev(apqn); if (!queue) return NULL; @@ -60,6 +52,19 @@ static struct vfio_ap_queue *vfio_ap_get_queue( return q; } +static struct vfio_ap_queue * +vfio_ap_mdev_get_queue(struct ap_matrix_mdev *matrix_mdev, unsigned long apqn) +{ + struct vfio_ap_queue *q; + + hash_for_each_possible(matrix_mdev->qtable, q, mdev_qnode, apqn) { + if (q && (q->apqn == apqn)) + return q; + } + + return NULL; +} + /** * vfio_ap_wait_for_irqclear * @apqn: The AP Queue number @@ -171,7 +176,6 @@ static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q) status.response_code); end_free: vfio_ap_free_aqic_resources(q); - q->matrix_mdev = NULL; return status; } @@ -284,14 +288,14 @@ static int handle_pqap(struct kvm_vcpu *vcpu) if (!vcpu->kvm->arch.crypto.pqap_hook) goto out_unlock; + matrix_mdev = container_of(vcpu->kvm->arch.crypto.pqap_hook, struct ap_matrix_mdev, pqap_hook); - q = vfio_ap_get_queue(matrix_mdev, apqn); + q = vfio_ap_mdev_get_queue(matrix_mdev, apqn); if (!q) goto out_unlock; - q->matrix_mdev = matrix_mdev; status = vcpu->run->s.regs.gprs[1]; /* If IR bit(16) is set we enable the interrupt */ @@ -331,6 +335,7 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) matrix_mdev->mdev = mdev; vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); + hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; matrix_mdev->pqap_hook.owner = THIS_MODULE; @@ -559,6 +564,87 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) return 0; } +enum qlink_type { + LINK_APID, + LINK_APQI, + UNLINK_APID, + UNLINK_APQI, +}; + +static void vfio_ap_mdev_link_queue(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid, unsigned long apqi) +{ + struct vfio_ap_queue *q; + + q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); + if (q) { + q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, + &q->mdev_qnode, q->apqn); + } +} + +static void vfio_ap_mdev_unlink_queue(unsigned long apid, unsigned long apqi) +{ + struct vfio_ap_queue *q; + + q = vfio_ap_get_queue(AP_MKQID(apid, apqi)); + if (q) { + q->matrix_mdev = NULL; + hash_del(&q->mdev_qnode); + } +} + +/** + * vfio_ap_mdev_link_queues + * + * @matrix_mdev: The matrix mdev to link. + * @type: The type of @qlink_id. + * @qlink_id: The APID or APQI of the queues to link. + * + * Sets or clears the links between the queues with the specified @qlink_id + * and the @matrix_mdev: + * @type == LINK_APID: Set the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APID) + * @type == LINK_APQI: Set the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APQI) + * @type == UNLINK_APID: Clear the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APID) + * @type == UNLINK_APQI: Clear the links between the @matrix_mdev and the + * queues with the specified @qlink_id (APQI) + */ +static void vfio_ap_mdev_link_queues(struct ap_matrix_mdev *matrix_mdev, + enum qlink_type type, + unsigned long qlink_id) +{ + unsigned long id; + + switch (type) { + case LINK_APID: + for_each_set_bit_inv(id, matrix_mdev->matrix.aqm, + matrix_mdev->matrix.aqm_max + 1) + vfio_ap_mdev_link_queue(matrix_mdev, qlink_id, id); + break; + case UNLINK_APID: + for_each_set_bit_inv(id, matrix_mdev->matrix.aqm, + matrix_mdev->matrix.aqm_max + 1) + vfio_ap_mdev_unlink_queue(qlink_id, id); + break; + case LINK_APQI: + for_each_set_bit_inv(id, matrix_mdev->matrix.apm, + matrix_mdev->matrix.apm_max + 1) + vfio_ap_mdev_link_queue(matrix_mdev, id, qlink_id); + break; + case UNLINK_APQI: + for_each_set_bit_inv(id, matrix_mdev->matrix.apm, + matrix_mdev->matrix.apm_max + 1) + vfio_ap_mdev_link_queue(matrix_mdev, id, qlink_id); + break; + default: + WARN_ON_ONCE(1); + } +} + /** * assign_adapter_store * @@ -628,6 +714,7 @@ static ssize_t assign_adapter_store(struct device *dev, if (ret) goto share_err; + vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); ret = count; goto done; @@ -679,6 +766,7 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); + vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid); mutex_unlock(&matrix_dev->lock); return count; @@ -769,6 +857,7 @@ static ssize_t assign_domain_store(struct device *dev, if (ret) goto share_err; + vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); ret = count; goto done; @@ -821,6 +910,7 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); + vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi); mutex_unlock(&matrix_dev->lock); return count; @@ -1159,8 +1249,8 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev) */ if (ret) rc = ret; - q = vfio_ap_get_queue(matrix_mdev, - AP_MKQID(apid, apqi)); + q = vfio_ap_mdev_get_queue(matrix_mdev, + AP_MKQID(apid, apqi)); if (q) vfio_ap_free_aqic_resources(q); } @@ -1288,6 +1378,29 @@ void vfio_ap_mdev_unregister(void) mdev_unregister_device(&matrix_dev->device); } +/** + * vfio_ap_queue_link_mdev + * + * @q: The queue to link with the matrix mdev. + * + * Links @q with the matrix mdev to which the queue's APQN is assigned. + */ +static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q) +{ + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); + struct ap_matrix_mdev *matrix_mdev; + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (test_bit_inv(apid, matrix_mdev->matrix.apm) && + test_bit_inv(apqi, matrix_mdev->matrix.aqm)) { + q->matrix_mdev = matrix_mdev; + hash_add(matrix_mdev->qtable, &q->mdev_qnode, q->apqn); + break; + } + } +} + int vfio_ap_mdev_probe_queue(struct ap_device *apdev) { struct vfio_ap_queue *q; @@ -1299,9 +1412,12 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) if (!q) return -ENOMEM; + mutex_lock(&matrix_dev->lock); dev_set_drvdata(&queue->ap_dev.device, q); q->apqn = queue->qid; q->saved_isc = VFIO_AP_ISC_INVALID; + vfio_ap_queue_link_mdev(q); + mutex_unlock(&matrix_dev->lock); return 0; } @@ -1321,6 +1437,8 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) apqi = AP_QID_QUEUE(q->apqn); vfio_ap_mdev_reset_queue(apid, apqi, 1); vfio_ap_free_aqic_resources(q); + if (q->matrix_mdev) + hash_del(&q->mdev_qnode); kfree(q); mutex_unlock(&matrix_dev->lock); } diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index d9003de4fbad..4e5cc72fc0db 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -18,6 +18,7 @@ #include #include #include +#include #include "ap_bus.h" @@ -86,6 +87,7 @@ struct ap_matrix_mdev { struct kvm *kvm; struct kvm_s390_module_hook pqap_hook; struct mdev_device *mdev; + DECLARE_HASHTABLE(qtable, 8); }; extern int vfio_ap_mdev_register(void); @@ -97,6 +99,7 @@ struct vfio_ap_queue { int apqn; #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; + struct hlist_node mdev_qnode; }; int vfio_ap_mdev_probe_queue(struct ap_device *queue); From patchwork Thu Oct 22 17:11:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_NONE,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 517B6C5DF9D for ; Thu, 22 Oct 2020 17:12:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C37B82464B for ; Thu, 22 Oct 2020 17:12:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Azbnpumv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902344AbgJVRMi (ORCPT ); Thu, 22 Oct 2020 13:12:38 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:15578 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2507013AbgJVRMh (ORCPT ); Thu, 22 Oct 2020 13:12:37 -0400 Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH1q4i011945; Thu, 22 Oct 2020 13:12:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Ep2zQCA5Q6Yb2JBE00lEaxikihlvLF0BMl8dVM7Vf6k=; b=AzbnpumvwJNyOepZHgSY23w/LwF1ab+3iyloYDym/f3rKvGA6ofE5Lj160wuJ9ZfapNP Ft+Mj5ABsgqMRt0u8YFSPvbZeDB5C8ll3vGSYpmsSQElPY0HJnC9Rhq1DpXK5ZS17wRw jZLwv/QYYorC3BoE+isJhoRh+7Z9Dy3Z+th4NqRjyztkBD7woGWwGLgpYEPrYMb2Xtwv 46D7tt+Ye08Anf2L9Xrg3wXTr/f8UFmPCcQANUgMJ0XjaRS4DbVFsmcMAnjmUnpVBUUF xW88p6nJlheMHOBB1ZeLOLao2bU+He7gCWvE08oLf8rBEaPeYe03+3sGsQAevlslpq8x OA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b0ekwddf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:31 -0400 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH1wn3012373; Thu, 22 Oct 2020 13:12:31 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b0ekwdd2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:31 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCEDM000712; Thu, 22 Oct 2020 17:12:30 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma02dal.us.ibm.com with ESMTP id 347r89vdr5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:30 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCRAV2687600 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:27 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 36F847805C; Thu, 22 Oct 2020 17:12:27 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6B61278064; Thu, 22 Oct 2020 17:12:25 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:25 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 04/14] s390/zcrypt: driver callback to indicate resource in use Date: Thu, 22 Oct 2020 13:11:59 -0400 Message-Id: <20201022171209.19494-5-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_11:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 priorityscore=1501 spamscore=0 phishscore=0 lowpriorityscore=0 bulkscore=0 suspectscore=0 adultscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduces a new driver callback to prevent a root user from unbinding an AP queue from its device driver if the queue is in use. The callback will be invoked whenever a change to the AP bus's sysfs apmask or aqmask attributes would result in one or more AP queues being removed from its driver. If the callback responds in the affirmative for any driver queried, the change to the apmask or aqmask will be rejected with a device in use error. For this patch, only non-default drivers will be queried. Currently, there is only one non-default driver, the vfio_ap device driver. The vfio_ap device driver facilitates pass-through of an AP queue to a guest. The idea here is that a guest may be administered by a different sysadmin than the host and we don't want AP resources to unexpectedly disappear from a guest's AP configuration (i.e., adapters and domains assigned to the matrix mdev). This will enforce the proper procedure for removing AP resources intended for guest usage which is to first unassign them from the matrix mdev, then unbind them from the vfio_ap device driver. Signed-off-by: Tony Krowiak Acked-by: Halil Pasic Reviewed-by: Harald Freudenberger --- drivers/s390/crypto/ap_bus.c | 148 ++++++++++++++++++++++++++++++++--- drivers/s390/crypto/ap_bus.h | 4 + 2 files changed, 142 insertions(+), 10 deletions(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index 485cbfcbf06e..998e61cd86d9 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -35,6 +35,7 @@ #include #include #include +#include #include "ap_bus.h" #include "ap_debug.h" @@ -893,6 +894,23 @@ static int modify_bitmap(const char *str, unsigned long *bitmap, int bits) return 0; } +static int ap_parse_bitmap_str(const char *str, unsigned long *bitmap, int bits, + unsigned long *newmap) +{ + unsigned long size; + int rc; + + size = BITS_TO_LONGS(bits)*sizeof(unsigned long); + if (*str == '+' || *str == '-') { + memcpy(newmap, bitmap, size); + rc = modify_bitmap(str, newmap, bits); + } else { + memset(newmap, 0, size); + rc = hex2bitmap(str, newmap, bits); + } + return rc; +} + int ap_parse_mask_str(const char *str, unsigned long *bitmap, int bits, struct mutex *lock) @@ -912,14 +930,7 @@ int ap_parse_mask_str(const char *str, kfree(newmap); return -ERESTARTSYS; } - - if (*str == '+' || *str == '-') { - memcpy(newmap, bitmap, size); - rc = modify_bitmap(str, newmap, bits); - } else { - memset(newmap, 0, size); - rc = hex2bitmap(str, newmap, bits); - } + rc = ap_parse_bitmap_str(str, bitmap, bits, newmap); if (rc == 0) memcpy(bitmap, newmap, size); mutex_unlock(lock); @@ -1111,12 +1122,70 @@ static ssize_t apmask_show(struct bus_type *bus, char *buf) return rc; } +static int __verify_card_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newapm = (unsigned long *)data; + + /* + * No need to verify whether the driver is using the queues if it is the + * default driver. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(newapm, ap_perms.aqm)) + rc = -EBUSY; + + module_put(drv->owner); + + return rc; +} + +static int apmask_commit(unsigned long *newapm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DEVICES)]; + + /* + * Check if any bits in the apmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newapm, ap_perms.apm, AP_DEVICES)) { + rc = bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_card_reservations); + if (rc) + return rc; + } + + memcpy(ap_perms.apm, newapm, APMASKSIZE); + + return 0; +} + static ssize_t apmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + DECLARE_BITMAP(newapm, AP_DEVICES); + + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + rc = ap_parse_bitmap_str(buf, ap_perms.apm, AP_DEVICES, newapm); + if (rc) + goto done; - rc = ap_parse_mask_str(buf, ap_perms.apm, AP_DEVICES, &ap_perms_mutex); + rc = apmask_commit(newapm); + +done: + mutex_unlock(&ap_perms_mutex); if (rc) return rc; @@ -1142,12 +1211,71 @@ static ssize_t aqmask_show(struct bus_type *bus, char *buf) return rc; } +static int __verify_queue_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newaqm = (unsigned long *)data; + + /* + * If the reserved bits do not identify queues reserved for use by the + * non-default driver, there is no need to verify the driver is using + * the queues. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(ap_perms.apm, newaqm)) + rc = -EBUSY; + + module_put(drv->owner); + + return rc; +} + +static int aqmask_commit(unsigned long *newaqm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DOMAINS)]; + + /* + * Check if any bits in the aqmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newaqm, ap_perms.aqm, AP_DOMAINS)) { + rc = bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_queue_reservations); + if (rc) + return rc; + } + + memcpy(ap_perms.aqm, newaqm, AQMASKSIZE); + + return 0; +} + static ssize_t aqmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + DECLARE_BITMAP(newaqm, AP_DOMAINS); - rc = ap_parse_mask_str(buf, ap_perms.aqm, AP_DOMAINS, &ap_perms_mutex); + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + rc = ap_parse_bitmap_str(buf, ap_perms.aqm, AP_DOMAINS, newaqm); + if (rc) + goto done; + + rc = aqmask_commit(newaqm); + +done: + mutex_unlock(&ap_perms_mutex); if (rc) return rc; diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 5029b80132aa..6ce154d924d3 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -145,6 +145,7 @@ struct ap_driver { int (*probe)(struct ap_device *); void (*remove)(struct ap_device *); + bool (*in_use)(unsigned long *apm, unsigned long *aqm); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) @@ -293,6 +294,9 @@ void ap_queue_init_state(struct ap_queue *aq); struct ap_card *ap_card_create(int id, int queue_depth, int raw_device_type, int comp_device_type, unsigned int functions); +#define APMASKSIZE (BITS_TO_LONGS(AP_DEVICES) * sizeof(unsigned long)) +#define AQMASKSIZE (BITS_TO_LONGS(AP_DOMAINS) * sizeof(unsigned long)) + struct ap_perms { unsigned long ioctlm[BITS_TO_LONGS(AP_IOCTLS)]; unsigned long apm[BITS_TO_LONGS(AP_DEVICES)]; From patchwork Thu Oct 22 17:12:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71F3DC388F9 for ; Thu, 22 Oct 2020 17:12:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1188D2464B for ; Thu, 22 Oct 2020 17:12:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Rirtmkaf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902395AbgJVRMl (ORCPT ); Thu, 22 Oct 2020 13:12:41 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:2294 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2902367AbgJVRMj (ORCPT ); Thu, 22 Oct 2020 13:12:39 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH28mq054543; Thu, 22 Oct 2020 13:12:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=v3jO95AaVvEElo4NrAGwZiMGCqe6xcobZljhqbHeaqw=; b=Rirtmkaf24kl4lBNBp/0OxhX4aNuuzqm454stJP9T67ZL/04oAsyopvGytBc69v9UQt6 pSWGPkPUVhaRkPiZUcoKIO8AFLgcoa6o3CsaYM4aF62+vANmMYKlnFMXy3RHQftuSmR6 Zc49OjP1i9qiGVupb+7P0btUnhR9ZdVIojB0aUjSQrlxLvohDuxCpvbh6cMk2gzWh2zt ohC5YquZLTH1V295fLulNnTXCQ8BHioh1BGP0sEHJX13zegflGVpVYDQyA/RqyFKE75l QBhaMFkDZAZi9Byw60twIBaYuLdbnVsVPiNUzaLMT4RLlGPhaz9Hgsq0VS10X4oP7z0J 9A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 34b00e2cgm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:33 -0400 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH2iSk057254; Thu, 22 Oct 2020 13:12:33 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0b-001b2d01.pphosted.com with ESMTP id 34b00e2cga-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:33 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MGsqxP006119; Thu, 22 Oct 2020 17:12:32 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma01wdc.us.ibm.com with ESMTP id 347r89hw6v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:32 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCTfU49611014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:29 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 40F6E78064; Thu, 22 Oct 2020 17:12:29 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6D70C78060; Thu, 22 Oct 2020 17:12:27 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:27 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 05/14] s390/vfio-ap: implement in-use callback for vfio_ap driver Date: Thu, 22 Oct 2020 13:12:00 -0400 Message-Id: <20201022171209.19494-6-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_11:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 impostorscore=0 suspectscore=0 spamscore=0 clxscore=1015 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's implement the callback to indicate when an APQN is in use by the vfio_ap device driver. The callback is invoked whenever a change to the apmask or aqmask would result in one or more queue devices being removed from the driver. The vfio_ap device driver will indicate a resource is in use if the APQN of any of the queue devices to be removed are assigned to any of the matrix mdevs under the driver's control. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 1 + drivers/s390/crypto/vfio_ap_ops.c | 78 +++++++++++++++++++-------- drivers/s390/crypto/vfio_ap_private.h | 2 + 3 files changed, 60 insertions(+), 21 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 73bd073fd5d3..8934471b7944 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -147,6 +147,7 @@ static int __init vfio_ap_init(void) memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv)); vfio_ap_drv.probe = vfio_ap_mdev_probe_queue; vfio_ap_drv.remove = vfio_ap_mdev_remove_queue; + vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 1357f8f8b7e4..9e9fad560859 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -522,18 +522,40 @@ vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, return 0; } +#define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \ + "already assigned to %s" + +static void vfio_ap_mdev_log_sharing_err(const char *mdev_name, + unsigned long *apm, + unsigned long *aqm) +{ + unsigned long apid, apqi; + + for_each_set_bit_inv(apid, apm, AP_DEVICES) + for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) + pr_err(MDEV_SHARING_ERR, apid, apqi, mdev_name); +} + /** * vfio_ap_mdev_verify_no_sharing * - * Verifies that the APQNs derived from the cross product of the AP adapter IDs - * and AP queue indexes comprising the AP matrix are not configured for another + * Verifies that each APQN derived from the cross product of the AP adapter IDs + * and AP queue indexes comprising an AP matrix is not assigned to a * mediated device. AP queue sharing is not allowed. * - * @matrix_mdev: the mediated matrix device + * @matrix_mdev: the mediated matrix device to which the APQNs being verified + * are assigned. If the value is not NULL, then verification will + * proceed for all other matrix mediated devices; otherwise, all + * matrix mediated devices will be verified. + * @mdev_apm: mask indicating the APIDs of the APQNs to be verified + * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified * - * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE. + * Returns 0 if no APQNs are not shared, otherwise; returns -EADDRINUSE if one + * or more APQNs are shared. */ -static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) +static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, + unsigned long *mdev_apm, + unsigned long *mdev_aqm) { struct ap_matrix_mdev *lstdev; DECLARE_BITMAP(apm, AP_DEVICES); @@ -550,14 +572,15 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) * We work on full longs, as we can only exclude the leftover * bits in non-inverse order. The leftover is all zeros. */ - if (!bitmap_and(apm, matrix_mdev->matrix.apm, - lstdev->matrix.apm, AP_DEVICES)) + if (!bitmap_and(apm, mdev_apm, lstdev->matrix.apm, AP_DEVICES)) continue; - if (!bitmap_and(aqm, matrix_mdev->matrix.aqm, - lstdev->matrix.aqm, AP_DOMAINS)) + if (!bitmap_and(aqm, mdev_aqm, lstdev->matrix.aqm, AP_DOMAINS)) continue; + vfio_ap_mdev_log_sharing_err(dev_name(mdev_dev(lstdev->mdev)), + apm, aqm); + return -EADDRINUSE; } @@ -683,6 +706,7 @@ static ssize_t assign_adapter_store(struct device *dev, { int ret; unsigned long apid; + DECLARE_BITMAP(apm, AP_DEVICES); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); @@ -708,18 +732,18 @@ static ssize_t assign_adapter_store(struct device *dev, if (ret) goto done; - set_bit_inv(apid, matrix_mdev->matrix.apm); + memset(apm, 0, sizeof(apm)); + set_bit_inv(apid, apm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm, + matrix_mdev->matrix.aqm); if (ret) - goto share_err; + goto done; + set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); ret = count; - goto done; -share_err: - clear_bit_inv(apid, matrix_mdev->matrix.apm); done: mutex_unlock(&matrix_dev->lock); @@ -831,6 +855,7 @@ static ssize_t assign_domain_store(struct device *dev, { int ret; unsigned long apqi; + DECLARE_BITMAP(aqm, AP_DOMAINS); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; @@ -851,18 +876,18 @@ static ssize_t assign_domain_store(struct device *dev, if (ret) goto done; - set_bit_inv(apqi, matrix_mdev->matrix.aqm); + memset(aqm, 0, sizeof(aqm)); + set_bit_inv(apqi, aqm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); + ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, + matrix_mdev->matrix.apm, aqm); if (ret) - goto share_err; + goto done; + set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); ret = count; - goto done; -share_err: - clear_bit_inv(apqi, matrix_mdev->matrix.aqm); done: mutex_unlock(&matrix_dev->lock); @@ -1442,3 +1467,14 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) kfree(q); mutex_unlock(&matrix_dev->lock); } + +bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) +{ + bool in_use; + + mutex_lock(&matrix_dev->lock); + in_use = !!vfio_ap_mdev_verify_no_sharing(NULL, apm, aqm); + mutex_unlock(&matrix_dev->lock); + + return in_use; +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 4e5cc72fc0db..c1d8b5507610 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -105,4 +105,6 @@ struct vfio_ap_queue { int vfio_ap_mdev_probe_queue(struct ap_device *queue); void vfio_ap_mdev_remove_queue(struct ap_device *queue); +bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Thu Oct 22 17:12:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 976D6C56202 for ; Thu, 22 Oct 2020 17:12:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A48924655 for ; Thu, 22 Oct 2020 17:12:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="gD1TaEfW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902378AbgJVRMi (ORCPT ); Thu, 22 Oct 2020 13:12:38 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:7662 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902361AbgJVRMh (ORCPT ); Thu, 22 Oct 2020 13:12:37 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH2Mlt030302; Thu, 22 Oct 2020 13:12:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=sWLA/DFuD0MdaWaRnl32jj9lZLh6NspRDhzEWeVsFgI=; b=gD1TaEfWz69lGM3EU4Xtx6w8QnU+pcBP179CfDzxYSr9cy1Sixgpp3xz4d8mCuIy9u/4 kmLcjPGu1t5poiH9WwFgMBHikizCzAQlKKm5rEMxDjY0wJJwnMh3S+itZPv09JRkpkPG 1aD1GADwl4VNhd6cNG/hPmIrrhPpqfNaHp2v8Pw+cJ4Q8zv/FgKDuycr2fugGDog5Hs7 VpwPBF9qRgQSTE2pCQPCWglx6S25xzNPi5VpKvntVe09Ns+UlQrAxjQdNWErAp/FnWXp gw8p969UKKWlmY3iirxuEBHsxnHOI2j6xVYtWmxEJJe5F4KPvrGkQ0Z4Qn4eW1wwoFca iA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b6n8r6qb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:36 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH4wrj044433; Thu, 22 Oct 2020 13:12:36 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b6n8r6q3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:36 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHC7sw002136; Thu, 22 Oct 2020 17:12:35 GMT Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by ppma03dal.us.ibm.com with ESMTP id 347r89vbrs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:35 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCQOi32768438 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:26 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 80CCE78068; Thu, 22 Oct 2020 17:12:31 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7F06E7805F; Thu, 22 Oct 2020 17:12:29 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:29 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 06/14] s390/vfio-ap: introduce shadow APCB Date: Thu, 22 Oct 2020 13:12:01 -0400 Message-Id: <20201022171209.19494-7-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 adultscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 impostorscore=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The APCB is a field within the CRYCB that provides the AP configuration to a KVM guest. Let's introduce a shadow copy of the KVM guest's APCB and maintain it for the lifespan of the guest. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic --- drivers/s390/crypto/vfio_ap_ops.c | 24 +++++++++++++++++++----- drivers/s390/crypto/vfio_ap_private.h | 2 ++ 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 9e9fad560859..9791761aa7fd 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -320,6 +320,19 @@ static void vfio_ap_matrix_init(struct ap_config_info *info, matrix->adm_max = info->apxa ? info->Nd : 15; } +static bool vfio_ap_mdev_has_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + return (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd); +} + +static void vfio_ap_mdev_commit_shadow_apcb(struct ap_matrix_mdev *matrix_mdev) +{ + kvm_arch_crypto_set_masks(matrix_mdev->kvm, + matrix_mdev->shadow_apcb.apm, + matrix_mdev->shadow_apcb.aqm, + matrix_mdev->shadow_apcb.adm); +} + static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) { struct ap_matrix_mdev *matrix_mdev; @@ -335,6 +348,7 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) matrix_mdev->mdev = mdev; vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); + vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb); hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; @@ -1213,13 +1227,12 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (ret) return NOTIFY_DONE; - /* If there is no CRYCB pointer, then we can't copy the masks */ - if (!matrix_mdev->kvm->arch.crypto.crycbd) + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) return NOTIFY_DONE; - kvm_arch_crypto_set_masks(matrix_mdev->kvm, matrix_mdev->matrix.apm, - matrix_mdev->matrix.aqm, - matrix_mdev->matrix.adm); + memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix, + sizeof(matrix_mdev->shadow_apcb)); + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); return NOTIFY_OK; } @@ -1329,6 +1342,7 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev) kvm_put_kvm(matrix_mdev->kvm); matrix_mdev->kvm = NULL; } + mutex_unlock(&matrix_dev->lock); vfio_unregister_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY, diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index c1d8b5507610..fc8634cee485 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -75,6 +75,7 @@ struct ap_matrix { * @list: allows the ap_matrix_mdev struct to be added to a list * @matrix: the adapters, usage domains and control domains assigned to the * mediated matrix device. + * @shadow_apcb: the shadow copy of the APCB field of the KVM guest's CRYCB * @group_notifier: notifier block used for specifying callback function for * handling the VFIO_GROUP_NOTIFY_SET_KVM event * @kvm: the struct holding guest's state @@ -82,6 +83,7 @@ struct ap_matrix { struct ap_matrix_mdev { struct list_head node; struct ap_matrix matrix; + struct ap_matrix shadow_apcb; struct notifier_block group_notifier; struct notifier_block iommu_notifier; struct kvm *kvm; From patchwork Thu Oct 22 17:12:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30EA4C388F9 for ; Thu, 22 Oct 2020 17:13:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B407924630 for ; Thu, 22 Oct 2020 17:13:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="pAhuRmBy" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902391AbgJVRNi (ORCPT ); Thu, 22 Oct 2020 13:13:38 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:53810 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2902380AbgJVRMk (ORCPT ); Thu, 22 Oct 2020 13:12:40 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH29Gk054665; Thu, 22 Oct 2020 13:12:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=gcLZYySwvxnWbRDdjGHe0uIhlOME1LIX5y9Cda/eySo=; b=pAhuRmByksQwTzmZK4CDhBpP2dLzZVKXe0ti+9SXCsIwtrjFfLr/LbLMxoTHwyuCCi/2 yehHWs5Wq8fvUjwqRSc+3Wjv7rUh/pHZSVgEO71QZKrpAzlwtf3ZdNLldUpGIzTO/1rF RFUyUnNtYvefpazrk3Nr6STtmFA1+8V23KLR29Xg583zRuso15Gk3l5McYqub08ffvJw Btoq9jtL28Ww8QqEaezweGLLkCD+5AglLnhMIBKJM1VRvlhp3EaEchM0q4CjIrecBl2E fqJb4pIqYPyhpokmlV66ywx0o75HaeL0RgiSALICsuMbPq5uUnBmT/lH6joF1EAmg2FO Ow== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 34b00e2cjm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:38 -0400 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH3O5a060686; Thu, 22 Oct 2020 13:12:37 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0b-001b2d01.pphosted.com with ESMTP id 34b00e2cj4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:37 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCLYB015375; Thu, 22 Oct 2020 17:12:37 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma01dal.us.ibm.com with ESMTP id 347r89vbb4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:37 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCXmC58392910 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:33 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A6EAB7805C; Thu, 22 Oct 2020 17:12:33 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B4C8A78068; Thu, 22 Oct 2020 17:12:31 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:31 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 07/14] s390/vfio-ap: sysfs attribute to display the guest's matrix Date: Thu, 22 Oct 2020 13:12:02 -0400 Message-Id: <20201022171209.19494-8-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_11:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 impostorscore=0 suspectscore=0 spamscore=0 clxscore=1015 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The matrix of adapters and domains configured in a guest's APCB may differ from the matrix of adapters and domains assigned to the matrix mdev, so this patch introduces a sysfs attribute to display the matrix of a guest using the matrix mdev. For a matrix mdev denoted by $uuid, the crycb for a guest using the matrix mdev can be displayed as follows: cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix If a guest is not using the matrix mdev at the time the crycb is displayed, an error (ENODEV) will be returned. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 54 +++++++++++++++++++++++-------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 9791761aa7fd..7bad70d7bcef 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1073,29 +1073,24 @@ static ssize_t control_domains_show(struct device *dev, } static DEVICE_ATTR_RO(control_domains); -static ssize_t matrix_show(struct device *dev, struct device_attribute *attr, - char *buf) +static ssize_t vfio_ap_mdev_matrix_show(struct ap_matrix *matrix, char *buf) { - struct mdev_device *mdev = mdev_from_dev(dev); - struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); char *bufpos = buf; unsigned long apid; unsigned long apqi; unsigned long apid1; unsigned long apqi1; - unsigned long napm_bits = matrix_mdev->matrix.apm_max + 1; - unsigned long naqm_bits = matrix_mdev->matrix.aqm_max + 1; + unsigned long napm_bits = matrix->apm_max + 1; + unsigned long naqm_bits = matrix->aqm_max + 1; int nchars = 0; int n; - apid1 = find_first_bit_inv(matrix_mdev->matrix.apm, napm_bits); - apqi1 = find_first_bit_inv(matrix_mdev->matrix.aqm, naqm_bits); - - mutex_lock(&matrix_dev->lock); + apid1 = find_first_bit_inv(matrix->apm, napm_bits); + apqi1 = find_first_bit_inv(matrix->aqm, naqm_bits); if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) { - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) { - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, + for_each_set_bit_inv(apid, matrix->apm, napm_bits) { + for_each_set_bit_inv(apqi, matrix->aqm, naqm_bits) { n = sprintf(bufpos, "%02lx.%04lx\n", apid, apqi); @@ -1104,25 +1099,55 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr, } } } else if (apid1 < napm_bits) { - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) { + for_each_set_bit_inv(apid, matrix->apm, napm_bits) { n = sprintf(bufpos, "%02lx.\n", apid); bufpos += n; nchars += n; } } else if (apqi1 < naqm_bits) { - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, naqm_bits) { + for_each_set_bit_inv(apqi, matrix->aqm, naqm_bits) { n = sprintf(bufpos, ".%04lx\n", apqi); bufpos += n; nchars += n; } } + return nchars; +} + +static ssize_t matrix_show(struct device *dev, struct device_attribute *attr, + char *buf) +{ + ssize_t nchars; + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + + mutex_lock(&matrix_dev->lock); + nchars = vfio_ap_mdev_matrix_show(&matrix_mdev->matrix, buf); mutex_unlock(&matrix_dev->lock); return nchars; } static DEVICE_ATTR_RO(matrix); +static ssize_t guest_matrix_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + ssize_t nchars; + struct mdev_device *mdev = mdev_from_dev(dev); + struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); + + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + return -ENODEV; + + mutex_lock(&matrix_dev->lock); + nchars = vfio_ap_mdev_matrix_show(&matrix_mdev->shadow_apcb, buf); + mutex_unlock(&matrix_dev->lock); + + return nchars; +} +static DEVICE_ATTR_RO(guest_matrix); + static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_assign_adapter.attr, &dev_attr_unassign_adapter.attr, @@ -1132,6 +1157,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = { &dev_attr_unassign_control_domain.attr, &dev_attr_control_domains.attr, &dev_attr_matrix.attr, + &dev_attr_guest_matrix.attr, NULL, }; From patchwork Thu Oct 22 17:12:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722DEC388F7 for ; Thu, 22 Oct 2020 17:12:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1DCE224650 for ; Thu, 22 Oct 2020 17:12:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="mJU1umRf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902414AbgJVRMo (ORCPT ); Thu, 22 Oct 2020 13:12:44 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:35422 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2902392AbgJVRMn (ORCPT ); Thu, 22 Oct 2020 13:12:43 -0400 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH2NQK145431; Thu, 22 Oct 2020 13:12:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=qDcnxp73WNwHtdIhFfgvsQAiN/ur7Hycdt5nS76/5i0=; b=mJU1umRfXO/p/q9N2i2ttZ3HOeuopydmXIiA+wL1CGzsrsgeRVpBLR9KFy7oLXP+64qO CHlkkpFmZfLaIwItAxczlSX9jSy4IR3iqazB9BbQYJ4H11zE/1/k9E/rUlZPc5ybvmEN ZCV+89js/Of1TTixmE7lJ3NeJCILlGyE2CUDjj+CNxS023ckOfZvVVmN8hBVSm/xvA/w GqqMv4aGlnek8mMfIfGw4O7hREvR0dazSs5rmS3xXTz/gLKmYUwiF7EDYdvu5CqK8/kk E9vMguS9TMlILFnD8v0Wv9PiPTClXeMHjDk6THzKq1PAMsZKjwwalWRt0H1w+wgzYDEo 4A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 34bd0rjkrr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:40 -0400 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH2Xuv146808; Thu, 22 Oct 2020 13:12:39 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0b-001b2d01.pphosted.com with ESMTP id 34bd0rjkre-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:39 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCDiG000591; Thu, 22 Oct 2020 17:12:39 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma02dal.us.ibm.com with ESMTP id 347r89vdrx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:39 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCZxf30409018 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:35 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9C66778060; Thu, 22 Oct 2020 17:12:35 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DC1F27805C; Thu, 22 Oct 2020 17:12:33 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:33 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 08/14] s390/vfio-ap: hot plug/unplug queues on bind/unbind of queue device Date: Thu, 22 Oct 2020 13:12:03 -0400 Message-Id: <20201022171209.19494-9-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 malwarescore=0 phishscore=0 suspectscore=0 spamscore=0 impostorscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 priorityscore=1501 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In response to the probe or remove of a queue device, if a KVM guest is using the matrix mdev to which the APQN of the queue device is assigned, the vfio_ap device driver must respond accordingly. In an ideal world, the queue device being probed would be hot plugged into the guest. Likewise, the queue corresponding to the queue device being removed would be hot unplugged from the guest. Unfortunately, the AP architecture precludes plugging or unplugging individual queues. We must also consider the fact that the linux device model precludes us from passing a queue device through to a KVM guest that is not bound to the driver facilitating the pass-through. Consequently, we are left with the choice of plugging/unplugging the adapter or the domain. In the latter case, this would result in taking access to the domain away for each adapter the guest is using. In either case, the operation will alter a KVM guest's access to one or more queues, so let's plug/unplug the adapter on bind/unbind of the queue device since this corresponds to the hardware entity that may be physically plugged/unplugged - i.e., a domain is not a piece of hardware. Example: ======= Queue devices bound to vfio_ap device driver: 04.0004 04.0047 04.0054 05.0005 05.0047 Adapters and domains assigned to matrix mdev: Adapters Domains -> Queues 04 0004 04.0004 05 0047 04.0047 0054 04.0054 05.0004 05.0047 05.0054 KVM guest matrix at is startup: Adapters Domains -> Queues 04 0004 04.0004 0047 04.0047 0054 04.0054 Adapter 05 is filtered because queue 05.0054 is not bound. KVM guest matrix after queue 05.0054 is bound to the vfio_ap driver: Adapters Domains -> Queues 04 0004 04.0004 05 0047 04.0047 0054 04.0054 05.0004 05.0047 05.0054 All queues assigned to the matrix mdev are now bound. KVM guest matrix after queue 04.0004 is unbound: Adapters Domains -> Queues 05 0004 05.0004 0047 05.0047 0054 05.0054 Adapter 04 is filtered because 04.0004 is no longer bound. Signed-off-by: Tony Krowiak Reported-by: kernel test robot --- drivers/s390/crypto/vfio_ap_ops.c | 158 +++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 3 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 7bad70d7bcef..5b34bc8fca31 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -312,6 +312,13 @@ static int handle_pqap(struct kvm_vcpu *vcpu) return 0; } +static void vfio_ap_matrix_clear_masks(struct ap_matrix *matrix) +{ + bitmap_clear(matrix->apm, 0, AP_DEVICES); + bitmap_clear(matrix->aqm, 0, AP_DOMAINS); + bitmap_clear(matrix->adm, 0, AP_DOMAINS); +} + static void vfio_ap_matrix_init(struct ap_config_info *info, struct ap_matrix *matrix) { @@ -601,6 +608,104 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, return 0; } +static bool vfio_ap_mdev_matrixes_equal(struct ap_matrix *matrix1, + struct ap_matrix *matrix2) +{ + return (bitmap_equal(matrix1->apm, matrix2->apm, AP_DEVICES) && + bitmap_equal(matrix1->aqm, matrix2->aqm, AP_DOMAINS) && + bitmap_equal(matrix1->adm, matrix2->adm, AP_DOMAINS)); +} + +/** + * vfio_ap_mdev_filter_matrix + * + * Filters the matrix of adapters, domains, and control domains assigned to + * a matrix mdev's AP configuration and stores the result in the shadow copy of + * the APCB used to supply a KVM guest's AP configuration. + * + * @matrix_mdev: the matrix mdev whose AP configuration is to be filtered + * + * Returns true if filtering has changed the shadow copy of the APCB used + * to supply a KVM guest's AP configuration; otherwise, returns false. + */ +static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev) +{ + struct ap_matrix shadow_apcb; + unsigned long apid, apqi, apqn; + + memcpy(&shadow_apcb, &matrix_mdev->matrix, sizeof(struct ap_matrix)); + + for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) { + /* + * If the APID is not assigned to the host AP configuration, + * we can not assign it to the guest's AP configuration + */ + if (!test_bit_inv(apid, + (unsigned long *)matrix_dev->info.apm)) { + clear_bit_inv(apid, shadow_apcb.apm); + continue; + } + + for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, + AP_DOMAINS) { + /* + * If the APQI is not assigned to the host AP + * configuration, then it can not be assigned to the + * guest's AP configuration + */ + if (!test_bit_inv(apqi, (unsigned long *) + matrix_dev->info.aqm)) { + clear_bit_inv(apqi, shadow_apcb.aqm); + continue; + } + + /* + * If the APQN is not bound to the vfio_ap device + * driver, then we can't assign it to the guest's + * AP configuration. The AP architecture won't + * allow filtering of a single APQN, so let's filter + * the APID. + */ + apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_mdev_get_queue(matrix_mdev, apqn)) { + clear_bit_inv(apid, shadow_apcb.apm); + break; + } + } + + /* + * If all APIDs have been cleared, then clear the APQIs from the + * shadow APCB and quit filtering. + */ + if (bitmap_empty(shadow_apcb.apm, AP_DEVICES)) { + if (!bitmap_empty(shadow_apcb.aqm, AP_DOMAINS)) + bitmap_clear(shadow_apcb.aqm, 0, AP_DOMAINS); + + break; + } + + /* + * If all APQIs have been cleared, then clear the APIDs from the + * shadow APCB and quit filtering. + */ + if (bitmap_empty(shadow_apcb.aqm, AP_DOMAINS)) { + if (!bitmap_empty(shadow_apcb.apm, AP_DEVICES)) + bitmap_clear(shadow_apcb.apm, 0, AP_DEVICES); + + break; + } + } + + if (vfio_ap_mdev_matrixes_equal(&matrix_mdev->shadow_apcb, + &shadow_apcb)) + return false; + + memcpy(&matrix_mdev->shadow_apcb, &shadow_apcb, + sizeof(struct ap_matrix)); + + return true; +} + enum qlink_type { LINK_APID, LINK_APQI, @@ -1256,9 +1361,8 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (!vfio_ap_mdev_has_crycb(matrix_mdev)) return NOTIFY_DONE; - memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix, - sizeof(matrix_mdev->shadow_apcb)); - vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); + if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); return NOTIFY_OK; } @@ -1369,6 +1473,18 @@ static void vfio_ap_mdev_release(struct mdev_device *mdev) matrix_mdev->kvm = NULL; } + /* + * The shadow_apcb must be cleared. + * + * The shadow_apcb is committed to the guest only if the masks resulting + * from filtering the matrix_mdev->matrix differs from the masks in the + * shadow_apcb. Consequently, if we don't clear the masks here and a + * guest is subsequently started, the filtering may not result in a + * change to the shadow_apcb which will not get committed to the guest; + * in that case, the guest will be left without any queues. + */ + vfio_ap_matrix_clear_masks(&matrix_mdev->shadow_apcb); + mutex_unlock(&matrix_dev->lock); vfio_unregister_notifier(mdev_dev(mdev), VFIO_IOMMU_NOTIFY, @@ -1466,6 +1582,16 @@ static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q) } } +static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q) +{ + + if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + return; + + if (vfio_ap_mdev_filter_guest_matrix(q->matrix_mdev)) + vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev); +} + int vfio_ap_mdev_probe_queue(struct ap_device *apdev) { struct vfio_ap_queue *q; @@ -1482,11 +1608,36 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) q->apqn = queue->qid; q->saved_isc = VFIO_AP_ISC_INVALID; vfio_ap_queue_link_mdev(q); + vfio_ap_mdev_hot_plug_queue(q); mutex_unlock(&matrix_dev->lock); return 0; } +void vfio_ap_mdev_hot_unplug_queue(struct vfio_ap_queue *q) +{ + unsigned long apid = AP_QID_CARD(q->apqn); + + if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + return; + + /* + * If the APID is assigned to the guest, then let's + * go ahead and unplug the adapter since the + * architecture does not provide a means to unplug + * an individual queue. + */ + if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm)) { + clear_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm); + + if (bitmap_empty(q->matrix_mdev->shadow_apcb.apm, AP_DEVICES)) + bitmap_clear(q->matrix_mdev->shadow_apcb.aqm, 0, + AP_DOMAINS); + + vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev); + } +} + void vfio_ap_mdev_remove_queue(struct ap_device *apdev) { struct vfio_ap_queue *q; @@ -1497,6 +1648,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) mutex_lock(&matrix_dev->lock); q = dev_get_drvdata(&queue->ap_dev.device); + vfio_ap_mdev_hot_unplug_queue(q); dev_set_drvdata(&queue->ap_dev.device, NULL); apid = AP_QID_CARD(q->apqn); apqi = AP_QID_QUEUE(q->apqn); From patchwork Thu Oct 22 17:12:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 853A0C388F9 for ; Thu, 22 Oct 2020 17:13:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 279D82464B for ; Thu, 22 Oct 2020 17:13:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ggzJ/J0G" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S369070AbgJVRNU (ORCPT ); Thu, 22 Oct 2020 13:13:20 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:17932 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902409AbgJVRMo (ORCPT ); Thu, 22 Oct 2020 13:12:44 -0400 Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MGXp8F023651; Thu, 22 Oct 2020 13:12:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=QgAblzuxdam56JcSghBnHk8PBdkQTQNYppt6SwLuk6Q=; b=ggzJ/J0Gwbj3jYTdHCoDplgRgNOoG3KRA8Bdhm7h1zJJbtXkiAHdfk8ovMMWrfiRkyPc /g7WXYqLrEBSnXYrTDDOKTClYxcFQnNhuvtOWtQmsq6DorCevoogweCrzdMM2xauUvhC UK9J0TpLyZaiH1bVpF5QBqjy6pBToZlHWgoXLqw0kOQhuzLQYuvmoJDqhLdcfAyFmc2q K5b5gpz5UlqoFhu1+TCuJ1mjQH2o07afSK9FT3klPEHijILskd4kK9dJnlUax9GszoSW Rb9OPfh2UDQJ/V7CBs2BfNFyhlNL+au4AeYY8O1Vu9Xn0guEOfAiCt63zjWddzT45zGN 5Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b082tmu6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:42 -0400 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MGYiue027144; Thu, 22 Oct 2020 13:12:42 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b082tmtr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:42 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCMfC015470; Thu, 22 Oct 2020 17:12:41 GMT Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by ppma01dal.us.ibm.com with ESMTP id 347r89vbbh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:41 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCWvj32768446 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:32 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A8CF878064; Thu, 22 Oct 2020 17:12:37 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D29117805F; Thu, 22 Oct 2020 17:12:35 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:35 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 09/14] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device Date: Thu, 22 Oct 2020 13:12:04 -0400 Message-Id: <20201022171209.19494-10-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 priorityscore=1501 adultscore=0 impostorscore=0 bulkscore=0 phishscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The current implementation does not allow assignment of an AP adapter or domain to an mdev device if each APQN resulting from the assignment does not reference an AP queue device that is bound to the vfio_ap device driver. This patch allows assignment of AP resources to the matrix mdev as long as the APQNs resulting from the assignment: 1. Are not reserved by the AP BUS for use by the zcrypt device drivers. 2. Are not assigned to another matrix mdev. The rationale behind this is twofold: 1. The AP architecture does not preclude assignment of APQNs to an AP configuration that are not available to the system. 2. APQNs that do not reference a queue device bound to the vfio_ap device driver will not be assigned to the guest's CRYCB, so the guest will not get access to queues not bound to the vfio_ap driver. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 197 ++++-------------------------- 1 file changed, 26 insertions(+), 171 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 5b34bc8fca31..c2c6dcec8829 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -427,122 +427,6 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = { NULL, }; -struct vfio_ap_queue_reserved { - unsigned long *apid; - unsigned long *apqi; - bool reserved; -}; - -/** - * vfio_ap_has_queue - * - * @dev: an AP queue device - * @data: a struct vfio_ap_queue_reserved reference - * - * Flags whether the AP queue device (@dev) has a queue ID containing the APQN, - * apid or apqi specified in @data: - * - * - If @data contains both an apid and apqi value, then @data will be flagged - * as reserved if the APID and APQI fields for the AP queue device matches - * - * - If @data contains only an apid value, @data will be flagged as - * reserved if the APID field in the AP queue device matches - * - * - If @data contains only an apqi value, @data will be flagged as - * reserved if the APQI field in the AP queue device matches - * - * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if - * @data does not contain either an apid or apqi. - */ -static int vfio_ap_has_queue(struct device *dev, void *data) -{ - struct vfio_ap_queue_reserved *qres = data; - struct ap_queue *ap_queue = to_ap_queue(dev); - ap_qid_t qid; - unsigned long id; - - if (qres->apid && qres->apqi) { - qid = AP_MKQID(*qres->apid, *qres->apqi); - if (qid == ap_queue->qid) - qres->reserved = true; - } else if (qres->apid && !qres->apqi) { - id = AP_QID_CARD(ap_queue->qid); - if (id == *qres->apid) - qres->reserved = true; - } else if (!qres->apid && qres->apqi) { - id = AP_QID_QUEUE(ap_queue->qid); - if (id == *qres->apqi) - qres->reserved = true; - } else { - return -EINVAL; - } - - return 0; -} - -/** - * vfio_ap_verify_queue_reserved - * - * @matrix_dev: a mediated matrix device - * @apid: an AP adapter ID - * @apqi: an AP queue index - * - * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device - * driver according to the following rules: - * - * - If both @apid and @apqi are not NULL, then there must be an AP queue - * device bound to the vfio_ap driver with the APQN identified by @apid and - * @apqi - * - * - If only @apid is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apid - * - * - If only @apqi is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apqi - * - * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL. - */ -static int vfio_ap_verify_queue_reserved(unsigned long *apid, - unsigned long *apqi) -{ - int ret; - struct vfio_ap_queue_reserved qres; - - qres.apid = apid; - qres.apqi = apqi; - qres.reserved = false; - - ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &qres, vfio_ap_has_queue); - if (ret) - return ret; - - if (qres.reserved) - return 0; - - return -EADDRNOTAVAIL; -} - -static int -vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, - unsigned long apid) -{ - int ret; - unsigned long apqi; - unsigned long nbits = matrix_mdev->matrix.aqm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(&apid, NULL); - - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - #define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \ "already assigned to %s" @@ -608,6 +492,16 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev, return 0; } +static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev, + unsigned long *mdev_apm, + unsigned long *mdev_aqm) +{ + if (ap_apqn_in_matrix_owned_by_def_drv(mdev_apm, mdev_aqm)) + return -EADDRNOTAVAIL; + + return vfio_ap_mdev_verify_no_sharing(matrix_mdev, mdev_apm, mdev_aqm); +} + static bool vfio_ap_mdev_matrixes_equal(struct ap_matrix *matrix1, struct ap_matrix *matrix2) { @@ -840,33 +734,21 @@ static ssize_t assign_adapter_store(struct device *dev, if (apid > matrix_mdev->matrix.apm_max) return -ENODEV; - /* - * Set the bit in the AP mask (APM) corresponding to the AP adapter - * number (APID). The bits in the mask, from most significant to least - * significant bit, correspond to APIDs 0-255. - */ - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid); - if (ret) - goto done; - memset(apm, 0, sizeof(apm)); set_bit_inv(apid, apm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, apm, - matrix_mdev->matrix.aqm); - if (ret) - goto done; - + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev, apm, + matrix_mdev->matrix.aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); - ret = count; - -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_adapter); @@ -916,26 +798,6 @@ static ssize_t unassign_adapter_store(struct device *dev, } static DEVICE_ATTR_WO(unassign_adapter); -static int -vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev, - unsigned long apqi) -{ - int ret; - unsigned long apid; - unsigned long nbits = matrix_mdev->matrix.apm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(NULL, &apqi); - - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - /** * assign_domain_store * @@ -989,28 +851,21 @@ static ssize_t assign_domain_store(struct device *dev, if (apqi > max_apqi) return -ENODEV; - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi); - if (ret) - goto done; - memset(aqm, 0, sizeof(aqm)); set_bit_inv(apqi, aqm); - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev, - matrix_mdev->matrix.apm, aqm); - if (ret) - goto done; - + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev, matrix_mdev->matrix.apm, + aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); - ret = count; - -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_domain); From patchwork Thu Oct 22 17:12:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8755EC5DF9E for ; Thu, 22 Oct 2020 17:12:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1E5EB2464B for ; Thu, 22 Oct 2020 17:12:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="iWvZhRWc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902426AbgJVRMq (ORCPT ); Thu, 22 Oct 2020 13:12:46 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:45026 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902416AbgJVRMp (ORCPT ); Thu, 22 Oct 2020 13:12:45 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH2ThD138710; Thu, 22 Oct 2020 13:12:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=fEsR73KSzySNjHccG3Vkn3ywmRLH1siIOLgRb6GdUPk=; b=iWvZhRWc7nHCGRSozCCgB9RjcKm3N0/7sl3fVWH+el8KEMwlCc0dAcvEGtJTBlkqJqOu sEwAIoG2Q7GSclwqgImBi6ChLPZz8u+FBOHSvaPV/fG3p4yhWVi9/b2+DDiA9s0YG0ln JdFIKa1Taeu7V21KRhvLRmjCl1qpqyWMPlPpS9B/W5NS8wifuunpIsyy7jI1YTmloLxE OrwUyOCDAF1UhPxH+cs0hAM97GsPNClo0FAxAaXOPUpVDD9IwKThsBGqA0DuXyONNbnV zOd7HIk5xF3/YWDf3SU3EvzvHOlwvoxYNMbtXDFDHeOxqvWv0jIXa04yurzHgO1jFusf sw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34ba2gh22w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:44 -0400 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH2Zkk139396; Thu, 22 Oct 2020 13:12:44 -0400 Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com with ESMTP id 34ba2gh22k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:44 -0400 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCDed000598; Thu, 22 Oct 2020 17:12:43 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma02dal.us.ibm.com with ESMTP id 347r89vdsa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:43 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCdoX131796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:39 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AE3A778064; Thu, 22 Oct 2020 17:12:39 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D285D7805F; Thu, 22 Oct 2020 17:12:37 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:37 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 10/14] s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device Date: Thu, 22 Oct 2020 13:12:05 -0400 Message-Id: <20201022171209.19494-11-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 mlxlogscore=999 clxscore=1015 bulkscore=0 priorityscore=1501 adultscore=0 malwarescore=0 lowpriorityscore=0 phishscore=0 spamscore=0 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's hot plug/unplug adapters, domains and control domains assigned to or unassigned from an AP matrix mdev device while it is in use by a guest per the following: * Hot plug AP adapter: When the APID of an adapter is assigned to a matrix mdev in use by a KVM guest, the adapter will be hot plugged into the KVM guest as long as each APQN derived from the Cartesian product of the APID being assigned and the APQIs already assigned to the matrix mdev references a queue device bound to the vfio_ap device driver. * Hot unplug adapter: When the APID of an adapter is unassigned from a matrix mdev in use by a KVM guest, the adapter will be hot unplugged from the KVM guest. * Hot plug domain: When the APQI of a domain is assigned to a matrix mdev in use by a KVM guest, the domain will be hot plugged into the KVM guest as long as each APQN derived from the Cartesian product of the APQI being assigned and the APIDs already assigned to the matrix mdev references a queue device bound to the vfio_ap device driver. * Hot unplug domain: When the APQI of a domain is unassigned from a matrix mdev in use by a KVM guest, the domain will be hot unplugged from the KVM guest * Hot plug control domain: When the domain number of a control domain is assigned to a matrix mdev in use by a KVM guest, the control domain will be hot plugged into the KVM guest. The AP architecture ensures a guest will only get access to the control domain if it is in the host's AP configuration, so there is no risk in hot plugging it; however, it will become automatically available to the guest when it is added to the host configuration. * Hot unplug control domain: When the domain number of a control domain is unassigned from a matrix mdev in use by a KVM guest, the control domain will be hot unplugged from the KVM guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 148 ++++++++++++++++++++++++------ 1 file changed, 119 insertions(+), 29 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index c2c6dcec8829..dae1fba41941 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -517,12 +517,18 @@ static bool vfio_ap_mdev_matrixes_equal(struct ap_matrix *matrix1, * a matrix mdev's AP configuration and stores the result in the shadow copy of * the APCB used to supply a KVM guest's AP configuration. * + * Note: Filtering is applied only to adapters and domains. Changes to control + * domains will always be reflected in the shadow APCB. + * * @matrix_mdev: the matrix mdev whose AP configuration is to be filtered + * @filter_apid: indicates whether APIDs (true) or APQIs (false) shall be + * filtered * * Returns true if filtering has changed the shadow copy of the APCB used * to supply a KVM guest's AP configuration; otherwise, returns false. */ -static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev) +static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev, + bool filter_apid) { struct ap_matrix shadow_apcb; unsigned long apid, apqi, apqn; @@ -561,9 +567,15 @@ static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev) * the APID. */ apqn = AP_MKQID(apid, apqi); + if (!vfio_ap_mdev_get_queue(matrix_mdev, apqn)) { - clear_bit_inv(apid, shadow_apcb.apm); - break; + if (filter_apid) { + clear_bit_inv(apid, shadow_apcb.apm); + break; + } + + clear_bit_inv(apqi, shadow_apcb.aqm); + continue; } } @@ -723,10 +735,6 @@ static ssize_t assign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -746,12 +754,44 @@ static ssize_t assign_adapter_store(struct device *dev, } set_bit_inv(apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APID, apid); + + if (vfio_ap_mdev_has_crycb(matrix_mdev)) + if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, true)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_adapter); +static bool vfio_ap_mdev_unassign_guest_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) { + clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); + + /* + * If there are no APIDs assigned to the guest, then + * the guest will not have access to any queues, so + * let's also go ahead and unassign the APQIs. Keeping + * them around may yield unpredictable results during + * a probe that is not related to a host AP + * configuration change (i.e., an AP adapter is + * configured online). + */ + if (bitmap_empty(matrix_mdev->shadow_apcb.apm, + AP_DEVICES)) + bitmap_clear(matrix_mdev->shadow_apcb.aqm, 0, + AP_DOMAINS); + + return true; + } + } + + return false; +} + /** * unassign_adapter_store * @@ -778,10 +818,6 @@ static ssize_t unassign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -792,6 +828,9 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APID, apid); + + if (vfio_ap_mdev_unassign_guest_apid(matrix_mdev, apid)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -841,10 +880,6 @@ static ssize_t assign_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; - /* If the guest is running, disallow assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -863,12 +898,43 @@ static ssize_t assign_domain_store(struct device *dev, } set_bit_inv(apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, LINK_APQI, apqi); + + if (vfio_ap_mdev_has_crycb(matrix_mdev)) + if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, false)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_domain); +static bool vfio_ap_mdev_unassign_guest_apqi(struct ap_matrix_mdev *matrix_mdev, + unsigned long apqi) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) { + clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm); + + /* + * If there are no APQIs assigned to the guest, then + * the guest will not have access to any queues, so + * let's also go ahead and unassign the APIDs. Keeping + * them around may yield unpredictable results during + * a probe that is not related to a host AP + * configuration change (i.e., an AP adapter is + * configured online). + */ + if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, + AP_DOMAINS)) + bitmap_clear(matrix_mdev->shadow_apcb.apm, 0, + AP_DEVICES); + + return true; + } + } + + return false; +} /** * unassign_domain_store @@ -896,10 +962,6 @@ static ssize_t unassign_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -910,12 +972,29 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); vfio_ap_mdev_link_queues(matrix_mdev, UNLINK_APQI, apqi); + + if (vfio_ap_mdev_unassign_guest_apqi(matrix_mdev, apqi)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(unassign_domain); +static bool vfio_ap_mdev_assign_guest_cdom(struct ap_matrix_mdev *matrix_mdev, + unsigned long domid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (!test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) { + set_bit_inv(domid, matrix_mdev->shadow_apcb.adm); + + return true; + } + } + + return false; +} + /** * assign_control_domain_store * @@ -941,10 +1020,6 @@ static ssize_t assign_control_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &id); if (ret) return ret; @@ -959,12 +1034,29 @@ static ssize_t assign_control_domain_store(struct device *dev, */ mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); + if (vfio_ap_mdev_assign_guest_cdom(matrix_mdev, id)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; } static DEVICE_ATTR_WO(assign_control_domain); +static bool +vfio_ap_mdev_unassign_guest_cdom(struct ap_matrix_mdev *matrix_mdev, + unsigned long domid) +{ + if (vfio_ap_mdev_has_crycb(matrix_mdev)) { + if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) { + clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm); + + return true; + } + } + + return false; +} + /** * unassign_control_domain_store * @@ -991,10 +1083,6 @@ static ssize_t unassign_control_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_domid = matrix_mdev->matrix.adm_max; - /* If the guest is running, disallow un-assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &domid); if (ret) return ret; @@ -1003,6 +1091,8 @@ static ssize_t unassign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv(domid, matrix_mdev->matrix.adm); + if (vfio_ap_mdev_unassign_guest_cdom(matrix_mdev, domid)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -1216,7 +1306,7 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb, if (!vfio_ap_mdev_has_crycb(matrix_mdev)) return NOTIFY_DONE; - if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev)) + if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, true)) vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); return NOTIFY_OK; @@ -1443,7 +1533,7 @@ static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q) if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) return; - if (vfio_ap_mdev_filter_guest_matrix(q->matrix_mdev)) + if (vfio_ap_mdev_filter_guest_matrix(q->matrix_mdev, true)) vfio_ap_mdev_commit_shadow_apcb(q->matrix_mdev); } From patchwork Thu Oct 22 17:12:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EDC1C56202 for ; Thu, 22 Oct 2020 17:13:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0AF392464B for ; Thu, 22 Oct 2020 17:13:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="cWSdH4uB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2507414AbgJVRM7 (ORCPT ); Thu, 22 Oct 2020 13:12:59 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:29014 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902428AbgJVRMs (ORCPT ); Thu, 22 Oct 2020 13:12:48 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH4Eiw076443; Thu, 22 Oct 2020 13:12:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=8jnQpJX5qgVlgWO8LSXx9dD6GxrEPWhuVGLKBZw9+uw=; b=cWSdH4uBD+e0DUBBkgbeyYm2mpI7vEX3Gw2J4bYX+tTyjjwjGq6Vc5hwFKaa+MjIa8dd OWagwUQ3ka6g6PfglPS5J7lQ/0uMknTP4LB812dkEavLFdYHERiTTwQocrKGoudGqL+S nXaS0K/WKh7cMYasVD8YwCsPEQvE+yj2sAl9W3C+SVsOudmJBiJxHCgp9LlsL8dR1dV3 l9Kt9a8FFuDYbQo2ty5WcJC8lWy8V6LHzgY+a707P8+8WHmeOJUD9PqL8Cc+/ibADDSp Hrq0tQgWJaAnjU708rUMyIVp9/12eMbNs9c/8uY3jzkBGuWHFYGxV3EsUUq9KNDRimGe qw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34be0trhpp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:46 -0400 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH4anp079491; Thu, 22 Oct 2020 13:12:46 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 34be0trhnq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:45 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCLb5015337; Thu, 22 Oct 2020 17:12:45 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma01dal.us.ibm.com with ESMTP id 347r89vbc1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:45 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCfdZ28901690 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:41 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9AEDD78067; Thu, 22 Oct 2020 17:12:41 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E0E1B7805C; Thu, 22 Oct 2020 17:12:39 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:39 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 11/14] s390/zcrypt: Notify driver on config changed and scan complete callbacks Date: Thu, 22 Oct 2020 13:12:06 -0400 Message-Id: <20201022171209.19494-12-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 mlxscore=0 phishscore=0 spamscore=0 clxscore=1015 priorityscore=1501 lowpriorityscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220111 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This patch intruduces an extension to the ap bus to notify device drivers when the host AP configuration changes - i.e., adapters, domains or control domains are added or removed. To that end, two new callbacks are introduced for AP device drivers: void (*on_config_changed)(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); This callback is invoked at the start of the AP bus scan function when it determines that the host AP configuration information has changed since the previous scan. This is done by storing an old and current QCI info struct and comparing them. If there is any difference, the callback is invoked. Note that when the AP bus scan detects that AP adapters, domains or control domains have been removed from the host's AP configuration, it will remove the associated devices from the AP bus subsystem's device model. This callback gives the device driver a chance to respond to the removal of the AP devices from the host configuration prior to calling the device driver's remove callback. The primary purpose of this callback is to allow the vfio_ap driver to do a bulk unplug of all affected adapters, domains and control domains from affected guests rather than unplugging them one at a time when the remove callback is invoked. void (*on_scan_complete)(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); The on_scan_complete callback is invoked after the ap bus scan is complete if the host AP configuration data has changed. Note that when the AP bus scan detects that adapters, domains or control domains have been added to the host's configuration, it will create new devices in the AP bus subsystem's device model. The primary purpose of this callback is to allow the vfio_ap driver to do a bulk plug of all affected adapters, domains and control domains into affected guests rather than plugging them one at a time when the probe callback is invoked. Please note that changes to the apmask and aqmask do not trigger these two callbacks since the bus scan function is not invoked by changes to those masks. Signed-off-by: Harald Freudenberger Signed-off-by: Tony Krowiak --- drivers/s390/crypto/ap_bus.c | 88 ++++++++++++++++++++++++++- drivers/s390/crypto/ap_bus.h | 12 ++++ drivers/s390/crypto/vfio_ap_drv.c | 2 +- drivers/s390/crypto/vfio_ap_ops.c | 11 ++-- drivers/s390/crypto/vfio_ap_private.h | 2 +- 5 files changed, 106 insertions(+), 9 deletions(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index 998e61cd86d9..5b94956ef6bc 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -73,8 +73,10 @@ struct ap_perms ap_perms; EXPORT_SYMBOL(ap_perms); DEFINE_MUTEX(ap_perms_mutex); EXPORT_SYMBOL(ap_perms_mutex); +DEFINE_MUTEX(ap_config_lock); static struct ap_config_info *ap_qci_info; +static struct ap_config_info *ap_qci_info_old; /* * AP bus related debug feature things. @@ -1420,6 +1422,52 @@ static int __match_queue_device_with_queue_id(struct device *dev, const void *da && AP_QID_QUEUE(to_ap_queue(dev)->qid) == (int)(long) data; } +/* Helper function for notify_config_changed */ +static int __drv_notify_config_changed(struct device_driver *drv, void *data) +{ + struct ap_driver *ap_drv = to_ap_drv(drv); + + if (try_module_get(drv->owner)) { + if (ap_drv->on_config_changed) + ap_drv->on_config_changed(ap_qci_info, + ap_qci_info_old); + module_put(drv->owner); + } + + return 0; +} + +/* Notify all drivers about an qci config change */ +static inline void notify_config_changed(void) +{ + bus_for_each_drv(&ap_bus_type, NULL, NULL, + __drv_notify_config_changed); +} + +/* Helper function for notify_scan_complete */ +static int __drv_notify_scan_complete(struct device_driver *drv, void *data) +{ + struct ap_driver *ap_drv = to_ap_drv(drv); + + if (try_module_get(drv->owner)) { + if (ap_drv->on_scan_complete) + ap_drv->on_scan_complete(ap_qci_info, + ap_qci_info_old); + module_put(drv->owner); + } + + return 0; +} + +/* Notify all drivers about bus scan complete */ +static inline void notify_scan_complete(void) +{ + bus_for_each_drv(&ap_bus_type, NULL, NULL, + __drv_notify_scan_complete); +} + + + /* * Helper function for ap_scan_bus(). * Remove card device and associated queue devices. @@ -1696,15 +1744,45 @@ static inline void ap_scan_adapter(int ap) put_device(&ac->ap_dev.device); } +static int ap_config_changed(void) +{ + int cfg_chg = 0; + + if (ap_qci_info) { + if (!ap_qci_info_old) { + ap_qci_info_old = kzalloc(sizeof(*ap_qci_info_old), + GFP_KERNEL); + if (!ap_qci_info_old) + return 0; + } else { + memcpy(ap_qci_info_old, ap_qci_info, + sizeof(struct ap_config_info)); + } + ap_fetch_qci_info(ap_qci_info); + cfg_chg = memcmp(ap_qci_info, + ap_qci_info_old, + sizeof(struct ap_config_info)) != 0; + } + + return cfg_chg; +} + /** * ap_scan_bus(): Scan the AP bus for new devices * Runs periodically, workqueue timer (ap_config_time) */ static void ap_scan_bus(struct work_struct *unused) { - int ap; + int ap, config_changed = 0; + + mutex_lock(&ap_config_lock); - ap_fetch_qci_info(ap_qci_info); + /* config change notify */ + config_changed = ap_config_changed(); + if (config_changed) + notify_config_changed(); + memcpy(ap_qci_info_old, ap_qci_info, + sizeof(struct ap_config_info)); ap_select_domain(); AP_DBF_DBG("%s running\n", __func__); @@ -1713,6 +1791,12 @@ static void ap_scan_bus(struct work_struct *unused) for (ap = 0; ap <= ap_max_adapter_id; ap++) ap_scan_adapter(ap); + /* scan complete notify */ + if (config_changed) + notify_scan_complete(); + + mutex_unlock(&ap_config_lock); + /* check if there is at least one queue available with default domain */ if (ap_domain_index >= 0) { struct device *dev = diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 6ce154d924d3..c021ea5121a9 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -146,6 +146,18 @@ struct ap_driver { int (*probe)(struct ap_device *); void (*remove)(struct ap_device *); bool (*in_use)(unsigned long *apm, unsigned long *aqm); + /* + * Called at the start of the ap bus scan function when + * the crypto config information (qci) has changed. + */ + void (*on_config_changed)(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); + /* + * Called at the end of the ap bus scan function when + * the crypto config information (qci) has changed. + */ + void (*on_scan_complete)(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 8934471b7944..f06e19754de3 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -87,7 +87,7 @@ static int vfio_ap_matrix_dev_create(void) /* Fill in config info via PQAP(QCI), if available */ if (test_facility(12)) { - ret = ap_qci(&matrix_dev->info); + ret = ap_qci(&matrix_dev->config_info); if (ret) goto matrix_alloc_err; } diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index dae1fba41941..c4ea80ec8599 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -354,8 +354,9 @@ static int vfio_ap_mdev_create(struct kobject *kobj, struct mdev_device *mdev) } matrix_mdev->mdev = mdev; - vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); - vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb); + vfio_ap_matrix_init(&matrix_dev->config_info, &matrix_mdev->matrix); + vfio_ap_matrix_init(&matrix_dev->config_info, + &matrix_mdev->shadow_apcb); hash_init(matrix_mdev->qtable); mdev_set_drvdata(mdev, matrix_mdev); matrix_mdev->pqap_hook.hook = handle_pqap; @@ -540,8 +541,8 @@ static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev, * If the APID is not assigned to the host AP configuration, * we can not assign it to the guest's AP configuration */ - if (!test_bit_inv(apid, - (unsigned long *)matrix_dev->info.apm)) { + if (!test_bit_inv(apid, (unsigned long *) + matrix_dev->config_info.apm)) { clear_bit_inv(apid, shadow_apcb.apm); continue; } @@ -554,7 +555,7 @@ static int vfio_ap_mdev_filter_guest_matrix(struct ap_matrix_mdev *matrix_mdev, * guest's AP configuration */ if (!test_bit_inv(apqi, (unsigned long *) - matrix_dev->info.aqm)) { + matrix_dev->config_info.aqm)) { clear_bit_inv(apqi, shadow_apcb.aqm); continue; } diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index fc8634cee485..5065f0367ea2 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -40,7 +40,7 @@ struct ap_matrix_dev { struct device device; atomic_t available_instances; - struct ap_config_info info; + struct ap_config_info config_info; struct list_head mdev_list; struct mutex lock; struct ap_driver *vfio_ap_drv; From patchwork Thu Oct 22 17:12:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CA4BC388F7 for ; Thu, 22 Oct 2020 17:12:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E4F7D2464B for ; Thu, 22 Oct 2020 17:12:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="L3fC6kGv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2902468AbgJVRM5 (ORCPT ); Thu, 22 Oct 2020 13:12:57 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:15890 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902437AbgJVRMt (ORCPT ); Thu, 22 Oct 2020 13:12:49 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH4BQ0076214; Thu, 22 Oct 2020 13:12:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=enXwWcnHPKX/v2D/xvYGsc2U1NtmKX7RhDyfEftdVe8=; b=L3fC6kGv+uhvFBA7GtgSKX9m9/tBzrueZGS3mb38fu/4dnpc/i1sub/hJAcqQ82g7Rea LJf2ZJBMeKiL1n8Lm9AS1wNfKvrlaIk8YdUwRbZ2u/7uZiunGr1Hr/6mtzjAkGl8Jr7f UJrWVo0NJEnFPz2lzhY6nRwiOXNBYl39TrnV1cvRKek7OoNyFTOPX6J8gXuzxuRZjy28 PlpU8R1pG9eXaHzeT9ewvKWBLH4Pr5B7GRpv6Huu1UxSfxF23+sLW4Cel2eP63226fS7 ykgvbyMBIOB3H8r5EGIUf6aphzZCPGOgYkReP70MEf+/e4A9LZ3PDxI2F6DZvbB0losE Qw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34be0trhrd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:48 -0400 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH4N0p077472; Thu, 22 Oct 2020 13:12:48 -0400 Received: from ppma01wdc.us.ibm.com (fd.55.37a9.ip4.static.sl-reverse.com [169.55.85.253]) by mx0a-001b2d01.pphosted.com with ESMTP id 34be0trhq1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:48 -0400 Received: from pps.filterd (ppma01wdc.us.ibm.com [127.0.0.1]) by ppma01wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCiMe031884; Thu, 22 Oct 2020 17:12:46 GMT Received: from b03cxnp08027.gho.boulder.ibm.com (b03cxnp08027.gho.boulder.ibm.com [9.17.130.19]) by ppma01wdc.us.ibm.com with ESMTP id 347r89hw82-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:46 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCchn27919048 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:38 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B4D0B78060; Thu, 22 Oct 2020 17:12:43 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D94177805F; Thu, 22 Oct 2020 17:12:41 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:41 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 12/14] s390/vfio-ap: handle host AP config change notification Date: Thu, 22 Oct 2020 13:12:07 -0400 Message-Id: <20201022171209.19494-13-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 mlxscore=0 phishscore=0 spamscore=0 clxscore=1015 priorityscore=1501 lowpriorityscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220111 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The motivation for config change notification is to enable the vfio_ap device driver to handle hot plug/unplug of AP queues for a KVM guest as a bulk operation. For example, if a new APID is dynamically assigned to the host configuration, then a queue device will be created for each APQN that can be formulated from the new APID and all APQIs already assigned to the host configuration. Each of these new queue devices will get bound to their respective driver one at a time, as they are created. In the case of the vfio_ap driver, if the APQN of the queue device being bound to the driver is assigned to a matrix mdev in use by a KVM guest, it will be hot plugged into the guest if possible. Given that the AP architecture allows for 256 adapters and 256 domains, one can see the possibility of the vfio_ap driver's probe/remove callbacks getting invoked an inordinate number of times when the host configuration changes. Keep in mind that in order to plug/unplug an AP queue for a guest, the guest's VCPUs must be suspended, then the guest's AP configuration must be updated followed by the VCPUs being resumed. If this is done each time the probe or remove callback is invoked and there are hundreds or thousands of queues to be probed or removed, this would be incredibly inefficient and could have a large impact on guest performance. What the config notification does is allow us to make the changes to the guest in a single operation. This patch implements the on_cfg_changed callback which notifies the AP device drivers that the host AP configuration has changed (i.e., adapters, domains and/or control domains are added to or removed from the host AP configuration). Adapters added to host configuration: * The APIDs of the adapters added will be stored in a bitmap contained within the struct representing the matrix device which is the parent device of all matrix mediated devices. * When a queue is probed, if the APID of the queue being probed is contained in the bitmap of adapters added, the queue hot plug operation will be skipped until the AP bus notifies the driver that its scan operation has completed. Domains added to host configuration: * The APQIs of the domains added will be stored in a bitmap contained within the struct representing the matrix device which is the parent device of all matrix mediated devices. * When a queue is probed, if the APQI of the queue being probed is contained in the bitmap of domains added, the queue hot plug operation will be skipped until the AP bus notifies the driver that its scan operation has completed. Control domains added to the host configuration: * Since control domains are not devices in the linux device model, there is no concern with whether they are bound to a device driver. * The AP architecture will mask off control domains not in the host AP configuration from the guest, so there is also no concern about a guest changing a domain to which it is not authorized. Adapters removed from configuration: * Each adapter removed from the host configuration will be hot unplugged from each guest using it. * Each queue device with the APID identifying an adapter removed from the host AP configuration will be unlinked from the matrix mdev to which the queue's APQN is assigned. * When the vfio_ap driver's remove callback is invoked, if the queue device is not linked to the matrix mdev, the hot unplug operation will be skipped until the vfio_ap driver is notified that the AP bus scan has completed. Adapters removed from configuration: * Each domain removed from the host configuration will be hot unplugged from each guest using it. * Each queue device with the APQI identifying a domain removed from the host AP configuration will be unlinked from the matrix mdev to which the queue's APQN is assigned. * When the vfio_ap driver's remove callback is invoked, if the queue device is not linked to the matrix mdev, the hot unplug operation will be until the vfio_ap driver is notified that the AP bus scan has completed. Signed-off-by: Tony Krowiak Reported-by: kernel test robot Reported-by: kernel test robot --- drivers/s390/crypto/vfio_ap_drv.c | 3 + drivers/s390/crypto/vfio_ap_ops.c | 223 +++++++++++++++++++++++++- drivers/s390/crypto/vfio_ap_private.h | 11 ++ 3 files changed, 236 insertions(+), 1 deletion(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index f06e19754de3..d7aa5543afef 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -90,6 +90,8 @@ static int vfio_ap_matrix_dev_create(void) ret = ap_qci(&matrix_dev->config_info); if (ret) goto matrix_alloc_err; + memcpy(&matrix_dev->config_info_prev, &matrix_dev->config_info, + sizeof(struct ap_config_info)); } mutex_init(&matrix_dev->lock); @@ -149,6 +151,7 @@ static int __init vfio_ap_init(void) vfio_ap_drv.remove = vfio_ap_mdev_remove_queue; vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; + vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index c4ea80ec8599..075096adbfd3 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1530,8 +1530,13 @@ static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q) static void vfio_ap_mdev_hot_plug_queue(struct vfio_ap_queue *q) { + unsigned long apid = AP_QID_CARD(q->apqn); + unsigned long apqi = AP_QID_QUEUE(q->apqn); - if ((q->matrix_mdev == NULL) || !vfio_ap_mdev_has_crycb(q->matrix_mdev)) + if ((q->matrix_mdev == NULL) || + !vfio_ap_mdev_has_crycb(q->matrix_mdev) || + test_bit_inv(apid, matrix_dev->ap_add) || + test_bit_inv(apqi, matrix_dev->aq_add)) return; if (vfio_ap_mdev_filter_guest_matrix(q->matrix_mdev, true)) @@ -1616,3 +1621,219 @@ bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) return in_use; } + +/** + * vfio_ap_mdev_unassign_apids + * + * @matrix_mdev: The matrix mediated device + * + * @apid_rem: The bitmap specifying the APIDs of the adapters removed from + * the host's AP configuration + * + * Unassigns each APID specified in @apid_rem that is assigned to the + * shadow APCB. Returns true if at least one APID is unassigned; otherwise, + * returns false. + */ +static bool vfio_ap_mdev_unassign_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apid_rem) +{ + DECLARE_BITMAP(shadow_apm, AP_DEVICES); + + /* + * Get the result of filtering the APIDs removed from the host AP + * configuration out of the shadow APCB + */ + bitmap_andnot(shadow_apm, matrix_mdev->shadow_apcb.apm, apid_rem, + AP_DEVICES); + + /* + * If filtering removed any APIDs from the shadow APCB, then let's go + * ahead and update the shadow APCB accordingly + */ + if (!bitmap_equal(matrix_mdev->shadow_apcb.apm, shadow_apm, + AP_DEVICES)) { + memcpy(matrix_mdev->shadow_apcb.apm, shadow_apm, + sizeof(struct ap_matrix)); + + /* + * If all APIDs have been filtered from the shadow APCB, then + * let's also filter all of the APQIs. You need both APIDs and + * APQIs to identify the APQNs of the queues to assign to a + * guest. + */ + if (bitmap_empty(matrix_mdev->shadow_apcb.apm, AP_DEVICES)) + bitmap_clear(matrix_mdev->shadow_apcb.aqm, 0, + AP_DOMAINS); + + return true; + } + + return false; +} + +/* + * vfio_ap_mdev_unlink_apids + * + * @matrix_mdev: The matrix mediated device + * + * @apid_rem: The bitmap specifying the APIDs of the adapters removed from + * the host's AP configuration + * + * Unlinks @matrix_mdev from each queue assigned to @matrix_mdev whose APQN + * contains an APID specified in @apid_rem. + */ +static void vfio_ap_mdev_unlink_apids(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apid_rem) +{ + int bkt, apid; + struct vfio_ap_queue *q; + + hash_for_each(matrix_mdev->qtable, bkt, q, mdev_qnode) { + apid = AP_QID_CARD(q->apqn); + if (test_bit_inv(apid, apid_rem)) { + q->matrix_mdev = NULL; + hash_del(&q->mdev_qnode); + } + } +} + +/** + * vfio_ap_mdev_unassign_apqis + * + * @matrix_mdev: The matrix mediated device + * + * @apqi_rem: The bitmap specifying the APQIs of the domains removed from + * the host's AP configuration + * + * Unassigns each APQI specified in @apqi_rem that is assigned to the + * shadow APCB. Returns true if at least one APQI is unassigned; otherwise, + * returns false. + */ +static bool vfio_ap_mdev_unassign_apqis(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apqi_rem) +{ + DECLARE_BITMAP(shadow_aqm, AP_DOMAINS); + + /* + * Get the result of filtering the APQIs removed from the host AP + * configuration out of the shadow APCB + */ + bitmap_andnot(shadow_aqm, matrix_mdev->shadow_apcb.aqm, apqi_rem, + AP_DOMAINS); + + /* + * If filtering removed any APQIs from the shadow APCB, then let's go + * ahead and update the shadow APCB accordingly + */ + if (!bitmap_equal(matrix_mdev->shadow_apcb.aqm, shadow_aqm, + AP_DOMAINS)) { + memcpy(matrix_mdev->shadow_apcb.aqm, shadow_aqm, + sizeof(struct ap_matrix)); + + /* + * If all APQIs have been filtered from the shadow APCB, then + * let's also filter all of the APIDs. You need both APIDs and + * APQIs to identify the APQNs of the queues to assign to a + * guest. + */ + if (bitmap_empty(matrix_mdev->shadow_apcb.aqm, AP_DOMAINS)) + bitmap_clear(matrix_mdev->shadow_apcb.apm, 0, + AP_DEVICES); + + return true; + } + + return false; +} + +/* + * vfio_ap_mdev_unlink_apqis + * + * @matrix_mdev: The matrix mediated device + * + * @apqi_rem: The bitmap specifying the APQIs of the domains removed from + * the host's AP configuration + * + * Unlinks @matrix_mdev from each queue assigned to @matrix_mdev whose APQN + * contains an APQI specified in @apqi_rem. + */ +static void vfio_ap_mdev_unlink_apqis(struct ap_matrix_mdev *matrix_mdev, + unsigned long *apqi_rem) +{ + int bkt, apqi; + struct vfio_ap_queue *q; + + hash_for_each(matrix_mdev->qtable, bkt, q, mdev_qnode) { + apqi = AP_QID_QUEUE(q->apqn); + if (test_bit_inv(apqi, apqi_rem)) { + q->matrix_mdev = NULL; + hash_del(&q->mdev_qnode); + } + } +} + +static void vfio_ap_mdev_on_cfg_remove(void) +{ + bool unassigned = false; + int ap_remove, aq_remove; + struct ap_matrix_mdev *matrix_mdev; + DECLARE_BITMAP(apid_rem, AP_DEVICES); + DECLARE_BITMAP(apqi_rem, AP_DOMAINS); + unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm; + + cur_apm = (unsigned long *)matrix_dev->config_info.apm; + cur_aqm = (unsigned long *)matrix_dev->config_info.aqm; + prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm; + prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm; + + ap_remove = bitmap_andnot(apid_rem, prev_apm, cur_apm, AP_DEVICES); + aq_remove = bitmap_andnot(apqi_rem, prev_aqm, cur_aqm, AP_DOMAINS); + + if (!ap_remove && !aq_remove) + return; + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + continue; + + if (ap_remove) { + if (vfio_ap_mdev_unassign_apids(matrix_mdev, apid_rem)) + unassigned = true; + vfio_ap_mdev_unlink_apids(matrix_mdev, apid_rem); + } + + if (aq_remove) { + if (vfio_ap_mdev_unassign_apqis(matrix_mdev, apqi_rem)) + unassigned = true; + vfio_ap_mdev_unlink_apqis(matrix_mdev, apqi_rem); + } + } +} + +void vfio_ap_mdev_on_cfg_add(void) +{ + unsigned long *cur_apm, *cur_aqm, *prev_apm, *prev_aqm; + + cur_apm = (unsigned long *)matrix_dev->config_info.apm; + cur_aqm = (unsigned long *)matrix_dev->config_info.aqm; + + prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm; + prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm; + + bitmap_andnot(matrix_dev->ap_add, cur_apm, prev_apm, AP_DEVICES); + bitmap_andnot(matrix_dev->aq_add, cur_aqm, prev_aqm, AP_DOMAINS); +} + +void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info) +{ + mutex_lock(&matrix_dev->lock); + memcpy(&matrix_dev->config_info, new_config_info, + sizeof(struct ap_config_info)); + memcpy(&matrix_dev->config_info_prev, old_config_info, + sizeof(struct ap_config_info)); + + vfio_ap_mdev_on_cfg_remove(); + vfio_ap_mdev_on_cfg_add(); + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 5065f0367ea2..64f1f5b820f6 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -36,14 +36,21 @@ * driver, be it using @mdev_list or writing the state of a * single ap_matrix_mdev device. It's quite coarse but we don't * expect much contention. + ** @ap_add: a bitmap specifying the APIDs added to the host AP configuration + * as notified by the AP bus via the on_cfg_chg callback. + * @aq_add: a bitmap specifying the APQIs added to the host AP configuration + * as notified by the AP bus via the on_cfg_chg callback. */ struct ap_matrix_dev { struct device device; atomic_t available_instances; struct ap_config_info config_info; + struct ap_config_info config_info_prev; struct list_head mdev_list; struct mutex lock; struct ap_driver *vfio_ap_drv; + DECLARE_BITMAP(ap_add, AP_DEVICES); + DECLARE_BITMAP(aq_add, AP_DEVICES); }; extern struct ap_matrix_dev *matrix_dev; @@ -90,6 +97,8 @@ struct ap_matrix_mdev { struct kvm_s390_module_hook pqap_hook; struct mdev_device *mdev; DECLARE_HASHTABLE(qtable, 8); + DECLARE_BITMAP(ap_add, AP_DEVICES); + DECLARE_BITMAP(aq_add, AP_DEVICES); }; extern int vfio_ap_mdev_register(void); @@ -108,5 +117,7 @@ int vfio_ap_mdev_probe_queue(struct ap_device *queue); void vfio_ap_mdev_remove_queue(struct ap_device *queue); bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); +void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Thu Oct 22 17:12:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 216E4C388F7 for ; Thu, 22 Oct 2020 17:13:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B0B8724673 for ; Thu, 22 Oct 2020 17:13:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Id9vabkn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S369106AbgJVRNx (ORCPT ); Thu, 22 Oct 2020 13:13:53 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:13448 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2898968AbgJVRNw (ORCPT ); Thu, 22 Oct 2020 13:13:52 -0400 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MH3gvV030206; Thu, 22 Oct 2020 13:13:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=IS0bnmL5yTCvtacFu3pktFy4Bzp3gilOC6zZ20NXIHM=; b=Id9vabknlzN2cLUgtqAeSIIjv1B1VWgfO8UHRyksTrAFgdgM6m+psJPtnEx/OnLXjqmz CbKR4RixcSJ0SNfvhJztoxx5GP7N871hO1cahlTyK8Ytz6DgSYi1/1Gxxje8QUYK7p2C BsuotjwtrpwrHoD7LFV7RYctNEtAsyaGRB610ffHg8GBxU6K+rKA3c/xDMdHn81LRaTJ zBGiN120rBHONySyNFxuVXrEocKckjdTvLKFcEcUVXJ0V4JEZysqXwWmd2G+DzCxIDqU XphQsq6vipZ0+AFEG61AY9D/YLnvhKUfcn01A9KkojfDJSK3i685Nob3EB6wguE2/Z/d jA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 34be27rjb7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:13:49 -0400 Received: from m0098414.ppops.net (m0098414.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MH3uls031796; Thu, 22 Oct 2020 13:13:49 -0400 Received: from ppma03wdc.us.ibm.com (ba.79.3fa9.ip4.static.sl-reverse.com [169.63.121.186]) by mx0b-001b2d01.pphosted.com with ESMTP id 34be27rjax-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:13:49 -0400 Received: from pps.filterd (ppma03wdc.us.ibm.com [127.0.0.1]) by ppma03wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MHCxoL014939; Thu, 22 Oct 2020 17:13:48 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma03wdc.us.ibm.com with ESMTP id 347r89hypn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:13:48 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCjAK49021358 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:45 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78FFD7805F; Thu, 22 Oct 2020 17:12:45 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E04D778060; Thu, 22 Oct 2020 17:12:43 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:43 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 13/14] s390/vfio-ap: handle AP bus scan completed notification Date: Thu, 22 Oct 2020 13:12:08 -0400 Message-Id: <20201022171209.19494-14-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 bulkscore=0 adultscore=0 spamscore=0 lowpriorityscore=0 malwarescore=0 suspectscore=0 clxscore=1015 mlxlogscore=999 mlxscore=0 priorityscore=1501 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Implements the driver callback invoked by the AP bus when the AP bus scan has completed. Since this callback is invoked after binding the newly added devices to their respective device drivers, the vfio_ap driver will attempt to hot plug the adapters, domains and control domains into each guest using the matrix mdev to which they are assigned. Keep in mind that an adapter or domain can be plugged in only if each APQN with the APID of the adapter or the APQI of the domain references a queue device bound to the vfio_ap device driver. Consequently, not all newly added adapters and domains will necessarily get hot plugged. The same filtering operation used when the guest is started will again be used to filter the APQNs assigned to the guest when the vfio_ap driver is notified the AP bus scan has completed for those matrix mediated devices to which the newly added APID(s) and/or APQI(s) are assigned. To recap the filtering process employed: For each APQN formulated from the Cartesian product of the APIDs and APQIs assigned to the matrix mdev, if the APQN does not reference a queue device bound to the vfio_ap device driver, the APID will not be hot plugged into the guest. If any APIDs are left after filtering, all of the queues referenced by the APQNs formulated by the remaining APIDs and the APQIs assigned to the matrix mdev will be hot plugged into the guest. Control domains will not be filtered and will always be hot plugged. Example: ======= Queue devices bound to vfio_ap device driver: 04.0004 04.0047 04.0054 05.0005 05.0047 Adapters and domains assigned to matrix mdev: Adapters Domains -> Queues 04 0004 04.0004 05 0047 04.0047 0054 04.0054 05.0004 05.0047 05.0054 KVM guest matrix after filtering: Adapters Domains -> Queues 04 0004 04.0004 0047 04.0047 0054 04.0054 Adapter 05 is filtered because queue 05.0054 is not bound. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 1 + drivers/s390/crypto/vfio_ap_ops.c | 26 ++++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 2 ++ 3 files changed, 29 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index d7aa5543afef..357481e80b0a 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -152,6 +152,7 @@ static int __init vfio_ap_init(void) vfio_ap_drv.in_use = vfio_ap_mdev_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; vfio_ap_drv.on_config_changed = vfio_ap_on_cfg_changed; + vfio_ap_drv.on_scan_complete = vfio_ap_on_scan_complete; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 075096adbfd3..824f936364ba 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1837,3 +1837,29 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, vfio_ap_mdev_on_cfg_add(); mutex_unlock(&matrix_dev->lock); } + +void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info) +{ + struct ap_matrix_mdev *matrix_mdev; + + mutex_lock(&matrix_dev->lock); + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (!vfio_ap_mdev_has_crycb(matrix_mdev)) + continue; + + if (!bitmap_intersects(matrix_mdev->matrix.apm, + matrix_dev->ap_add, AP_DEVICES) && + !bitmap_intersects(matrix_mdev->matrix.aqm, + matrix_dev->aq_add, AP_DOMAINS)) + continue; + + if (vfio_ap_mdev_filter_guest_matrix(matrix_mdev, + true)) + vfio_ap_mdev_commit_shadow_apcb(matrix_mdev); + } + + bitmap_clear(matrix_dev->ap_add, 0, AP_DEVICES); + bitmap_clear(matrix_dev->aq_add, 0, AP_DOMAINS); + mutex_unlock(&matrix_dev->lock); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index 64f1f5b820f6..d82d1e62cb2f 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -119,5 +119,7 @@ void vfio_ap_mdev_remove_queue(struct ap_device *queue); bool vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm); void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); +void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, + struct ap_config_info *old_config_info); #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Thu Oct 22 17:12:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 11851649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADFB1C388F7 for ; Thu, 22 Oct 2020 17:13:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 390992464B for ; Thu, 22 Oct 2020 17:13:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Wwj4mQqE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S369065AbgJVRNH (ORCPT ); Thu, 22 Oct 2020 13:13:07 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:52558 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2902462AbgJVRM6 (ORCPT ); Thu, 22 Oct 2020 13:12:58 -0400 Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MGXk7M023365; Thu, 22 Oct 2020 13:12:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=xW+57D3QqtwJU1Jpgq7Tu6eCFRAOKLSfskS/Mxa7NoI=; b=Wwj4mQqEIefemW1SQudNGvgNZ649j6BoGdX782ztVwiPNfKnZjDJpJ2XlE1qcrmSA6kw jz45choHQogzPVA0eGyJGK6NZv0ApRaTH+3iAIMIxnnl/8/Dq7DCqV002pMW/7A5Bq+V N6MTy6KJf9HN6CFrgNomoRnwWmeN5SVPup9CJu+3V3IR5rh5mITHUfzugq87J3aXBkV+ eL/viMIsabjFdt+qjPD9WpXRN5Qsp/k2VAgjj/0WOGyB36dz93m+71MCzbiZhwCbiwB2 i0GNLJS4AQ1YH3Gt3Tp8wTNcAulMqG+U0yfiR1foA04kiPxm9N91zCkzBpG966qfyaUX 9A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b082tn0y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:56 -0400 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 09MGYCO7025166; Thu, 22 Oct 2020 13:12:55 -0400 Received: from ppma02wdc.us.ibm.com (aa.5b.37a9.ip4.static.sl-reverse.com [169.55.91.170]) by mx0a-001b2d01.pphosted.com with ESMTP id 34b082tn0a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 13:12:55 -0400 Received: from pps.filterd (ppma02wdc.us.ibm.com [127.0.0.1]) by ppma02wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 09MGslNu030777; Thu, 22 Oct 2020 17:12:54 GMT Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by ppma02wdc.us.ibm.com with ESMTP id 347r89hwvp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 22 Oct 2020 17:12:54 +0000 Received: from b03ledav004.gho.boulder.ibm.com (b03ledav004.gho.boulder.ibm.com [9.17.130.235]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 09MHCeJ048300530 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Oct 2020 17:12:40 GMT Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 00B7578064; Thu, 22 Oct 2020 17:12:48 +0000 (GMT) Received: from b03ledav004.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B48067805F; Thu, 22 Oct 2020 17:12:45 +0000 (GMT) Received: from localhost.localdomain.com (unknown [9.85.170.177]) by b03ledav004.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 22 Oct 2020 17:12:45 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, fiuczy@linux.ibm.com, frankja@linux.ibm.com, david@redhat.com, hca@linux.ibm.com, gor@linux.ibm.com, Tony Krowiak Subject: [PATCH v11 14/14] s390/vfio-ap: update docs to include dynamic config support Date: Thu, 22 Oct 2020 13:12:09 -0400 Message-Id: <20201022171209.19494-15-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20201022171209.19494-1-akrowiak@linux.ibm.com> References: <20201022171209.19494-1-akrowiak@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_12:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 priorityscore=1501 adultscore=0 impostorscore=0 bulkscore=0 phishscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220108 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Update the documentation in vfio-ap.rst to include information about the AP dynamic configuration support (i.e., hot plug of adapters, domains and control domains via the matrix mediated device's sysfs assignment attributes). Signed-off-by: Tony Krowiak --- Documentation/s390/vfio-ap.rst | 362 ++++++++++++++++++++++++++------- 1 file changed, 285 insertions(+), 77 deletions(-) diff --git a/Documentation/s390/vfio-ap.rst b/Documentation/s390/vfio-ap.rst index e15436599086..888e15dbefc0 100644 --- a/Documentation/s390/vfio-ap.rst +++ b/Documentation/s390/vfio-ap.rst @@ -253,7 +253,7 @@ The process for reserving an AP queue for use by a KVM guest is: 1. The administrator loads the vfio_ap device driver 2. The vfio-ap driver during its initialization will register a single 'matrix' device with the device core. This will serve as the parent device for - all mediated matrix devices used to configure an AP matrix for a guest. + all matrix mediated devices used to configure an AP matrix for a guest. 3. The /sys/devices/vfio_ap/matrix device is created by the device core 4. The vfio_ap device driver will register with the AP bus for AP queue devices of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap @@ -269,7 +269,7 @@ The process for reserving an AP queue for use by a KVM guest is: default zcrypt cex4queue driver. 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for it. -9. The administrator creates a passthrough type mediated matrix device to be +9. The administrator creates a passthrough type matrix mediated device to be used by a guest 10. The administrator assigns the adapters, usage domains and control domains to be exclusively used by a guest. @@ -279,14 +279,14 @@ Set up the VFIO mediated device interfaces The VFIO AP device driver utilizes the common interface of the VFIO mediated device core driver to: -* Register an AP mediated bus driver to add a mediated matrix device to and +* Register an AP mediated bus driver to add a matrix mediated device to and remove it from a VFIO group. -* Create and destroy a mediated matrix device -* Add a mediated matrix device to and remove it from the AP mediated bus driver -* Add a mediated matrix device to and remove it from an IOMMU group +* Create and destroy a matrix mediated device +* Add a matrix mediated device to and remove it from the AP mediated bus driver +* Add a matrix mediated device to and remove it from an IOMMU group The following high-level block diagram shows the main components and interfaces -of the VFIO AP mediated matrix device driver:: +of the VFIO AP matrix mediated device driver:: +-------------+ | | @@ -351,29 +351,37 @@ matrix device. This attribute group identifies the user-defined sysfs attributes of the mediated device. When a device is registered with the VFIO mediated device framework, the sysfs attribute files identified in the 'mdev_attr_groups' - structure will be created in the mediated matrix device's directory. The - sysfs attributes for a mediated matrix device are: + structure will be created in the matrix mediated device's directory. The + sysfs attributes for a matrix mediated device are: assign_adapter / unassign_adapter: Write-only attributes for assigning/unassigning an AP adapter to/from the - mediated matrix device. To assign/unassign an adapter, the APID of the + matrix mediated device. To assign/unassign an adapter, the APID of the adapter is echoed to the respective attribute file. assign_domain / unassign_domain: Write-only attributes for assigning/unassigning an AP usage domain to/from - the mediated matrix device. To assign/unassign a domain, the domain + the matrix mediated device. To assign/unassign a domain, the domain number of the usage domain is echoed to the respective attribute file. matrix: - A read-only file for displaying the APQNs derived from the cross product - of the adapter and domain numbers assigned to the mediated matrix device. + A read-only file for displaying the APQNs derived from the Cartesian + product of the adapter and domain numbers assigned to the mediated matrix + device. + guest_matrix: + A read-only file for displaying the APQNs derived from the Cartesian + product of the adapter and domain numbers assigned to the APM and AQM + fields respectively of the KVM guest's CRYCB. This will differ from the + matrix if any APQNs assigned to the matrix mediated device do not + reference a queue device bound to the vfio_ap device driver (i.e., the + queue is not in the AP configuration). assign_control_domain / unassign_control_domain: Write-only attributes for assigning/unassigning an AP control domain - to/from the mediated matrix device. To assign/unassign a control domain, + to/from the matrix mediated device. To assign/unassign a control domain, the ID of the domain to be assigned/unassigned is echoed to the respective attribute file. control_domains: A read-only file for displaying the control domain numbers assigned to the - mediated matrix device. + matrix mediated device. * functions: @@ -385,7 +393,7 @@ matrix device. domains assigned via the corresponding sysfs attributes files remove: - deallocates the mediated matrix device's ap_matrix_mdev structure. This will + deallocates the matrix mediated device's ap_matrix_mdev structure. This will be allowed only if a running guest is not using the mdev. * callback interfaces @@ -397,7 +405,7 @@ matrix device. for the mdev matrix device to the MDEV bus. Access to the KVM structure used to configure the KVM guest is provided via this callback. The KVM structure, is used to configure the guest's access to the AP matrix defined via the - mediated matrix device's sysfs attribute files. + matrix mediated device's sysfs attribute files. release: unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix device and deconfigures the guest's AP matrix. @@ -410,11 +418,49 @@ function is called when QEMU connects to KVM. The guest's AP matrix is configured via it's CRYCB by: * Setting the bits in the APM corresponding to the APIDs assigned to the - mediated matrix device via its 'assign_adapter' interface. + matrix mediated device via its 'assign_adapter' interface. * Setting the bits in the AQM corresponding to the domains assigned to the - mediated matrix device via its 'assign_domain' interface. + matrix mediated device via its 'assign_domain' interface. * Setting the bits in the ADM corresponding to the domain dIDs assigned to the - mediated matrix device via its 'assign_control_domains' interface. + matrix mediated device via its 'assign_control_domains' interface. + +The linux device model precludes passing a device through to a KVM guest that +is not bound to the device driver facilitating its pass-through. Consequently, +an APQN that does not reference a queue device bound to the vfio_ap device +driver will not be assigned to a KVM guest's CRYCB. The AP architecture, +however, does not provide a means to filter individual APQNs from the guest's +CRYCB, so the following logic is employed to filter them: + +* Filter the APQNs assigned to the matrix mediated device by APID. + + To filter APQNs by APID, each APQN derived from the Cartesian product of the + adapter numbers (APID) and domain numbers (APQI) assigned to the mdev is + examined and if any one of them does not reference a queue device bound to the + vfio_ap device driver, the adapter will not be plugged into the guest (i.e., + the bit corresponding to its APID will not be set in the APM of the guest's + CRYCB). + + If at least one adapter is plugged into the guest, then all domains assigned + to the mdev will also be plugged into the guest (i.e., the bits corresponding + to the APQIs of the domains assigned to the mdev will be set in the AQM field + of the guest's CRYCB). + +* Filter the APQNs assigned to the matrix mediated device by APQI. + + The APQNs will be filtered by APQI if filtering by APID does not result in any + adapters or domains getting plugged into the guest. + + To filter APQNs by APQI, each APQN derived from the Cartesian product of the + adapter numbers (APID) and domain numbers (APQI) assigned to the mdev is + examined and if any one of them does not reference a queue device bound to the + vfio_ap device driver, the domain will not be plugged into the guest (i.e., + the bit corresponding to its APQI will not be set in the AQM of the guest's + CRYCB). + + If at least one domain is plugged into the guest, then all adapters assigned + to the mdev will also be plugged into the guest (i.e., the bits corresponding + to the APIDs of the adapters assigned to the mdev will be set in the APM field + of the guest's CRYCB). The CPU model features for AP ----------------------------- @@ -435,6 +481,10 @@ available to a KVM guest via the following CPU model features: can be made available to the guest only if it is available on the host (i.e., facility bit 12 is set). +4. apqi: Indicates AP queue interrupts are available on the guest. This facility + can be made available to the guest only if it is available on the host (i.e., + facility bit 65 is set). + Note: If the user chooses to specify a CPU model different than the 'host' model to QEMU, the CPU model features and facilities need to be turned on explicitly; for example:: @@ -444,7 +494,7 @@ explicitly; for example:: A guest can be precluded from using AP features/facilities by turning them off explicitly; for example:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off + /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off Note: If the APFT facility is turned off (apft=off) for the guest, the guest will not see any AP devices. The zcrypt device drivers that register for type 10 @@ -530,40 +580,56 @@ These are the steps: 2. Secure the AP queues to be used by the three guests so that the host can not access them. To secure them, there are two sysfs files that specify - bitmasks marking a subset of the APQN range as 'usable by the default AP - queue device drivers' or 'not usable by the default device drivers' and thus - available for use by the vfio_ap device driver'. The location of the sysfs - files containing the masks are:: + bitmasks marking a subset of the APQN range as usable only by the default AP + queue device drivers. All remaining APQNs are available available for use by + any other device driver. The vfio_ap device driver is currently the only + non-default device driver. The location of the sysfs files containing the + masks are:: /sys/bus/ap/apmask /sys/bus/ap/aqmask The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs - (APID). Each bit in the mask, from left to right (i.e., from most significant - to least significant bit in big endian order), corresponds to an APID from - 0-255. If a bit is set, the APID is marked as usable only by the default AP - queue device drivers; otherwise, the APID is usable by the vfio_ap - device driver. + (APID). Each bit in the mask, from left to right corresponds to an APID from + 0-255. If a bit is set, the APID is marked as available to the default AP + queue device drivers. The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes - (APQI). Each bit in the mask, from left to right (i.e., from most significant - to least significant bit in big endian order), corresponds to an APQI from - 0-255. If a bit is set, the APQI is marked as usable only by the default AP - queue device drivers; otherwise, the APQI is usable by the vfio_ap device - driver. + (APQI). Each bit in the mask, from left to right corresponds to an APQI from + 0-255. If a bit is set, the APQI is marked as available to the default AP + queue device drivers. + + The Cartesian product of the APIDs corresponding to the bits set in the + apmask and the APQIs corresponding to the bits set in the aqmask comprise + the subset of APQNs that can be used only by the host default device drivers. + All other APQNs are available to the non-default device drivers such as the + vfio_ap driver. + + Take, for example, the following masks:: + + apmask: + 0x7d00000000000000000000000000000000000000000000000000000000000000 + + aqmask: + 0x8000000000000000000000000000000000000000000000000000000000000000 + + The masks indicate: - Take, for example, the following mask:: + * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default + device drivers. - 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff + * Domain 0 is available for use by the host default device drivers - It indicates: + * The subset of APQNs available for use only by the default host device + drivers are: - 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6 - belong to the vfio_ap device driver's pool. + (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0) + + * All other APQNs are available for use by the non-default device drivers. The APQN of each AP queue device assigned to the linux host is checked by the - AP bus against the set of APQNs derived from the cross product of APIDs - and APQIs marked as usable only by the default AP queue device drivers. If a + AP bus against the set of APQNs derived from the Cartesian product of APIDs + and APQIs marked as available to the default AP queue device drivers. If a match is detected, only the default AP queue device drivers will be probed; otherwise, the vfio_ap device driver will be probed. @@ -627,6 +693,16 @@ These are the steps: default drivers pool: adapter 0-15, domain 1 alternate drivers pool: adapter 16-255, domains 0, 2-255 + Note ***: + Changing a mask such that one or more APQNs will be taken from a matrix + mediated device (see below) will fail with an error (EADDRINUSE). The error + is logged to the kernel ring buffer which can be viewed with the 'dmesg' + command. The output identifies each APQN flagged as 'in use' and the matrix + mediated device to which it is assigned; for example: + + Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804 + Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4 + Securing the APQNs for our example ---------------------------------- To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, @@ -684,7 +760,7 @@ Securing the APQNs for our example /sys/devices/vfio_ap/matrix/ --- [mdev_supported_types] - ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) + ------ [vfio_ap-passthrough] (passthrough matrix mediated device type) --------- create --------- [devices] @@ -775,17 +851,18 @@ Securing the APQNs for our example higher than the maximum is specified, the operation will terminate with an error (ENODEV). - * All APQNs that can be derived from the adapter ID and the IDs of - the previously assigned domains must be bound to the vfio_ap device - driver. If no domains have yet been assigned, then there must be at least - one APQN with the specified APID bound to the vfio_ap driver. If no such - APQNs are bound to the driver, the operation will terminate with an - error (EADDRNOTAVAIL). + * All APQNs that can be derived from the Cartesian product of the APID of the + adapter being assigned and the APQIs of the previously assigned domains + must be available to the vfio_ap device driver as specified in the sysfs + /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even one APQN + is reserved for use by the host device driver, the operation will terminate + with an error (EADDRNOTAVAIL). - No APQN that can be derived from the adapter ID and the IDs of the - previously assigned domains can be assigned to another mediated matrix - device. If an APQN is assigned to another mediated matrix device, the - operation will terminate with an error (EADDRINUSE). + * No APQN that can be derived from the Cartesian product of the APID of the + adapter being assigned and the APQIs of the previously assigned domains can + be assigned to another matrix mediated device. If even one APQN is assigned + to another matrix mediated device, the operation will terminate with an + error (EADDRINUSE). In order to successfully assign a domain: @@ -794,17 +871,18 @@ Securing the APQNs for our example higher than the maximum is specified, the operation will terminate with an error (ENODEV). - * All APQNs that can be derived from the domain ID and the IDs of - the previously assigned adapters must be bound to the vfio_ap device - driver. If no domains have yet been assigned, then there must be at least - one APQN with the specified APQI bound to the vfio_ap driver. If no such - APQNs are bound to the driver, the operation will terminate with an - error (EADDRNOTAVAIL). + * All APQNs that can be derived from the Cartesian product of the APQI of the + domain being assigned and the APIDs of the previously assigned adapters + must be available to the vfio_ap device driver as specified in the sysfs + /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even one APQN + is reserved for use by the host device driver, the operation will terminate + with an error (EADDRNOTAVAIL). - No APQN that can be derived from the domain ID and the IDs of the - previously assigned adapters can be assigned to another mediated matrix - device. If an APQN is assigned to another mediated matrix device, the - operation will terminate with an error (EADDRINUSE). + * No APQN that can be derived from the Cartesian product of the APQI of the + domain being assigned and the APIDs of the previously assigned adapters can + be assigned to another matrix mediated device. If even one APQN is assigned + to another matrix mediated device, the operation will terminate with an + error (EADDRINUSE). In order to successfully assign a control domain, the domain number specified must represent a value from 0 up to the maximum domain number @@ -813,22 +891,22 @@ Securing the APQNs for our example 5. Start Guest1:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ... 7. Start Guest2:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ... 7. Start Guest3:: - /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ + /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \ -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ... -When the guest is shut down, the mediated matrix devices may be removed. +When the guest is shut down, the matrix mediated devices may be removed. -Using our example again, to remove the mediated matrix device $uuid1:: +Using our example again, to remove the matrix mediated device $uuid1:: /sys/devices/vfio_ap/matrix/ --- [mdev_supported_types] @@ -851,16 +929,146 @@ remove it if no guest will use it during the remaining lifetime of the linux host. If the mdev matrix device is removed, one may want to also reconfigure the pool of adapters and queues reserved for use by the default drivers. +Hot plug support: +================ +An adapter, domain or control domain may be hot plugged into a running KVM +guest by assigning it to the matrix mediated device being used by the guest. +Control domains will always be hot plugged; however, an adapter or domain will +be hot plugged only if each new APQN resulting from its assignment +references a queue device bound to the vfio_ap device driver as described +below. + +When an adapter is assigned to a matrix mediated device in use by a KVM guest: + +* If no domains have yet been plugged into the KVM guest: + + Hot plug the adapter and every domain previously assigned to the mdev if each + APQN derived from the Cartesian product of the APID of the adapter being + assigned and the APQIs of the domains previously assigned references a queue + device bound to the vfio_ap device driver. + +* If one or more domains have previously been plugged into the guest: + + Hot plug the adapter if each APQN derived from the Cartesian product of the + APID of the adapter being assigned and the APQIs of the domains already + plugged into the guest references a queue device bound to the vfio_ap device + driver. + +When a domain is assigned to a matrix mediated device in use by a KVM guest: + +* If no adapters have yet been plugged into the KVM guest: + + Hot plug the domain and every adapter previously assigned to the mdev if each + APQN derived from the Cartesian product of the APIDs of the adapters + previously assigned and the APQI of the domain being assigned references a + queue device bound to the vfio_ap device driver. + +* If one or more adapters have previously been plugged into the guest: + + Hot plug the domain if each APQN derived from the Cartesian product of the + APIDs of the adapters already plugged into the guest and the APQI of the + domain being assigned references a queue device bound to the vfio_ap device + driver. + +Over-provisioning of AP queues for a KVM guest: +============================================== +Over-provisioning is defined herein as the assignment of adapters or domains to +a matrix mediated device that do not reference AP devices in the host's AP +configuration. The idea here is that when the adapter or domain becomes +available, it will be automatically hot-plugged into the KVM guest using +the matrix mediated device to which it is assigned as long as each new APQN +resulting from plugging it in references a queue device bound to the vfio_ap +device driver. + Limitations =========== -* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN - to the default drivers pool of a queue that is still assigned to a mediated - device in use by a guest. It is incumbent upon the administrator to - ensure there is no mediated device in use by a guest to which the APQN is - assigned lest the host be given access to the private data of the AP queue - device such as a private key configured specifically for the guest. +Live guest migration is not supported for guests using AP devices without +intervention by a system administrator. Before a KVM guest can be migrated, +the matrix mediated device must be removed. Unfortunately, it can not be +removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while +the mdev is in use by a KVM guest. If the guest is being emulated by QEMU, +its mdev can be hot unplugged from the guest in one of two ways: + +1. If the KVM guest was started with libvirt, you can hot unplug the mdev via + the following commands: + + virsh detach-device + + For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from + the guest named 'my-guest': + + virsh detach-device my-guest ~/config/my-guest-hostdev.xml + + The contents of my-guest-hostdev.xml: + + + +
+ + + + + virsh qemu-monitor-command --hmp "device-del " + + For example, to hot unplug the matrix mediated device identified on the + qemu command line with 'id=hostdev0' from the guest named 'my-guest': + + virsh qemu-monitor-command my-guest --hmp "device_del hostdev0" + +2. A matrix mediated device can be hot unplugged by attaching the qemu monitor + to the guest and using the following qemu monitor command: + + (QEMU) device-del id= + + For example, to hot unplug the matrix mediated device that was specified + on the qemu command line with 'id=hostdev0' when the guest was started: + + (QEMU) device-del id=hostdev0 + +After live migration of the KVM guest completes, an AP configuration can be +restored to the KVM guest by hot plugging a matrix mediated device on the target +system into the guest in one of two ways: + +1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated + device into the guest via the following virsh commands: + + virsh attach-device + + For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into + the guest named 'my-guest': + + virsh attach-device my-guest ~/config/my-guest-hostdev.xml + + The contents of my-guest-hostdev.xml: + + + +
+ + + + + virsh qemu-monitor-command --hmp \ + "device_add vfio-ap,sysfsdev=,id=" + + For example, to hot plug the matrix mediated device + 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with + device-id hostdev0: + + virsh qemu-monitor-command my-guest --hmp \ + "device_add vfio-ap,\ + sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ + id=hostdev0" + +2. A matrix mediated device can be hot plugged by attaching the qemu monitor + to the guest and using the following qemu monitor command: + + (qemu) device_add "vfio-ap,sysfsdev=,id=" -* Dynamically modifying the AP matrix for a running guest (which would amount to - hot(un)plug of AP devices for the guest) is currently not supported + For example, to plug the matrix mediated device + 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id + hostdev0: -* Live guest migration is not supported for guests using AP devices. + (QEMU) device-add "vfio-ap,\ + sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\ + id=hostdev0"