From patchwork Thu Jun 13 19:39:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993585 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AAF314E5 for ; Thu, 13 Jun 2019 19:40:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05CE926255 for ; Thu, 13 Jun 2019 19:40:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E8599262AE; Thu, 13 Jun 2019 19:40:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69FFF26255 for ; Thu, 13 Jun 2019 19:40:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728992AbfFMTkD (ORCPT ); Thu, 13 Jun 2019 15:40:03 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:48308 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727510AbfFMTkD (ORCPT ); Thu, 13 Jun 2019 15:40:03 -0400 Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJab4x001597 for ; Thu, 13 Jun 2019 15:40:01 -0400 Received: from e34.co.us.ibm.com (e34.co.us.ibm.com [32.97.110.152]) by mx0b-001b2d01.pphosted.com with ESMTP id 2t3tpfw6xa-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 13 Jun 2019 15:40:01 -0400 Received: from localhost by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 13 Jun 2019 20:40:00 +0100 Received: from b03cxnp08026.gho.boulder.ibm.com (9.17.130.18) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 13 Jun 2019 20:39:57 +0100 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJdrsM36307454 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:39:53 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B93126E050; Thu, 13 Jun 2019 19:39:53 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A09D56E04C; Thu, 13 Jun 2019 19:39:51 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:39:51 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 1/7] s390: vfio-ap: Refactor vfio_ap driver probe and remove callbacks Date: Thu, 13 Jun 2019 15:39:34 -0400 X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 19061319-0016-0000-0000-000009C23B95 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00011256; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000286; SDB=6.01217510; UDB=6.00640242; IPR=6.00998621; MB=3.00027298; MTD=3.00000008; XFM=3.00000015; UTC=2019-06-13 19:40:00 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19061319-0017-0000-0000-000043A442E7 Message-Id: <1560454780-20359-2-git-send-email-akrowiak@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In order to limit the number of private mdev functions called from the vfio_ap device driver as well as to provide a landing spot for dynamic configuration code related to binding/unbinding AP queue devices to/from the vfio_ap driver, the following changes are being introduced: * Move code from the vfio_ap driver's probe callback into a function defined in the mdev private operations file. * Move code from the vfio_ap driver's remove callback into a function defined in the mdev private operations file. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 27 ++++++++++----------------- drivers/s390/crypto/vfio_ap_ops.c | 28 ++++++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 6 +++--- 3 files changed, 41 insertions(+), 20 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 003662aa8060..3c60df70891b 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -49,15 +49,15 @@ MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids); */ static int vfio_ap_queue_dev_probe(struct ap_device *apdev) { - struct vfio_ap_queue *q; - - q = kzalloc(sizeof(*q), GFP_KERNEL); - if (!q) - return -ENOMEM; - dev_set_drvdata(&apdev->device, q); - q->apqn = to_ap_queue(&apdev->device)->qid; - q->saved_isc = VFIO_AP_ISC_INVALID; + int ret; + struct ap_queue *queue = to_ap_queue(&apdev->device); + + ret = vfio_ap_mdev_probe_queue(queue); + if (ret) + return ret; + return 0; + } /** @@ -68,17 +68,10 @@ static int vfio_ap_queue_dev_probe(struct ap_device *apdev) */ static void vfio_ap_queue_dev_remove(struct ap_device *apdev) { - struct vfio_ap_queue *q; - int apid, apqi; + struct ap_queue *queue = to_ap_queue(&apdev->device); mutex_lock(&matrix_dev->lock); - q = dev_get_drvdata(&apdev->device); - dev_set_drvdata(&apdev->device, NULL); - apid = AP_QID_CARD(q->apqn); - apqi = AP_QID_QUEUE(q->apqn); - vfio_ap_mdev_reset_queue(apid, apqi, 1); - vfio_ap_irq_disable(q); - kfree(q); + vfio_ap_mdev_remove_queue(queue); mutex_unlock(&matrix_dev->lock); } diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 015174ff6f0a..bf2ab02b9a0b 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1302,3 +1302,31 @@ void vfio_ap_mdev_unregister(void) { mdev_unregister_device(&matrix_dev->device); } + +int vfio_ap_mdev_probe_queue(struct ap_queue *queue) +{ + struct vfio_ap_queue *q; + + q = kzalloc(sizeof(*q), GFP_KERNEL); + if (!q) + return -ENOMEM; + dev_set_drvdata(&queue->ap_dev.device, q); + q->apqn = queue->qid; + q->saved_isc = VFIO_AP_ISC_INVALID; + + return 0; +} + +void vfio_ap_mdev_remove_queue(struct ap_queue *queue) +{ + struct vfio_ap_queue *q; + int apid, apqi; + + q = dev_get_drvdata(&queue->ap_dev.device); + dev_set_drvdata(&queue->ap_dev.device, NULL); + apid = AP_QID_CARD(q->apqn); + apqi = AP_QID_QUEUE(q->apqn); + vfio_ap_mdev_reset_queue(apid, apqi, 1); + vfio_ap_irq_disable(q); + kfree(q); +} diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vfio_ap_private.h index f46dde56b464..5cc3c2ebf151 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -90,8 +90,6 @@ struct ap_matrix_mdev { extern int vfio_ap_mdev_register(void); extern void vfio_ap_mdev_unregister(void); -int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, - unsigned int retry); struct vfio_ap_queue { struct ap_matrix_mdev *matrix_mdev; @@ -100,5 +98,7 @@ struct vfio_ap_queue { #define VFIO_AP_ISC_INVALID 0xff unsigned char saved_isc; }; -struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q); +int vfio_ap_mdev_probe_queue(struct ap_queue *queue); +void vfio_ap_mdev_remove_queue(struct ap_queue *queue); + #endif /* _VFIO_AP_PRIVATE_H_ */ From patchwork Thu Jun 13 19:39:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993591 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0A6113AF for ; Thu, 13 Jun 2019 19:40:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2C4A205FB for ; Thu, 13 Jun 2019 19:40:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B632C212DB; Thu, 13 Jun 2019 19:40:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 557BC205FB for ; Thu, 13 Jun 2019 19:40:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729383AbfFMTkW (ORCPT ); Thu, 13 Jun 2019 15:40:22 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:42022 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729307AbfFMTkW (ORCPT ); Thu, 13 Jun 2019 15:40:22 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJaave014366; Thu, 13 Jun 2019 15:40:01 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 2t3v8g1ffq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 15:40:01 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id x5DJbOH0029513; Thu, 13 Jun 2019 19:40:00 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma04dal.us.ibm.com with ESMTP id 2t1xj30b4j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 19:40:00 +0000 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJduTa21102880 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:39:56 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ECEDF6E050; Thu, 13 Jun 2019 19:39:55 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 03C186E053; Thu, 13 Jun 2019 19:39:54 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:39:53 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 2/7] s390: vfio-ap: wait for queue empty on queue reset Date: Thu, 13 Jun 2019 15:39:35 -0400 Message-Id: <1560454780-20359-3-git-send-email-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Refactors the AP queue reset function to wait until the queue is empty after the PQAP(ZAPQ) instruction is executed to zero out the queue as required by the AP architecture. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 49 +++++++++++++++++++++++++++------------ 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index bf2ab02b9a0b..60efd3d7896d 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -1128,23 +1128,46 @@ static void vfio_ap_irq_disable_apqn(int apqn) } } -int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi, - unsigned int retry) +static void vfio_ap_mdev_wait_for_qempty(ap_qid_t qid) +{ + struct ap_queue_status status; + int retry = 5; + + do { + status = ap_tapq(qid, NULL); + switch (status.response_code) { + case AP_RESPONSE_NORMAL: + if (status.queue_empty) + return; + case AP_RESPONSE_RESET_IN_PROGRESS: + case AP_RESPONSE_BUSY: + msleep(20); + break; + default: + pr_warn("%s: tapq response %02x waiting for queue %04x.%02x empty\n", + __func__, status.response_code, + AP_QID_CARD(qid), AP_QID_QUEUE(qid)); + return; + } + } while (--retry); + + WARN_ON_ONCE(retry <= 0); +} + +int vfio_ap_mdev_reset_queue(unsigned int apid, unsigned int apqi) { struct ap_queue_status status; - int retry2 = 2; int apqn = AP_MKQID(apid, apqi); + int retry = 5; do { status = ap_zapq(apqn); switch (status.response_code) { case AP_RESPONSE_NORMAL: - while (!status.queue_empty && retry2--) { - msleep(20); - status = ap_tapq(apqn, NULL); - } - WARN_ON_ONCE(retry <= 0); + vfio_ap_mdev_wait_for_qempty(AP_MKQID(apid, apqi)); return 0; + case AP_RESPONSE_DECONFIGURED: + return -ENODEV; case AP_RESPONSE_RESET_IN_PROGRESS: case AP_RESPONSE_BUSY: msleep(20); @@ -1169,14 +1192,10 @@ static int vfio_ap_mdev_reset_queues(struct mdev_device *mdev) matrix_mdev->matrix.apm_max + 1) { for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, matrix_mdev->matrix.aqm_max + 1) { - ret = vfio_ap_mdev_reset_queue(apid, apqi, 1); - /* - * Regardless whether a queue turns out to be busy, or - * is not operational, we need to continue resetting - * the remaining queues. - */ + ret = vfio_ap_mdev_reset_queue(apid, apqi); if (ret) rc = ret; + vfio_ap_irq_disable_apqn(AP_MKQID(apid, apqi)); } } @@ -1326,7 +1345,7 @@ void vfio_ap_mdev_remove_queue(struct ap_queue *queue) dev_set_drvdata(&queue->ap_dev.device, NULL); apid = AP_QID_CARD(q->apqn); apqi = AP_QID_QUEUE(q->apqn); - vfio_ap_mdev_reset_queue(apid, apqi, 1); + vfio_ap_mdev_reset_queue(apid, apqi); vfio_ap_irq_disable(q); kfree(q); } From patchwork Thu Jun 13 19:39:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993597 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2987C13AF for ; Thu, 13 Jun 2019 19:40:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 195FD205FB for ; Thu, 13 Jun 2019 19:40:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C3FF212DB; Thu, 13 Jun 2019 19:40:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63936205FB for ; Thu, 13 Jun 2019 19:40:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729308AbfFMTki (ORCPT ); Thu, 13 Jun 2019 15:40:38 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:47730 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729444AbfFMTkY (ORCPT ); Thu, 13 Jun 2019 15:40:24 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJbSso071043; Thu, 13 Jun 2019 15:40:04 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 2t3s629k6e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 15:40:04 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id x5DJEReH014448; Thu, 13 Jun 2019 19:16:55 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma01dal.us.ibm.com with ESMTP id 2t1x6t0c7q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 19:16:55 +0000 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJdweu28770604 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:39:58 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 36CCF6E04E; Thu, 13 Jun 2019 19:39:58 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 32B196E050; Thu, 13 Jun 2019 19:39:56 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:39:56 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 3/7] s390: zcrypt: driver callback to indicate resource in use Date: Thu, 13 Jun 2019 15:39:36 -0400 Message-Id: <1560454780-20359-4-git-send-email-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduces a new driver callback to prevent a root user from unbinding an AP queue from its device driver if the queue is in use. This prevents a root user from inadvertently taking a queue away from a guest and giving it to the host, or vice versa. The callback will be invoked whenever a change to the AP bus's apmask or aqmask sysfs interfaces may result in one or more AP queues being removed from its driver. If the callback responds in the affirmative for any driver queried, the change to the apmask or aqmask will be rejected with a device in use error. For this patch, only non-default drivers will be queried. Currently, there is only one non-default driver, the vfio_ap device driver. The vfio_ap device driver manages AP queues passed through to one or more guests and we don't want to unexpectedly take AP resources away from guests which are most likely independently administered. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/ap_bus.c | 138 +++++++++++++++++++++++++++++++++++++++++-- drivers/s390/crypto/ap_bus.h | 3 + 2 files changed, 135 insertions(+), 6 deletions(-) diff --git a/drivers/s390/crypto/ap_bus.c b/drivers/s390/crypto/ap_bus.c index b9fc502c58c2..1b06f47d0d1c 100644 --- a/drivers/s390/crypto/ap_bus.c +++ b/drivers/s390/crypto/ap_bus.c @@ -35,6 +35,7 @@ #include #include #include +#include #include "ap_bus.h" #include "ap_debug.h" @@ -998,9 +999,11 @@ int ap_parse_mask_str(const char *str, newmap = kmalloc(size, GFP_KERNEL); if (!newmap) return -ENOMEM; - if (mutex_lock_interruptible(lock)) { - kfree(newmap); - return -ERESTARTSYS; + if (lock) { + if (mutex_lock_interruptible(lock)) { + kfree(newmap); + return -ERESTARTSYS; + } } if (*str == '+' || *str == '-') { @@ -1012,7 +1015,10 @@ int ap_parse_mask_str(const char *str, } if (rc == 0) memcpy(bitmap, newmap, size); - mutex_unlock(lock); + + if (lock) + mutex_unlock(lock); + kfree(newmap); return rc; } @@ -1199,12 +1205,72 @@ static ssize_t apmask_show(struct bus_type *bus, char *buf) return rc; } +int __verify_card_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newapm = (unsigned long *)data; + + /* + * If the reserved bits do not identify cards reserved for use by the + * non-default driver, there is no need to verify the driver is using + * the queues. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(newapm, ap_perms.aqm)) + rc = -EADDRINUSE; + + module_put(drv->owner); + + return rc; +} + +static int apmask_commit(unsigned long *newapm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DEVICES)]; + + /* + * Check if any bits in the apmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newapm, ap_perms.apm, AP_DEVICES)) { + rc = (bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_card_reservations)); + if (rc) + return rc; + } + + memcpy(ap_perms.apm, newapm, APMASKSIZE); + + return 0; +} + static ssize_t apmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + unsigned long newapm[BITS_TO_LONGS(AP_DEVICES)]; + + memcpy(newapm, ap_perms.apm, APMASKSIZE); + + rc = ap_parse_mask_str(buf, newapm, AP_DEVICES, NULL); + if (rc) + return rc; + + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + rc = apmask_commit(newapm); + mutex_unlock(&ap_perms_mutex); - rc = ap_parse_mask_str(buf, ap_perms.apm, AP_DEVICES, &ap_perms_mutex); if (rc) return rc; @@ -1230,12 +1296,72 @@ static ssize_t aqmask_show(struct bus_type *bus, char *buf) return rc; } +int __verify_queue_reservations(struct device_driver *drv, void *data) +{ + int rc = 0; + struct ap_driver *ap_drv = to_ap_drv(drv); + unsigned long *newaqm = (unsigned long *)data; + + /* + * If the reserved bits do not identify queues reserved for use by the + * non-default driver, there is no need to verify the driver is using + * the queues. + */ + if (ap_drv->flags & AP_DRIVER_FLAG_DEFAULT) + return 0; + + /* The non-default driver's module must be loaded */ + if (!try_module_get(drv->owner)) + return 0; + + if (ap_drv->in_use) + if (ap_drv->in_use(ap_perms.apm, newaqm)) + rc = -EADDRINUSE; + + module_put(drv->owner); + + return rc; +} + +static int aqmask_commit(unsigned long *newaqm) +{ + int rc; + unsigned long reserved[BITS_TO_LONGS(AP_DOMAINS)]; + + /* + * Check if any bits in the aqmask have been set which will + * result in queues being removed from non-default drivers + */ + if (bitmap_andnot(reserved, newaqm, ap_perms.aqm, AP_DOMAINS)) { + rc = (bus_for_each_drv(&ap_bus_type, NULL, reserved, + __verify_queue_reservations)); + if (rc) + return rc; + } + + memcpy(ap_perms.aqm, newaqm, APMASKSIZE); + + return 0; +} + static ssize_t aqmask_store(struct bus_type *bus, const char *buf, size_t count) { int rc; + unsigned long newaqm[BITS_TO_LONGS(AP_DEVICES)]; + + memcpy(newaqm, ap_perms.aqm, AQMASKSIZE); + + rc = ap_parse_mask_str(buf, newaqm, AP_DOMAINS, NULL); + if (rc) + return rc; + + if (mutex_lock_interruptible(&ap_perms_mutex)) + return -ERESTARTSYS; + + rc = aqmask_commit(newaqm); + mutex_unlock(&ap_perms_mutex); - rc = ap_parse_mask_str(buf, ap_perms.aqm, AP_DOMAINS, &ap_perms_mutex); if (rc) return rc; diff --git a/drivers/s390/crypto/ap_bus.h b/drivers/s390/crypto/ap_bus.h index 6f3cf37776ca..0f865c5545f2 100644 --- a/drivers/s390/crypto/ap_bus.h +++ b/drivers/s390/crypto/ap_bus.h @@ -137,6 +137,7 @@ struct ap_driver { void (*remove)(struct ap_device *); void (*suspend)(struct ap_device *); void (*resume)(struct ap_device *); + bool (*in_use)(unsigned long *apm, unsigned long *aqm); }; #define to_ap_drv(x) container_of((x), struct ap_driver, driver) @@ -265,6 +266,8 @@ void ap_queue_reinit_state(struct ap_queue *aq); struct ap_card *ap_card_create(int id, int queue_depth, int raw_device_type, int comp_device_type, unsigned int functions); +#define APMASKSIZE (BITS_TO_LONGS(AP_DEVICES) * sizeof(unsigned long)) +#define AQMASKSIZE (BITS_TO_LONGS(AP_DOMAINS) * sizeof(unsigned long)) struct ap_perms { unsigned long ioctlm[BITS_TO_LONGS(AP_IOCTLS)]; unsigned long apm[BITS_TO_LONGS(AP_DEVICES)]; From patchwork Thu Jun 13 19:39:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD48614E5 for ; Thu, 13 Jun 2019 19:40:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ADAD826255 for ; Thu, 13 Jun 2019 19:40:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A0B3F262AE; Thu, 13 Jun 2019 19:40:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CE9126255 for ; Thu, 13 Jun 2019 19:40:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729194AbfFMTkL (ORCPT ); Thu, 13 Jun 2019 15:40:11 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:39160 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729056AbfFMTkL (ORCPT ); Thu, 13 Jun 2019 15:40:11 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJadEL005798 for ; Thu, 13 Jun 2019 15:40:09 -0400 Received: from e31.co.us.ibm.com (e31.co.us.ibm.com [32.97.110.149]) by mx0a-001b2d01.pphosted.com with ESMTP id 2t3sdq8p3w-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 13 Jun 2019 15:40:09 -0400 Received: from localhost by e31.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 13 Jun 2019 20:40:08 +0100 Received: from b03cxnp08027.gho.boulder.ibm.com (9.17.130.19) by e31.co.us.ibm.com (192.168.1.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 13 Jun 2019 20:40:04 +0100 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJe0TI37093818 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:40:00 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7EF846E050; Thu, 13 Jun 2019 19:40:00 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7F3306E04E; Thu, 13 Jun 2019 19:39:58 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:39:58 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 4/7] s390: vfio-ap: implement in-use callback for vfio_ap driver Date: Thu, 13 Jun 2019 15:39:37 -0400 X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 19061319-8235-0000-0000-00000EA7837F X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00011256; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000286; SDB=6.01217510; UDB=6.00640241; IPR=6.00998621; MB=3.00027298; MTD=3.00000008; XFM=3.00000015; UTC=2019-06-13 19:40:07 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19061319-8236-0000-0000-000046025177 Message-Id: <1560454780-20359-5-git-send-email-akrowiak@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Let's implement the callback to indicate when an APQN is in use by the vfio_ap device driver. The callback is invoked whenever a change to the apmask or aqmask may result in one or APQNs being removed from the driver. The vfio_ap device driver will indicate a resource is in use if any of the removed APQNs are assigned to any of the matrix mdev devices. To ensure that the AP bus apmask/aqmask interfaces are used to control which AP queues get manually bound to or unbound from the vfio_ap device driver, the bind/unbind sysfs interfaces will be disabled for the vfio_ap device driver. The reasons for this are: * To prevent unbinding an AP queue device from the vfio_ap device driver representing a queue that is assigned to an mdev device. * To enforce the policy that the the AP resources must first be unassigned from the mdev device - which will hot unplug them from a guest using the mdev device - before changing ownership of APQNs from the vfio_ap driver to a zcrypt driver. This ensures that private crypto data intended for the guest will never be accessible from the host. * It takes advantage of the AP architecture to prevent dynamic changes to the LPAR configuration using the SE or SCLP commands from compromising the guest crypto devices. For example: * Even if an adapter is configured off, if and when it is configured back on, the queue devices associated with the adapter will be bound back to the vfio_ap driver and the queues will automatically be available to a guest using the mdev to which the APQN of the queue device is assigned. * If adapters or domains are dynamically unassigned from the LPAR in which the linux guest is running, effective masking will prevent access to the AP resources by a guest using them. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_ap_drv.c index 3c60df70891b..7b52393007c6 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -164,6 +164,28 @@ static void vfio_ap_matrix_dev_destroy(void) root_device_unregister(root_device); } +static bool vfio_ap_resource_in_use(unsigned long *apm, unsigned long *aqm) +{ + bool in_use = false; + struct ap_matrix_mdev *matrix_mdev; + + mutex_lock(&matrix_dev->lock); + + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + if (bitmap_intersects(matrix_mdev->matrix.apm, + apm, AP_DEVICES) && + bitmap_intersects(matrix_mdev->matrix.aqm, + aqm, AP_DOMAINS)) { + in_use = true; + break; + } + } + + mutex_unlock(&matrix_dev->lock); + + return in_use; +} + static int __init vfio_ap_init(void) { int ret; @@ -179,7 +201,9 @@ static int __init vfio_ap_init(void) memset(&vfio_ap_drv, 0, sizeof(vfio_ap_drv)); vfio_ap_drv.probe = vfio_ap_queue_dev_probe; vfio_ap_drv.remove = vfio_ap_queue_dev_remove; + vfio_ap_drv.in_use = vfio_ap_resource_in_use; vfio_ap_drv.ids = ap_queue_ids; + vfio_ap_drv.driver.suppress_bind_attrs = true; ret = ap_driver_register(&vfio_ap_drv, THIS_MODULE, VFIO_AP_DRV_NAME); if (ret) { From patchwork Thu Jun 13 19:39:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993599 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A960114E5 for ; Thu, 13 Jun 2019 19:40:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99167205FB for ; Thu, 13 Jun 2019 19:40:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8CA0F212DB; Thu, 13 Jun 2019 19:40:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B4B42205FB for ; Thu, 13 Jun 2019 19:40:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729337AbfFMTkW (ORCPT ); Thu, 13 Jun 2019 15:40:22 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58784 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729310AbfFMTkV (ORCPT ); Thu, 13 Jun 2019 15:40:21 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJajcY079455; Thu, 13 Jun 2019 15:40:10 -0400 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 2t3v7thkba-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 15:40:10 -0400 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id x5DJEUN9014463; Thu, 13 Jun 2019 19:17:01 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma01dal.us.ibm.com with ESMTP id 2t1x6t0c8a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 19:17:01 +0000 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJe3v229819214 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:40:03 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 599966E054; Thu, 13 Jun 2019 19:40:03 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B78A26E04C; Thu, 13 Jun 2019 19:40:00 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:40:00 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 5/7] s390: vfio-ap: allow assignment of unavailable AP resources to mdev device Date: Thu, 13 Jun 2019 15:39:38 -0400 Message-Id: <1560454780-20359-6-git-send-email-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The AP architecture does not preclude assignment of AP resources that are not available. Let's go ahead and implement this facet of the AP architecture for linux guests. The current implementation does not allow assignment of AP adapters or domains to an mdev device if the APQNs resulting from the assignment reference AP queue devices that are not bound to the vfio_ap device driver. This patch allows assignment of AP resources to the mdev device as long as the APQNs resulting from the assignment are not reserved by the AP BUS for use by the zcrypt device drivers and the APQNs are not assigned to another mdev device. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 231 ++++++++------------------------------ 1 file changed, 44 insertions(+), 187 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 60efd3d7896d..9db86c0db52e 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -406,122 +406,6 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = { NULL, }; -struct vfio_ap_queue_reserved { - unsigned long *apid; - unsigned long *apqi; - bool reserved; -}; - -/** - * vfio_ap_has_queue - * - * @dev: an AP queue device - * @data: a struct vfio_ap_queue_reserved reference - * - * Flags whether the AP queue device (@dev) has a queue ID containing the APQN, - * apid or apqi specified in @data: - * - * - If @data contains both an apid and apqi value, then @data will be flagged - * as reserved if the APID and APQI fields for the AP queue device matches - * - * - If @data contains only an apid value, @data will be flagged as - * reserved if the APID field in the AP queue device matches - * - * - If @data contains only an apqi value, @data will be flagged as - * reserved if the APQI field in the AP queue device matches - * - * Returns 0 to indicate the input to function succeeded. Returns -EINVAL if - * @data does not contain either an apid or apqi. - */ -static int vfio_ap_has_queue(struct device *dev, void *data) -{ - struct vfio_ap_queue_reserved *qres = data; - struct ap_queue *ap_queue = to_ap_queue(dev); - ap_qid_t qid; - unsigned long id; - - if (qres->apid && qres->apqi) { - qid = AP_MKQID(*qres->apid, *qres->apqi); - if (qid == ap_queue->qid) - qres->reserved = true; - } else if (qres->apid && !qres->apqi) { - id = AP_QID_CARD(ap_queue->qid); - if (id == *qres->apid) - qres->reserved = true; - } else if (!qres->apid && qres->apqi) { - id = AP_QID_QUEUE(ap_queue->qid); - if (id == *qres->apqi) - qres->reserved = true; - } else { - return -EINVAL; - } - - return 0; -} - -/** - * vfio_ap_verify_queue_reserved - * - * @matrix_dev: a mediated matrix device - * @apid: an AP adapter ID - * @apqi: an AP queue index - * - * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device - * driver according to the following rules: - * - * - If both @apid and @apqi are not NULL, then there must be an AP queue - * device bound to the vfio_ap driver with the APQN identified by @apid and - * @apqi - * - * - If only @apid is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apid - * - * - If only @apqi is not NULL, then there must be an AP queue device bound - * to the vfio_ap driver with an APQN containing @apqi - * - * Returns 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL. - */ -static int vfio_ap_verify_queue_reserved(unsigned long *apid, - unsigned long *apqi) -{ - int ret; - struct vfio_ap_queue_reserved qres; - - qres.apid = apid; - qres.apqi = apqi; - qres.reserved = false; - - ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL, - &qres, vfio_ap_has_queue); - if (ret) - return ret; - - if (qres.reserved) - return 0; - - return -EADDRNOTAVAIL; -} - -static int -vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, - unsigned long apid) -{ - int ret; - unsigned long apqi; - unsigned long nbits = matrix_mdev->matrix.aqm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(&apid, NULL); - - for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - /** * vfio_ap_mdev_verify_no_sharing * @@ -529,18 +413,26 @@ vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev, * and AP queue indexes comprising the AP matrix are not configured for another * mediated device. AP queue sharing is not allowed. * - * @matrix_mdev: the mediated matrix device + * @mdev_apm: the mask identifying the adapters assigned to mdev + * @mdev_apm: the mask identifying the adapters assigned to mdev * * Returns 0 if the APQNs are not shared, otherwise; returns -EADDRINUSE. */ -static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) +static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, + unsigned long *mdev_aqm) { struct ap_matrix_mdev *lstdev; DECLARE_BITMAP(apm, AP_DEVICES); DECLARE_BITMAP(aqm, AP_DOMAINS); list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) { - if (matrix_mdev == lstdev) + /* + * If either of the input masks belongs to the mdev to which an + * AP resource is being assigned, then we don't need to verify + * that mdev's masks. + */ + if ((mdev_apm == lstdev->matrix.apm) || + (mdev_aqm == lstdev->matrix.aqm)) continue; memset(apm, 0, sizeof(apm)); @@ -550,12 +442,10 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) * We work on full longs, as we can only exclude the leftover * bits in non-inverse order. The leftover is all zeros. */ - if (!bitmap_and(apm, matrix_mdev->matrix.apm, - lstdev->matrix.apm, AP_DEVICES)) + if (!bitmap_and(apm, mdev_apm, lstdev->matrix.apm, AP_DEVICES)) continue; - if (!bitmap_and(aqm, matrix_mdev->matrix.aqm, - lstdev->matrix.aqm, AP_DOMAINS)) + if (!bitmap_and(aqm, mdev_aqm, lstdev->matrix.aqm, AP_DOMAINS)) continue; return -EADDRINUSE; @@ -564,6 +454,17 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev) return 0; } +static int vfio_ap_mdev_validate_masks(unsigned long *apm, unsigned long *aqm) +{ + int ret; + + ret = ap_apqn_in_matrix_owned_by_def_drv(apm, aqm); + if (ret) + return (ret == 1) ? -EADDRNOTAVAIL : ret; + + return vfio_ap_mdev_verify_no_sharing(apm, aqm); +} + /** * assign_adapter_store * @@ -602,6 +503,7 @@ static ssize_t assign_adapter_store(struct device *dev, { int ret; unsigned long apid; + DECLARE_BITMAP(apm, AP_DEVICES); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); @@ -616,32 +518,19 @@ static ssize_t assign_adapter_store(struct device *dev, if (apid > matrix_mdev->matrix.apm_max) return -ENODEV; - /* - * Set the bit in the AP mask (APM) corresponding to the AP adapter - * number (APID). The bits in the mask, from most significant to least - * significant bit, correspond to APIDs 0-255. - */ - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid); - if (ret) - goto done; + memset(apm, 0, ARRAY_SIZE(apm) * sizeof(*apm)); + set_bit_inv(apid, apm); + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(apm, matrix_mdev->matrix.aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apid, matrix_mdev->matrix.apm); - - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); - if (ret) - goto share_err; - - ret = count; - goto done; - -share_err: - clear_bit_inv(apid, matrix_mdev->matrix.apm); -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_adapter); @@ -690,26 +579,6 @@ static ssize_t unassign_adapter_store(struct device *dev, } static DEVICE_ATTR_WO(unassign_adapter); -static int -vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev, - unsigned long apqi) -{ - int ret; - unsigned long apid; - unsigned long nbits = matrix_mdev->matrix.apm_max + 1; - - if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits) - return vfio_ap_verify_queue_reserved(NULL, &apqi); - - for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) { - ret = vfio_ap_verify_queue_reserved(&apid, &apqi); - if (ret) - return ret; - } - - return 0; -} - /** * assign_domain_store * @@ -748,6 +617,7 @@ static ssize_t assign_domain_store(struct device *dev, { int ret; unsigned long apqi; + DECLARE_BITMAP(aqm, AP_DEVICES); struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; @@ -762,27 +632,19 @@ static ssize_t assign_domain_store(struct device *dev, if (apqi > max_apqi) return -ENODEV; - mutex_lock(&matrix_dev->lock); - - ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi); - if (ret) - goto done; + memset(aqm, 0, ARRAY_SIZE(aqm) * sizeof(*aqm)); + set_bit_inv(apqi, aqm); + mutex_lock(&matrix_dev->lock); + ret = vfio_ap_mdev_validate_masks(matrix_mdev->matrix.apm, aqm); + if (ret) { + mutex_unlock(&matrix_dev->lock); + return ret; + } set_bit_inv(apqi, matrix_mdev->matrix.aqm); - - ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev); - if (ret) - goto share_err; - - ret = count; - goto done; - -share_err: - clear_bit_inv(apqi, matrix_mdev->matrix.aqm); -done: mutex_unlock(&matrix_dev->lock); - return ret; + return count; } static DEVICE_ATTR_WO(assign_domain); @@ -868,11 +730,6 @@ static ssize_t assign_control_domain_store(struct device *dev, if (id > matrix_mdev->matrix.adm_max) return -ENODEV; - /* Set the bit in the ADM (bitmask) corresponding to the AP control - * domain number (id). The bits in the mask, from most significant to - * least significant, correspond to IDs 0 up to the one less than the - * number of control domains that can be assigned. - */ mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); mutex_unlock(&matrix_dev->lock); From patchwork Thu Jun 13 19:39:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993589 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E8F113AF for ; Thu, 13 Jun 2019 19:40:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E28B26255 for ; Thu, 13 Jun 2019 19:40:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 50006262FF; Thu, 13 Jun 2019 19:40:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80C9826255 for ; Thu, 13 Jun 2019 19:40:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729195AbfFMTkP (ORCPT ); Thu, 13 Jun 2019 15:40:15 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:46930 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729224AbfFMTkP (ORCPT ); Thu, 13 Jun 2019 15:40:15 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJbSsu071043 for ; Thu, 13 Jun 2019 15:40:14 -0400 Received: from e35.co.us.ibm.com (e35.co.us.ibm.com [32.97.110.153]) by mx0a-001b2d01.pphosted.com with ESMTP id 2t3s629kef-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 13 Jun 2019 15:40:13 -0400 Received: from localhost by e35.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 13 Jun 2019 20:40:13 +0100 Received: from b03cxnp08027.gho.boulder.ibm.com (9.17.130.19) by e35.co.us.ibm.com (192.168.1.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 13 Jun 2019 20:40:10 +0100 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp08027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJe6jE24052060 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:40:06 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7841A6E056; Thu, 13 Jun 2019 19:40:06 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D16D16E058; Thu, 13 Jun 2019 19:40:03 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:40:03 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 6/7] s390: vfio-ap: allow hot plug/unplug of AP resources using mdev device Date: Thu, 13 Jun 2019 15:39:39 -0400 X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 19061319-0012-0000-0000-00001744096D X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00011256; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000286; SDB=6.01217510; UDB=6.00640241; IPR=6.00998621; MB=3.00027298; MTD=3.00000008; XFM=3.00000015; UTC=2019-06-13 19:40:13 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19061319-0013-0000-0000-000057AF4A11 Message-Id: <1560454780-20359-7-git-send-email-akrowiak@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Let's allow AP resources - i.e., adapters, domains and control domains - to be assigned to or unassigned from an AP matrix mdev while it is in use by a guest. If an AP resource is assigned while a guest is using the matrix mdev, the guest's CRYCB will be dynamically updated to grant access to the adapter, domain or control domain being assigned. If an AP resource is unassigned while a guest is using the matrix mdev, the guest's CRYCB will be dynamically updated to take access to the adapter, domain or control domain away from the guest. Signed-off-by: Tony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 68 ++++++++++++++++++++++----------------- 1 file changed, 38 insertions(+), 30 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 9db86c0db52e..57325eb47278 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -465,6 +465,16 @@ static int vfio_ap_mdev_validate_masks(unsigned long *apm, unsigned long *aqm) return vfio_ap_mdev_verify_no_sharing(apm, aqm); } +static void vfio_ap_mdev_update_crycb(struct ap_matrix_mdev *matrix_mdev) +{ + if (matrix_mdev->kvm && matrix_mdev->kvm->arch.crypto.crycbd) { + kvm_arch_crypto_set_masks(matrix_mdev->kvm, + matrix_mdev->matrix.apm, + matrix_mdev->matrix.aqm, + matrix_mdev->matrix.adm); + } +} + /** * assign_adapter_store * @@ -475,7 +485,10 @@ static int vfio_ap_mdev_validate_masks(unsigned long *apm, unsigned long *aqm) * @count: the number of bytes in @buf * * Parses the APID from @buf and sets the corresponding bit in the mediated - * matrix device's APM. + * matrix device's APM. If a guest is using the mediated matrix device and each + * new APQN formed as a result of the assignment identifies an AP queue device + * that is bound to the vfio_ap device driver, the guest will be granted access + * to the adapter with the specified APID. * * Returns the number of bytes processed if the APID is valid; otherwise, * returns one of the following errors: @@ -507,10 +520,6 @@ static ssize_t assign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -527,7 +536,9 @@ static ssize_t assign_adapter_store(struct device *dev, mutex_unlock(&matrix_dev->lock); return ret; } + set_bit_inv(apid, matrix_mdev->matrix.apm); + vfio_ap_mdev_update_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -543,7 +554,9 @@ static DEVICE_ATTR_WO(assign_adapter); * @count: the number of bytes in @buf * * Parses the APID from @buf and clears the corresponding bit in the mediated - * matrix device's APM. + * matrix device's APM. If a guest is using the mediated matrix device and has + * access to the AP adapter with the specified APID, access to the adapter will + * be taken from the guest. * * Returns the number of bytes processed if the APID is valid; otherwise, * returns one of the following errors: @@ -560,10 +573,6 @@ static ssize_t unassign_adapter_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of adapter */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apid); if (ret) return ret; @@ -573,6 +582,7 @@ static ssize_t unassign_adapter_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm); + vfio_ap_mdev_update_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -589,7 +599,10 @@ static DEVICE_ATTR_WO(unassign_adapter); * @count: the number of bytes in @buf * * Parses the APQI from @buf and sets the corresponding bit in the mediated - * matrix device's AQM. + * matrix device's AQM. If a guest is using the mediated matrix device and each + * new APQN formed as a result of the assignment identifies an AP queue device + * that is bound to the vfio_ap device driver, the guest will be given access + * to the AP queue(s) with the specified APQI. * * Returns the number of bytes processed if the APQI is valid; otherwise returns * one of the following errors: @@ -622,10 +635,6 @@ static ssize_t assign_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_apqi = matrix_mdev->matrix.aqm_max; - /* If the guest is running, disallow assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -641,7 +650,9 @@ static ssize_t assign_domain_store(struct device *dev, mutex_unlock(&matrix_dev->lock); return ret; } + set_bit_inv(apqi, matrix_mdev->matrix.aqm); + vfio_ap_mdev_update_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -659,7 +670,9 @@ static DEVICE_ATTR_WO(assign_domain); * @count: the number of bytes in @buf * * Parses the APQI from @buf and clears the corresponding bit in the - * mediated matrix device's AQM. + * mediated matrix device's AQM. If a guest is using the mediated matrix device + * and has access to queue(s) with the specified domain APQI, access to + * the queue(s) will be taken away from the guest. * * Returns the number of bytes processed if the APQI is valid; otherwise, * returns one of the following errors: @@ -675,10 +688,6 @@ static ssize_t unassign_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow un-assignment of domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &apqi); if (ret) return ret; @@ -688,6 +697,7 @@ static ssize_t unassign_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm); + vfio_ap_mdev_update_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -703,7 +713,9 @@ static DEVICE_ATTR_WO(unassign_domain); * @count: the number of bytes in @buf * * Parses the domain ID from @buf and sets the corresponding bit in the mediated - * matrix device's ADM. + * matrix device's ADM. If a guest is using the mediated matrix device and the + * guest does not have access to the control domain with the specified ID, the + * guest will be granted access to it. * * Returns the number of bytes processed if the domain ID is valid; otherwise, * returns one of the following errors: @@ -719,10 +731,6 @@ static ssize_t assign_control_domain_store(struct device *dev, struct mdev_device *mdev = mdev_from_dev(dev); struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); - /* If the guest is running, disallow assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &id); if (ret) return ret; @@ -732,6 +740,7 @@ static ssize_t assign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); set_bit_inv(id, matrix_mdev->matrix.adm); + vfio_ap_mdev_update_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; @@ -747,7 +756,9 @@ static DEVICE_ATTR_WO(assign_control_domain); * @count: the number of bytes in @buf * * Parses the domain ID from @buf and clears the corresponding bit in the - * mediated matrix device's ADM. + * mediated matrix device's ADM. If a guest is using the mediated matrix device + * and has access to control domain with the specified domain ID, access to + * the control domain will be taken from the guest. * * Returns the number of bytes processed if the domain ID is valid; otherwise, * returns one of the following errors: @@ -764,10 +775,6 @@ static ssize_t unassign_control_domain_store(struct device *dev, struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev); unsigned long max_domid = matrix_mdev->matrix.adm_max; - /* If the guest is running, disallow un-assignment of control domain */ - if (matrix_mdev->kvm) - return -EBUSY; - ret = kstrtoul(buf, 0, &domid); if (ret) return ret; @@ -776,6 +783,7 @@ static ssize_t unassign_control_domain_store(struct device *dev, mutex_lock(&matrix_dev->lock); clear_bit_inv(domid, matrix_mdev->matrix.adm); + vfio_ap_mdev_update_crycb(matrix_mdev); mutex_unlock(&matrix_dev->lock); return count; From patchwork Thu Jun 13 19:39:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anthony Krowiak X-Patchwork-Id: 10993595 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E440A14E5 for ; Thu, 13 Jun 2019 19:40:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D27BB205FB for ; Thu, 13 Jun 2019 19:40:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF186212DB; Thu, 13 Jun 2019 19:40:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D2D7205FB for ; Thu, 13 Jun 2019 19:40:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729627AbfFMTka (ORCPT ); Thu, 13 Jun 2019 15:40:30 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:59138 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729501AbfFMTk0 (ORCPT ); Thu, 13 Jun 2019 15:40:26 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x5DJaiIo079248; Thu, 13 Jun 2019 15:40:15 -0400 Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 2t3v7thkes-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 15:40:14 -0400 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id x5DJbOBI022355; Thu, 13 Jun 2019 19:40:14 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma03dal.us.ibm.com with ESMTP id 2t1x6frb84-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 Jun 2019 19:40:13 +0000 Received: from b03ledav001.gho.boulder.ibm.com (b03ledav001.gho.boulder.ibm.com [9.17.130.232]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x5DJe9ko35324396 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 13 Jun 2019 19:40:09 GMT Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2750B6E059; Thu, 13 Jun 2019 19:40:09 +0000 (GMT) Received: from b03ledav001.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DF77B6E04C; Thu, 13 Jun 2019 19:40:06 +0000 (GMT) Received: from akrowiak-ThinkPad-P50.ibm.com (unknown [9.85.158.129]) by b03ledav001.gho.boulder.ibm.com (Postfix) with ESMTPS; Thu, 13 Jun 2019 19:40:06 +0000 (GMT) From: Tony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: freude@linux.ibm.com, borntraeger@de.ibm.com, cohuck@redhat.com, frankja@linux.ibm.com, david@redhat.com, mjrosato@linux.ibm.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, pmorel@linux.ibm.com, pasic@linux.ibm.com, alex.williamson@redhat.com, kwankhede@nvidia.com, Tony Krowiak Subject: [PATCH v4 7/7] s390: vfio-ap: update documentation Date: Thu, 13 Jun 2019 15:39:40 -0400 Message-Id: <1560454780-20359-8-git-send-email-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> References: <1560454780-20359-1-git-send-email-akrowiak@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-06-13_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1906130146 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch updates the vfio-ap documentation to include the information below. Changes made to the mdev matrix assignment interfaces: * Allow assignment of APQNs that are not bound to the vfio-ap device driver as long as they are not owned by a zcrypt driver as identified in the AP bus sysfs apmask and aqmask interfaces. * Allow assignment of an AP resource to a mediated device which is in use by a guest to hot plug an adapter, domain and control domain into a running guest. * Allow unassignment of an AP resource from a mediated device which is in use by a guest to hot unplug an adapter, domain and control domain from a running guest. This patch also: * Clarifies the section on configuring the AP bus's apmask and aqmask. * Adds sections on dynamic configuration using the SE or SCLP command. Signed-off-by: Tony Krowiak --- Documentation/s390/vfio-ap.txt | 292 ++++++++++++++++++++++++++++++----------- 1 file changed, 213 insertions(+), 79 deletions(-) diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt index 65167cfe4485..9372a6570ce1 100644 --- a/Documentation/s390/vfio-ap.txt +++ b/Documentation/s390/vfio-ap.txt @@ -81,10 +81,19 @@ definitions: which the AP command is to be sent for processing. The AP bus will create a sysfs device for each APQN that can be derived from - the cross product of the AP adapter and usage domain numbers detected when the - AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage - domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the - following sysfs entries: + the Cartesian product of the AP adapter and usage domain numbers detected when + the AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and + usage domains 6 and 71 (0x47) are assigned to the LPAR, the Cartesian product + would be defined by the following table: + + 06 71 + +-----------+-----------+ + 04 | (04,06) | (04,47) | + +-----------|-----------+ + 10 | (0a,06) | (0a,47) | + +-----------|-----------+ + + The AP bus will create the following sysfs entries: /sys/devices/ap/card04/04.0006 /sys/devices/ap/card04/04.0047 @@ -146,10 +155,20 @@ If you recall from the description of an AP Queue, AP instructions include an APQN to identify the AP queue to which an AP command-request message is to be sent (NQAP and PQAP instructions), or from which a command-reply message is to be received (DQAP instruction). The validity of an APQN is defined by the matrix -calculated from the APM and AQM; it is the cross product of all assigned adapter -numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1 -and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6), -(2,5) and (2,6) will be valid for the guest. +calculated from the APM and AQM; it is the Cartesian product of all assigned +adapter numbers (APM) with all assigned queue indexes (AQM). For example, if +adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest: + + + 05 06 + +-----------+-----------+ + 01 | (01,05) | (01,06) | + +-----------|-----------+ + 02 | (02,05) | (02,06) | + +-----------|-----------+ + + +APQNs (01,05), (01,06), (02,05) and (02,06) will be valid for the guest. The APQNs can provide secure key functionality - i.e., a private key is stored on the adapter card for each of its domains - so each APQN must be assigned to @@ -349,8 +368,9 @@ matrix device. number of the the usage domain is echoed to the respective attribute file. * matrix: - A read-only file for displaying the APQNs derived from the cross product - of the adapter and domain numbers assigned to the mediated matrix device. + A read-only file for displaying the APQNs derived from the Caresian + product of the adapter and domain numbers assigned to the mediated matrix + device. * assign_control_domain: * unassign_control_domain: Write-only attributes for assigning/unassigning an AP control domain @@ -438,9 +458,10 @@ guest use. Example: ======= Let's now provide an example to illustrate how KVM guests may be given -access to AP facilities. For this example, we will show how to configure -three guests such that executing the lszcrypt command on the guests would -look like this: +access to AP facilities. For this example, we will assume that adapters 4, 5 +and 6 and domains 4, 71 (0x47), 171 (0xab) and 255 (0xff) are assigned to the +LPAR and are online. We will show how to configure three guests such that +executing the lszcrypt command on the guests would look like this: Guest1 ------ @@ -466,7 +487,7 @@ Guest2 CARD.DOMAIN TYPE MODE ------------------------------ 06 CEX5A Accelerator -06.0047 CEX5A Accelerator +06.0047 CEX5A AcceleratorNote that this directory may contain additional bindings depending upon what 06.00ff CEX5A Accelerator These are the steps: @@ -513,35 +534,44 @@ These are the steps: /sys/bus/ap/aqmask The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs - (APID). Each bit in the mask, from left to right (i.e., from most significant - to least significant bit in big endian order), corresponds to an APID from - 0-255. If a bit is set, the APID is marked as usable only by the default AP - queue device drivers; otherwise, the APID is usable by the vfio_ap - device driver. + (APID). Each bit in the mask, from left to right, corresponds to an APID from + 0-255. The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes - (APQI). Each bit in the mask, from left to right (i.e., from most significant - to least significant bit in big endian order), corresponds to an APQI from - 0-255. If a bit is set, the APQI is marked as usable only by the default AP - queue device drivers; otherwise, the APQI is usable by the vfio_ap device - driver. + (APQI). Each bit in the mask, from left to right, corresponds to an APQI from + 0-255. + + The Cartesian product of the APIDs set in the apmask and the APQIs set in + the aqmask identify the APQNs of AP queue devices owned by the zcrypt + device drivers. + + Take, for example, the following masks: - Take, for example, the following mask: + apmask: 0x7000000000000000000000000000000000000000000000000000000000000000 - 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff + aqmask: 0x0180000000000000000000000000000000000000000000000000000000000000 - It indicates: + The bits set in apmask are bits 1, 2 and 3. The bits set in aqmask are bits + 7 and 8. The Cartesian product of the bits set in the two masks is: - 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6 - belong to the vfio_ap device driver's pool. + 07 08 + +-----------+-----------+ + 01 | (01,07) | (01,08) | + +-----------|-----------+ + 02 | (02,07) | (02,08) | + +-----------|-----------+ + 03 | (03,07) | (03,08) | + +-----------|-----------+ - The APQN of each AP queue device assigned to the linux host is checked by the - AP bus against the set of APQNs derived from the cross product of APIDs - and APQIs marked as usable only by the default AP queue device drivers. If a - match is detected, only the default AP queue device drivers will be probed; - otherwise, the vfio_ap device driver will be probed. + The masks indicate that the queues with APQNs (01,07), (01,08), (02,07), + (02,08), (03,07) and (03,08) are owned by the zcrypt drivers. When the AP bus + detects an AP queue device, its APQN is checked against the set of APQNs + derived from the apmask and aqmask. If a match is detected, the zcrypt + device driver registered for the device type of the queue will be probed. If + a match is not detected and the device type of the queue is CEX4 or newer, + the vfio_ap device driver will be probed. - By default, the two masks are set to reserve all APQNs for use by the default + By default, the two masks are set to reserve all APQNs for use by the zcrypt AP queue device drivers. There are two ways the default masks can be changed: 1. The sysfs mask files can be edited by echoing a string into the @@ -554,8 +584,7 @@ These are the steps: 0x4100000000000000000000000000000000000000000000000000000000000000 - Keep in mind that the mask reads from left to right (i.e., most - significant to least significant bit in big endian order), so the mask + Keep in mind that the mask reads from left to right, so the mask above identifies device numbers 1 and 7 (01000001). If the string is longer than the mask, the operation is terminated with @@ -563,7 +592,7 @@ These are the steps: * Individual bits in the mask can be switched on and off by specifying each bit number to be switched in a comma separated list. Each bit - number string must be prepended with a ('+') or minus ('-') to indicate + number string must be prefixed with a ('+') or minus ('-') to indicate the corresponding bit is to be switched on ('+') or off ('-'). Some valid values are: @@ -594,19 +623,26 @@ These are the steps: aqmask: 0x4000000000000000000000000000000000000000000000000000000000000000 - Resulting in these two pools: - - default drivers pool: adapter 0-15, domain 1 - alternate drivers pool: adapter 16-255, domains 0, 2-255 +Recall that the Cartesian product of the APIDs set in the apmask and the APQIs +set in the aqmask identify the APQNs of AP queue devices owned by the zcrypt +device drivers. If an attempt is made to modify the apmask or aqmask such that +one or more APQNs changes ownership from a the vfio_ap device driver to a zcrypt +device driver and the APQN is assigned to a mediated device (see step 3 below), +the operation will fail with an error ('Address already in use'). Securing the APQNs for our example: ---------------------------------- - To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, - 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding - APQNs can either be removed from the default masks: + There is no way to secure the specific AP queues 05.0004, 05.0047, 05.00ab, + 05.00ff, 06.0004, 06.0047, 06.00ab, and 06.00ff for use by the vfio_ap device + driver, so we are left with either securing all queues on adapters 05 and + 06, or queues 0004, 0047, 00ab and 00ff can be secured on all adapters. + + To secure all queues on adapters 05 and 05: echo -5,-6 > /sys/bus/ap/apmask + To secure queues 0004, 0047, 00ab, and 00ff on all adapters: + echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask Or the masks can be set as follows: @@ -617,14 +653,20 @@ These are the steps: echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \ > aqmask - This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, - 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The - sysfs directory for the vfio_ap device driver will now contain symbolic links - to the AP queue devices bound to it: + For this example, we will choose to secure queues 0004, 0047, 00ab, and 00ff + on all adapters. This will result in AP queues 04.0004, 04.0047, 04.00ab, + 04.00ff, 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, 06.00ab and + 06.00ff getting bound to the vfio_ap device driver. The sysfs directory for + the vfio_ap device driver will now contain symbolic links to the AP queue + devices bound to it: /sys/bus/ap ... [drivers] ...... [vfio_ap] + ......... [04.0004] + ......... [04.0047] + ......... [04.00ab] + ......... [04.00ff] ......... [05.0004] ......... [05.0047] ......... [05.00ab] @@ -641,11 +683,12 @@ These are the steps: future and for which there are few older systems on which to test. The administrator, therefore, must take care to secure only AP queues that - can be bound to the vfio_ap device driver. The device type for a given AP - queue device can be read from the parent card's sysfs directory. For example, - to see the hardware type of the queue 05.0004: + can be bound to the vfio_ap device driver, or those queues will not get bound + to any driver. The device type for a given AP queue device can be read from + the parent card's sysfs directory. For example, to see the hardware type of + the queue 05.0004: - cat /sys/bus/ap/devices/card05/hwtype + cat /sys/bus/ap/devices/card05/hwtype The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the vfio_ap device driver. @@ -747,37 +790,48 @@ These are the steps: higher than the maximum is specified, the operation will terminate with an error (ENODEV). - * All APQNs that can be derived from the adapter ID and the IDs of - the previously assigned domains must be bound to the vfio_ap device - driver. If no domains have yet been assigned, then there must be at least - one APQN with the specified APID bound to the vfio_ap driver. If no such - APQNs are bound to the driver, the operation will terminate with an - error (EADDRNOTAVAIL). + * Each APQN that can be derived from the adapter ID and the IDs of + the previously assigned domains must not be reserved for use by the + zcrypt device drivers as specified by the /sys/bus/ap/apmask and + /sys/bus/ap/aqmask syfs interfaces. If any APQN is reserved, the operation + will terminate with an error (EADDRNOTAVAIL). - No APQN that can be derived from the adapter ID and the IDs of the + * No APQN that can be derived from the adapter ID and the IDs of the previously assigned domains can be assigned to another mediated matrix device. If an APQN is assigned to another mediated matrix device, the operation will terminate with an error (EADDRINUSE). + Note that adapters that are currently not available to the host may be + assigned to the mediated device. If the adapter subsequently becomes + available while a guest is using the mediated device, it will automatically + become available to the guest. + In order to successfully assign a domain: * The domain number specified must represent a value from 0 up to the maximum domain number configured for the system. If a domain number higher than the maximum is specified, the operation will terminate with - an error (ENODEV). + an error (ENODEV). The maximum domain number can be determined by + printing the sysfs /sys/bus/ap/ap_max_domain_id attribute: - * All APQNs that can be derived from the domain ID and the IDs of - the previously assigned adapters must be bound to the vfio_ap device - driver. If no domains have yet been assigned, then there must be at least - one APQN with the specified APQI bound to the vfio_ap driver. If no such - APQNs are bound to the driver, the operation will terminate with an - error (EADDRNOTAVAIL). + cat /sys/bus/ap/ap_max_domain_id - No APQN that can be derived from the domain ID and the IDs of the + * Each APQN that can be derived from the domain ID and the IDs of + the previously assigned adapters must not be reserved for use by the + zcrypt device drivers as specified by the /sys/bus/ap/apmask and + /sys/bus/ap/aqmask syfs interfaces. If any APQN is reserved, the operation + will terminate with an error (EADDRNOTAVAIL). + + * No APQN that can be derived from the domain ID and the IDs of the previously assigned adapters can be assigned to another mediated matrix device. If an APQN is assigned to another mediated matrix device, the operation will terminate with an error (EADDRINUSE). + Note that domains that are currently not available to the host may be + assigned to the mediated device. If the domain subsequently becomes + available while a guest is using the mediated device, it will automatically + become available to the guest. + In order to successfully assign a control domain, the domain number specified must represent a value from 0 up to the maximum domain number configured for the system. If a control domain number higher than the maximum @@ -822,16 +876,96 @@ Using our example again, to remove the mediated matrix device $uuid1: host. If the mdev matrix device is removed, one may want to also reconfigure the pool of adapters and queues reserved for use by the default drivers. -Limitations -=========== -* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN - to the default drivers pool of a queue that is still assigned to a mediated - device in use by a guest. It is incumbent upon the administrator to - ensure there is no mediated device in use by a guest to which the APQN is - assigned lest the host be given access to the private data of the AP queue - device such as a private key configured specifically for the guest. +Hot plug/unplug via mdev matrix device sysfs interfaces: +======================================================= +If an mdev matrix device is in use by a running guest, AP resources can be +plugged into or unplugged from the guest via the mdev device's sysfs +assignment interfaces. Below are some examples. + + To plug adapter 10 into a running guest: + + echo 0xa > assign_adapter + + To unplug domain 5 from a running guest: + + echo 5 > unassign_domain + +To display the matrix of a guest using the mdev matrix device: + + cat guest_matrix + +If you attempt to display the guest matrix when a guest is not using the +mdev matrix device, an error will be displayed (ENODEV). Note that adapters and +domains that are not yet available or not yet assigned to the LPAR can be +assigned and will become available to the guest as soon as they become available +to the host. + +Dynamic Changes to AP Configuration using the Support Element (SE): +================================================================== +The SE can be used to dynamically make the following changes to the AP +configuration for an LPAR in which a linux host is running: + + * Configure one or more adapters on + + Configuring an adapter on sets its state to online, thus making it + available to the LPAR to which it is assigned. When an adapter is + configured on, it immediately becomes available to the LPAR as well as to + any guests using a mediated device to which the adapter is assigned. + + * Configure one or more adapters off + + Configuring an adapter off sets its state to standby, thus making it + unavailable to the LPAR to which it is assigned. When an adapter is + configured off, it immediately becomes unavailable to the LPAR as well as + to any guests using a mediated device to which the adapter is assigned. + + * Add adapters or domains to the LPAR configuration + + Adapters and/or domains can be assigned to an LPAR using the Change LPAR + Cryptographic Controls task. To make dynamic changes to the AP + configuration for an LPAR Running a linux guest, the online adapters + assigned to the LPAR must first be configured off. After performing the + adapter and/or domain assignments, the AP resources will automatically + become available to the linux host running in the LPAR as well as any + guests using a mediated device to which the adpaters and/or domains are + assigned. + + * Remove adapters or domains from the LPAR configuration + + Adapters and/or domains can be unassigned from an LPAR using the Change + LPAR Cryptographic Controls task. To make dynamic changes to the AP + configuration for an LPAR Running a linux guest, the online adapters + assigned to the LPAR must first be configured off. After performing the + adapter and/or domain unassignments, the AP resources will automatically + become unavailable to the linux host running in the LPAR as well as any + guests using a mediated device to which the adpaters and/or domains are + assigned. + +Dynamic Changes to the AP Configuration using the SCLP command: +============================================================== +The following SCLP commands may be used to dynamically configure AP adapters on +and off: + +* Configure Adjunct Processor + + The 'Configure Adjunct Processor' command sets an AP adapter's state to + online, thus making it available to the LPARs to which it is assigned. It will + likewise become available to any linux guest using a mediated device to which + the adapter is assigned. + +* Deconfigure Adjunct Processor + + The 'Deconfigure Adjunct Processor' command sets an AP adapter's state to + standby, thus making it unavailable to the LPARs to which it is assigned. It + will likewise become unavailable to any linux guest using a mediated device to + which the adapter is assigned. -* Dynamically modifying the AP matrix for a running guest (which would amount to - hot(un)plug of AP devices for the guest) is currently not supported +Live migration: +============== +Live guest migration is not supported for guests using AP devices. All AP +devices in use by the guest must be unplugged prior to initiating live +migration (see "Hot plug/unplug via mdev matrix device sysfs interfaces" section +above). If you are using QEMU to run your guest and it supports hot plug/unplug +of the vfio-ap device, this would be another option (consult the QEMU +documentation for details). -* Live guest migration is not supported for guests using AP devices.