From patchwork Wed Dec 13 15:32:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Bryant G. Ly" X-Patchwork-Id: 10110267 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9BCB660352 for ; Wed, 13 Dec 2017 15:33:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8F1E728C58 for ; Wed, 13 Dec 2017 15:33:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8403E28C65; Wed, 13 Dec 2017 15:33:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F302528C58 for ; Wed, 13 Dec 2017 15:33:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753414AbdLMPdi (ORCPT ); Wed, 13 Dec 2017 10:33:38 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:53846 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753418AbdLMPda (ORCPT ); Wed, 13 Dec 2017 10:33:30 -0500 Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id vBDFTHis144097 for ; Wed, 13 Dec 2017 10:33:30 -0500 Received: from e12.ny.us.ibm.com (e12.ny.us.ibm.com [129.33.205.202]) by mx0a-001b2d01.pphosted.com with ESMTP id 2eu4a4gaek-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 13 Dec 2017 10:33:29 -0500 Received: from localhost by e12.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 13 Dec 2017 10:33:29 -0500 Received: from b01cxnp22035.gho.pok.ibm.com (9.57.198.25) by e12.ny.us.ibm.com (146.89.104.199) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 13 Dec 2017 10:33:24 -0500 Received: from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com [9.57.199.107]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id vBDFXNbA48365804; Wed, 13 Dec 2017 15:33:23 GMT Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4D1E312403F; Wed, 13 Dec 2017 10:30:22 -0500 (EST) Received: from bryants-mbp-3.rchland.ibm.com (unknown [9.10.79.73]) by b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP id 8E7C2124047; Wed, 13 Dec 2017 10:30:21 -0500 (EST) From: "Bryant G. Ly" To: benh@kernel.crashing.org, paulus@samba.org, mpe@ellerman.id.au Cc: seroyer@linux.vnet.ibm.com, jjalvare@linux.vnet.ibm.com, alex.williamson@redhat.com, helgaas@kernel.org, aik@ozlabs.ru, ruscur@russell.cc, linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, bodong@mellanox.com, eli@mellanox.com, saeedm@mellanox.com, "Bryant G. Ly" Subject: [PATCH v1 4/7] powerpc/kernel Add EEH operations to notify resume Date: Wed, 13 Dec 2017 09:32:39 -0600 X-Mailer: git-send-email 2.14.3 (Apple Git-98) In-Reply-To: <20171213153242.98015-1-bryantly@linux.vnet.ibm.com> References: <20171213153242.98015-1-bryantly@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17121315-0048-0000-0000-00000213B94B X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008199; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000244; SDB=6.00959664; UDB=6.00485359; IPR=6.00739684; BA=6.00005740; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00018523; XFM=3.00000015; UTC=2017-12-13 15:33:27 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17121315-0049-0000-0000-00004372437F Message-Id: <20171213153242.98015-5-bryantly@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-12-13_07:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1712130220 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When pseries SR-IOV is enabled and after a PF driver has resumed from EEH, platform has to be notified of the event so the child VFs can be allowed to resume their normal recovery path. This patch makes the EEH operation allow unfreeze platform dependent code and adds the call to pseries EEH code. Signed-off-by: Bryant G. Ly Signed-off-by: Juan J. Alvarez --- arch/powerpc/include/asm/eeh.h | 1 + arch/powerpc/kernel/eeh_driver.c | 4 ++ arch/powerpc/platforms/powernv/eeh-powernv.c | 3 +- arch/powerpc/platforms/pseries/eeh_pseries.c | 100 ++++++++++++++++++++++++++- 4 files changed, 106 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/include/asm/eeh.h b/arch/powerpc/include/asm/eeh.h index 5161c37dd039..12d52a0cd447 100644 --- a/arch/powerpc/include/asm/eeh.h +++ b/arch/powerpc/include/asm/eeh.h @@ -214,6 +214,7 @@ struct eeh_ops { int (*write_config)(struct pci_dn *pdn, int where, int size, u32 val); int (*next_error)(struct eeh_pe **pe); int (*restore_config)(struct pci_dn *pdn); + int (*notify_resume)(struct pci_dn *pdn); }; extern int eeh_subsystem_flags; diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c index c61bf770282b..dbda0cda559b 100644 --- a/arch/powerpc/kernel/eeh_driver.c +++ b/arch/powerpc/kernel/eeh_driver.c @@ -361,6 +361,7 @@ static void *eeh_report_resume(void *data, void *userdata) bool was_in_error; struct pci_driver *driver; char *envp[] = {"EVENT=EEH_RESUME", "ONLINE=1", NULL}; + struct pci_dn *pdn = eeh_dev_to_pdn(edev); if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe)) return NULL; @@ -384,6 +385,9 @@ static void *eeh_report_resume(void *data, void *userdata) driver->err_handler->resume(dev); eeh_pcid_put(dev); kobject_uevent_env(&dev->dev.kobj, KOBJ_CHANGE, envp); +#ifdef CONFIG_PCI_IOV + eeh_ops->notify_resume(pdn); +#endif return NULL; } diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c index 961e64115d92..8575b3a29e7c 100644 --- a/arch/powerpc/platforms/powernv/eeh-powernv.c +++ b/arch/powerpc/platforms/powernv/eeh-powernv.c @@ -1763,7 +1763,8 @@ static struct eeh_ops pnv_eeh_ops = { .read_config = pnv_eeh_read_config, .write_config = pnv_eeh_write_config, .next_error = pnv_eeh_next_error, - .restore_config = pnv_eeh_restore_config + .restore_config = pnv_eeh_restore_config, + .notify_resume = NULL }; #ifdef CONFIG_PCI_IOV diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c index 5bdd1678a9ff..2b36fbf4ce74 100644 --- a/arch/powerpc/platforms/pseries/eeh_pseries.c +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c @@ -798,6 +798,103 @@ static int pseries_eeh_restore_config(struct pci_dn *pdn) return 0; } +#ifdef CONFIG_PCI_IOV +int pseries_send_allow_unfreeze(struct eeh_pe *pe, + u16 *vf_pe_array, int cur_vfs) +{ + int rc, config_addr; + int ibm_allow_unfreeze = rtas_token("ibm,open-sriov-allow-unfreeze"); + + config_addr = pe->config_addr; + spin_lock(&rtas_data_buf_lock); + memcpy(rtas_data_buf, vf_pe_array, RTAS_DATA_BUF_SIZE); + rc = rtas_call(ibm_allow_unfreeze, 5, 1, NULL, + config_addr, + BUID_HI(pe->phb->buid), + BUID_LO(pe->phb->buid), + rtas_data_buf, cur_vfs * sizeof(u16)); + spin_unlock(&rtas_data_buf_lock); + if (rc) + pr_warn("%s: Failed to allow unfreeze for PHB#%x-PE#%x, rc=%x\n", + __func__, + pe->phb->global_number, + pe->config_addr, rc); + return rc; +} + +static int pseries_call_allow_unfreeze(struct eeh_dev *edev) +{ + struct eeh_pe *pe; + struct pci_dn *pdn, *tmp, *parent, *physfn_pdn; + int cur_vfs, rc, vf_index; + u16 *vf_pe_array; + + vf_pe_array = kzalloc(RTAS_DATA_BUF_SIZE, GFP_KERNEL); + if (!vf_pe_array) + return -ENOMEM; + + memset(vf_pe_array, 0, RTAS_DATA_BUF_SIZE); + cur_vfs = 0; + rc = 0; + if (edev->pdev->is_physfn) { + pe = eeh_dev_to_pe(edev); + cur_vfs = pci_num_vf(edev->pdev); + pdn = eeh_dev_to_pdn(edev); + parent = pdn->parent; + /* For each of its VF + * call allow unfreeze + */ + for (vf_index = 0; vf_index < cur_vfs; vf_index++) + vf_pe_array[vf_index] = + be16_to_cpu(pdn->pe_num_map[vf_index]); + + rc = pseries_send_allow_unfreeze(pe, vf_pe_array, cur_vfs); + pdn->last_allow_rc = rc; + for (vf_index = 0; vf_index < cur_vfs; vf_index++) { + list_for_each_entry_safe(pdn, tmp, &parent->child_list, + list) { + if (pdn->busno + != pci_iov_virtfn_bus(edev->pdev, + vf_index) || + pdn->devfn + != pci_iov_virtfn_devfn(edev->pdev, + vf_index)) + continue; + pdn->last_allow_rc = rc; + } + } + } else { + pdn = pci_get_pdn(edev->pdev); + vf_pe_array[0] = be16_to_cpu(pdn->pe_number); + physfn_pdn = pci_get_pdn(edev->physfn); + edev = pdn_to_eeh_dev(physfn_pdn); + pe = eeh_dev_to_pe(edev); + rc = pseries_send_allow_unfreeze(pe, vf_pe_array, 1); + pdn->last_allow_rc = rc; + } + + kfree(vf_pe_array); + return rc; +} + +static int pseries_notify_resume(struct pci_dn *pdn) +{ + struct eeh_dev *edev = pdn_to_eeh_dev(pdn); + + if (!edev) + return -EEXIST; + + if (rtas_token("ibm,open-sriov-allow-unfreeze") + == RTAS_UNKNOWN_SERVICE) + return -EINVAL; + + if (edev->pdev->is_physfn || edev->pdev->is_virtfn) + return pseries_call_allow_unfreeze(edev); + + return 0; +} +#endif + static struct eeh_ops pseries_eeh_ops = { .name = "pseries", .init = pseries_eeh_init, @@ -813,7 +910,8 @@ static struct eeh_ops pseries_eeh_ops = { .read_config = pseries_eeh_read_config, .write_config = pseries_eeh_write_config, .next_error = NULL, - .restore_config = pseries_eeh_restore_config + .restore_config = pseries_eeh_restore_config, + .notify_resume = pseries_notify_resume }; /**