From patchwork Sun Jul 1 18:22:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dexuan Cui X-Patchwork-Id: 10500081 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1796F60532 for ; Sun, 1 Jul 2018 18:22:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0ADD328747 for ; Sun, 1 Jul 2018 18:22:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F19F628759; Sun, 1 Jul 2018 18:22:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B30D28747 for ; Sun, 1 Jul 2018 18:22:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965067AbeGASWn (ORCPT ); Sun, 1 Jul 2018 14:22:43 -0400 Received: from mail-sg2apc01on0120.outbound.protection.outlook.com ([104.47.125.120]:47168 "EHLO APC01-SG2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S964921AbeGASWk (ORCPT ); Sun, 1 Jul 2018 14:22:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G5QqAO8+Uv7V9FJA3rW4kmmiWL7uraysBqfEuphuOsY=; b=iCU4whqdlQ1JZHSH62SHU3C0TKCg9Rda8+MdN2+7/z4aPrqOwEVGK0rEum7bYsGNc+JThJe6qkJLPymDtYQA2+DHw+sU8dGwmNPI5+bYDhKKqu1QyJ+hVXi3DaQ37BY0UXXp/28QOM1i8HFqL0weY2yYXx2cJlN3LVJ4AWCs/wg= Received: from KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM (10.170.167.17) by KL1P15301MB0038.APCP153.PROD.OUTLOOK.COM (10.170.167.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.952.4; Sun, 1 Jul 2018 18:22:24 +0000 Received: from KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM ([fe80::b9ce:51a5:609c:de90]) by KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM ([fe80::b9ce:51a5:609c:de90%5]) with mapi id 15.20.0952.003; Sun, 1 Jul 2018 18:22:24 +0000 From: Dexuan Cui To: 'Lorenzo Pieralisi' , 'Bjorn Helgaas' , "'linux-pci@vger.kernel.org'" , KY Srinivasan , Stephen Hemminger , "'olaf@aepfle.de'" , "'apw@canonical.com'" , "'jasowang@redhat.com'" CC: "'linux-kernel@vger.kernel.org'" , "'driverdev-devel@linuxdriverproject.org'" , Haiyang Zhang , "'vkuznets@redhat.com'" , "'marcelo.cerri@canonical.com'" Subject: [PATCH v2] PCI: hv: Disable/enable irq rather than bh in hv_compose_msi_msg() Thread-Topic: [PATCH v2] PCI: hv: Disable/enable irq rather than bh in hv_compose_msi_msg() Thread-Index: AdQRZ+8zPuSKPLT9RJW12qWmOo7nFw== Date: Sun, 1 Jul 2018 18:22:23 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=decui@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2018-07-01T18:22:20.3922402Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General authentication-results: spf=none (sender IP is ) smtp.mailfrom=decui@microsoft.com; x-originating-ip: [2601:600:a27f:df20:f40d:b20e:4954:656f] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; KL1P15301MB0038; 7:uYkGXOsMdfNSqBNuPS2TW7Ny0VGA2ZLtaL3b6RN+WCxnbilvclZOqXaU/gLFAbVO6VAkvaeJwaCNWRtTuDNSwgpUGVwOccSOsJ9LoAIvezWAw2bfqjPm5mCEskL02D+dJdbKeBU6L3UMIQl1wVBsLanSt6Wb84OgIfEsGEL9josus2rRLR6QfmUxDwV1u+LPptWVIm1uB4NCwxrhIKW1MtXLcDPoMu4vuOqZKiCCaBCkVSZVbcP8zEI8CDl3aaDj x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: eb7ec830-dada-49f7-44e3-08d5df7f97c7 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989117)(5600053)(711020)(48565401081)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(2017052603328)(7193020); SRVR:KL1P15301MB0038; x-ms-traffictypediagnostic: KL1P15301MB0038: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(89211679590171)(9452136761055); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3002001)(3231254)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(6072148)(201708071742011)(7699016); SRVR:KL1P15301MB0038; BCL:0; PCL:0; RULEID:; SRVR:KL1P15301MB0038; x-forefront-prvs: 07200C0526 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(396003)(376002)(366004)(136003)(39860400002)(346002)(199004)(54534003)(189003)(6116002)(54906003)(186003)(478600001)(74316002)(7696005)(81156014)(86612001)(86362001)(575784001)(305945005)(105586002)(53936002)(10090500001)(46003)(4326008)(8676002)(81166006)(6436002)(102836004)(99286004)(6506007)(7736002)(10290500003)(316002)(110136005)(256004)(14444005)(9686003)(8936002)(22452003)(14454004)(1511001)(486006)(55016002)(476003)(2900100001)(5250100002)(97736004)(25786009)(106356001)(2906002)(8990500004)(5660300001)(33656002)(7416002)(68736007)(491001); DIR:OUT; SFP:1102; SCL:1; SRVR:KL1P15301MB0038; H:KL1P15301MB0006.APCP153.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: qk1BVQwsxAnZkcVq5rSxfj95EsVDrV3o5fKKCFVFmZqG/Hvp/9bWCVxZXiHTm6JXD5GbVcs6ipiXHv1+4OMLm2h5dUvr+m8bPAJ71pBEI2w9wfchJwhFyOxl2DW0BVcd+GFb+39X9RHl768oGxdO23RdDmNI4AtfDpS4C0diV6EjkvctGUEenHpHZMvIrIzPYGJkIuMBa+b0TBKyG17vjWOBhig446dXLap9GCbP6GZQ0H8w/ylxA4b/44n5sG2RRUBjzl26zvZVv4d+uwhE3L6KTwOi9NySJxcbbUG1HOtMrCAPC80hbBfbH975VdmdVqgBzleQVxQJ0Dlo+CHHDMjfBAnMI2Rw+ebBLMy1OQU= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: eb7ec830-dada-49f7-44e3-08d5df7f97c7 X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Jul 2018 18:22:23.8990 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: KL1P15301MB0038 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Commit de0aa7b2f97d ("PCI: hv: Fix 2 hang issues in hv_compose_msi_msg()") uses local_bh_disable()/enable(), because hv_pci_onchannelcallback() can also run in tasklet context as the channel event callback, and here we want to avoid the race. With CONFIG_PROVE_LOCKING=y in the recent mainline, or old kernels that don't have commit f71b74bca637 ("irq/softirqs: Use lockdep to assert IRQs are disabled/enabled"), when the upper layer irq code calls hv_compose_msi_msg() with local irq DISABLED, we'll see a warning at the beginning of __local_bh_enable_ip(): IRQs not enabled as expected WARNING: CPU: 0 PID: 408 at kernel/softirq.c:162 __local_bh_enable_ip The warning exposes an issue in de0aa7b2f97d: local_bh_enable() can potentially call do_softirq(), which is not supposed to run when local irq is DISABLED. Let's fix this by using local_irq_save()/restore() instead. Note: hv_pci_onchannelcallback() is not a hot path because it's only called when the PCI device is hot added and removed, which is infrequent. Fixes: de0aa7b2f97d ("PCI: hv: Fix 2 hang issues in hv_compose_msi_msg()") Signed-off-by: Dexuan Cui Reviewed-by: Haiyang Zhang Cc: Cc: Stephen Hemminger Cc: K. Y. Srinivasan --- A trimmed version of the warning is: IRQs not enabled as expected WARNING: CPU: 0 PID: 408 at kernel/softirq.c:162 __local_bh_enable_ip+0xb0/0xe0 Call Trace: hv_compose_msi_msg+0x209/0x462 [pci_hyperv] irq_chip_compose_msi_msg+0x41/0x50 msi_domain_activate+0x1a/0x40 __irq_domain_activate_irq+0x59/0x90 irq_domain_activate_irq+0x25/0x40 __setup_irq+0x3ec/0x730 request_threaded_irq+0xfa/0x1a0 mlx4_init_eq_table+0x3c3/0x5f0 [mlx4_core] mlx4_setup_hca+0x1db/0x750 [mlx4_core] mlx4_load_one+0xad2/0x13b0 [mlx4_core] mlx4_init_one+0x578/0x710 [mlx4_core] local_pci_probe+0x1e/0x50 work_for_cpu_fn+0x10/0x20 process_one_work+0x1d4/0x5a0 worker_thread+0x1cb/0x3d0 kthread+0xf5/0x130 Changes since v1: Updated the changelog only (fixed typos and some inaccuracy) drivers/pci/controller/pci-hyperv.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c index ba1d4b5..eb20296 100644 --- a/drivers/pci/controller/pci-hyperv.c +++ b/drivers/pci/controller/pci-hyperv.c @@ -1073,6 +1073,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) struct pci_bus *pbus; struct pci_dev *pdev; struct cpumask *dest; + unsigned long flags; struct compose_comp_ctxt comp; struct tran_int_desc *int_desc; struct { @@ -1164,14 +1165,15 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) * the channel callback directly when channel->target_cpu is * the current CPU. When the higher level interrupt code * calls us with interrupt enabled, let's add the - * local_bh_disable()/enable() to avoid race. + * local_irq_save()/restore() to avoid race: + * hv_pci_onchannelcallback() can also run in tasklet. */ - local_bh_disable(); + local_irq_save(flags); if (hbus->hdev->channel->target_cpu == smp_processor_id()) hv_pci_onchannelcallback(hbus); - local_bh_enable(); + local_irq_restore(flags); if (hpdev->state == hv_pcichild_ejecting) { dev_err_once(&hbus->hdev->device,