From patchwork Wed Nov 1 20:30:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dexuan Cui X-Patchwork-Id: 10037051 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5C9DB600C5 for ; Wed, 1 Nov 2017 20:31:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 472D628BD4 for ; Wed, 1 Nov 2017 20:31:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 39DFC28BCC; Wed, 1 Nov 2017 20:31:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9846F28BD4 for ; Wed, 1 Nov 2017 20:31:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755040AbdKAUbN (ORCPT ); Wed, 1 Nov 2017 16:31:13 -0400 Received: from mail-sg2apc01on0111.outbound.protection.outlook.com ([104.47.125.111]:33504 "EHLO APC01-SG2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1755207AbdKAUbJ (ORCPT ); Wed, 1 Nov 2017 16:31:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=KKxiQB17lCejLiNJuniuizSalNc/8uUFLl+j1ouwDV0=; b=A6OxApFGFa0VGLpYpd176++QCBALys2lERhOa90QoxC7hAV96xLCqkMORPr62V2Uw6dM5ZDKq79jaoJa8fl09G1wnBKy5XpfJjh/hwtNpFSi7b9RgavUTqq+CWN8VTDynu+soQpGzSSBTbOytJhdKbUSrEk/cbfw44QHNOhxBCE= Received: from PS1P15301MB0011.APCP153.PROD.OUTLOOK.COM (10.170.183.149) by PS1P15301MB0009.APCP153.PROD.OUTLOOK.COM (10.170.183.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.218.4; Wed, 1 Nov 2017 20:30:53 +0000 Received: from PS1P15301MB0011.APCP153.PROD.OUTLOOK.COM ([fe80::849b:f2e3:7ada:99e2]) by PS1P15301MB0011.APCP153.PROD.OUTLOOK.COM ([fe80::849b:f2e3:7ada:99e2%17]) with mapi id 15.20.0218.004; Wed, 1 Nov 2017 20:30:53 +0000 From: Dexuan Cui To: Bjorn Helgaas , "linux-pci@vger.kernel.org" , Jake Oshins , KY Srinivasan , Stephen Hemminger CC: "devel@linuxdriverproject.org" , "linux-kernel@vger.kernel.org" , Haiyang Zhang , Jork Loeser , "Chris Valean (Cloudbase Solutions SRL)" , "Adrian Suhov (Cloudbase Solutions SRL)" , Simon Xiao , 'Eyal Mizrachi' , "Jack Morgenstein" , Armen Guezalian , Firas Mahameed , Tziporet Koren , Daniel Jurgens Subject: [PATCH] PCI: hv: use effective affinity mask Thread-Topic: [PATCH] PCI: hv: use effective affinity mask Thread-Index: AdNTSQFbm8/mhC8xRuOwOLkoIvvyZg== Date: Wed, 1 Nov 2017 20:30:53 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Ref=https://api.informationprotection.azure.com/api/72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=decui@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-11-01T13:30:50.3431552-07:00; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General authentication-results: spf=none (sender IP is ) smtp.mailfrom=decui@microsoft.com; x-originating-ip: [2001:4898:80e8:2::7ae] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; PS1P15301MB0009; 6:RCLn/1nrp9rsMPmxsqxVW3UivINanEGapziGXwO6K0dHKjEtrvGbMPkMkkBkcgmWalLlTDAjGAXHC0VtM1fGafcJxKxHL90iYK3936BFwHEvOBbJoqGHqhAx5vAKp3LHNdFkbIebK9NBjoZoEKNXyAAY2umJojSOlVSLmV+J3PNfUPkIqooPfYRO9P5MGAkSOayDURXWJffdZB5wcx2Te4uar85jZWiIznWyoB0ue5ErMctMOW6J6wX07mjlMdD/T/uAtQ4vHsuOuvMu688suQPUMF0eN+MuRrxTKTg5NvK+jo3WYbcfiwJ769DzQytbuLkZC3/LPj0tHnG9vgVv2xZTR9dWkSwG3oJZv1QcHzU=; 5:uzhv0M5bYFHi4ZsjUi9D0HGji1GiUOU1H0Ic/Ungg/HDR+/YPIXE37Ek+hYF35dccqCFcDlMDZ9uk6nsKDMcicii+4f/SV3mxA9KYhmaSOncO4Lbk0D9PtT4f33wTWH4JaxlLwmqxNgCddejI7fjcGiO8Ta8Y0VXz/VDGvouTkU=; 24:paw8gvNXDcfgSsrYWQ5QD8vzBcP7fIX4xBS0sZ+MTa25mtYxzPd66+NrAi3FBnrlz6e3Qg19tOoxJ8sUMJUCwfNgYHirKSNrePy0xOcp6sM=; 7:M5iHiGZ59Z5kdTpM7OuApPLRzcQKa4t5NLGEhOnzFH8bKHvNCKnM68mUR5DVlrYNiOJx3MbEqViOt+NfGhj02nwEw3QCFIeGOxlijVFZJaPDNJ3Mp9xfp0K5tsdEMT7ywcXbY8sywjRagphbFUgSsqUkQVNVLrkos/Rll4PeWyEutRvh5GoEcGmcZonl1ipzx7PdA6UL7l3+1Dub4UvS6PpiUOjOR6r6qN7WOgKVtUeZR6tznqvv8j59nT8NKZD3 x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: f5cfcc9a-74ea-4fd9-7e45-08d5216772ec x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(48565401081)(2017052603199); SRVR:PS1P15301MB0009; x-ms-traffictypediagnostic: PS1P15301MB0009: x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-exchange-antispam-report-test: UriScan:(89211679590171); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(3231020)(3002001)(6055026)(61426038)(61427038)(6041248)(20161123562025)(20161123560025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123555025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:PS1P15301MB0009; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:PS1P15301MB0009; x-forefront-prvs: 0478C23FE0 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(376002)(346002)(39860400002)(47760400005)(199003)(189002)(110136005)(105586002)(1511001)(2501003)(189998001)(55016002)(5250100002)(53936002)(2421001)(9686003)(2900100001)(6636002)(4326008)(2561002)(7416002)(7696004)(3660700001)(5660300001)(97736004)(101416001)(106356001)(68736007)(6116002)(102836003)(2906002)(3280700002)(50986999)(8990500004)(10090500001)(14454004)(6506006)(74316002)(6436002)(54356999)(25786009)(7736002)(305945005)(478600001)(8936002)(8676002)(81166006)(10290500003)(86362001)(81156014)(22452003)(86612001)(54906003)(33656002)(316002)(99286004); DIR:OUT; SFP:1102; SCL:1; SRVR:PS1P15301MB0009; H:PS1P15301MB0011.APCP153.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: f5cfcc9a-74ea-4fd9-7e45-08d5216772ec X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2017 20:30:53.2336 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PS1P15301MB0009 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The effective_affinity_mask is always set when an interrupt is assigned in __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic apic_physflat: -> default_cpu_mask_to_apicid() -> irq_data_update_effective_affinity(), but it looks d->common->affinity remains all-1's before the user space or the kernel changes it later. In the early allocation/initialization phase of an irq, we should use the effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt to the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts. Signed-off-by: Dexuan Cui Cc: Jake Oshins Cc: Jork Loeser Cc: Stephen Hemminger Cc: K. Y. Srinivasan Signed-off-by: Jake Oshins Reviewed-by: Jake Oshins --- Please consider this for v4.14, if it's not too late. drivers/pci/host/pci-hyperv.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644 --- a/drivers/pci/host/pci-hyperv.c +++ b/drivers/pci/host/pci-hyperv.c @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data) int cpu; u64 res; - dest = irq_data_get_affinity_mask(data); + dest = irq_data_get_effective_affinity_mask(data); pdev = msi_desc_to_pci_dev(msi_desc); pbus = pdev->bus; hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) struct hv_pci_dev *hpdev; struct pci_bus *pbus; struct pci_dev *pdev; + struct cpumask *dest; struct compose_comp_ctxt comp; struct tran_int_desc *int_desc; struct { @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) int ret; pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data)); + dest = irq_data_get_effective_affinity_mask(data); pbus = pdev->bus; hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata); hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn)); @@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) switch (pci_protocol_version) { case PCI_PROTOCOL_VERSION_1_1: size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1, - irq_data_get_affinity_mask(data), + dest, hpdev->desc.win_slot.slot, cfg->vector); break; case PCI_PROTOCOL_VERSION_1_2: size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2, - irq_data_get_affinity_mask(data), + dest, hpdev->desc.win_slot.slot, cfg->vector); break;