From patchwork Wed Aug 2 17:53:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Casey Leedom X-Patchwork-Id: 9877327 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 07617602BC for ; Wed, 2 Aug 2017 17:55:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 06CC328816 for ; Wed, 2 Aug 2017 17:55:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E88EC28825; Wed, 2 Aug 2017 17:55:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D476C28818 for ; Wed, 2 Aug 2017 17:55:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752052AbdHBRyB (ORCPT ); Wed, 2 Aug 2017 13:54:01 -0400 Received: from mail-by2nam01on0108.outbound.protection.outlook.com ([104.47.34.108]:51840 "EHLO NAM01-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751699AbdHBRx7 (ORCPT ); Wed, 2 Aug 2017 13:53:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chelsious.onmicrosoft.com; s=selector1-chelsio-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=xPU3T4lkoOMiQE35MFlrsYE3BlgNcIk1pjZDQ2ydVIA=; b=h90OGfGEquACVCAAL0J+m1l+US8Dty9JayPPUqXAXaSxEIk/lJFXqvjz/gNp8yGEKSEAR2D2Yl1RundgwSwls4yoYxylnnB3/kzL4D6Lj0MHS50uFPIQ/791gLsPk2Q+nIFMTXrarOxTEQZ7trnO1zx5e9LllRBDsVqB/mpsc58= Received: from MWHPR12MB1600.namprd12.prod.outlook.com (10.172.56.13) by MWHPR12MB1439.namprd12.prod.outlook.com (10.169.206.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1304.22; Wed, 2 Aug 2017 17:53:53 +0000 Received: from MWHPR12MB1600.namprd12.prod.outlook.com ([10.172.56.13]) by MWHPR12MB1600.namprd12.prod.outlook.com ([10.172.56.13]) with mapi id 15.01.1304.023; Wed, 2 Aug 2017 17:53:53 +0000 From: Casey Leedom To: Ding Tianhong , Alexander Duyck CC: Alex Williamson , Sinan Kaya , "ashok.raj@intel.com" , "bhelgaas@google.com" , "helgaas@kernel.org" , Michael Werner , Ganesh GR , "asit.k.mallick@intel.com" , "patrick.j.cramer@intel.com" , "Suravee.Suthikulpanit@amd.com" , "Bob.Shaw@amd.com" , "l.stach@pengutronix.de" , "amira@mellanox.com" , "gabriele.paoloni@huawei.com" , "David.Laight@aculab.com" , "jeffrey.t.kirsher@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "mark.rutland@arm.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "linux-arm-kernel@lists.infradead.org" , "netdev@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linuxarm@huawei.com" Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Topic: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Index: AQHS++OWrGKhofvtIUCheprjRIXqU6JSQPmAgABHnoCAANEhgIAL8fcAgAPZDQCAA1q+dYAAcmoAgAEXs4CAAJoQgIAI088D Date: Wed, 2 Aug 2017 17:53:52 +0000 Message-ID: References: <1499955692-26556-1-git-send-email-dingtianhong@huawei.com> <1499955692-26556-3-git-send-email-dingtianhong@huawei.com> <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> , <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> In-Reply-To: <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=leedom@chelsio.com; x-originating-ip: [12.32.117.8] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; MWHPR12MB1439; 7:4JoNg7hSERFnFyifwBH4B8vE58er7tHSovI5D3gnb9JdEbjF4sk8oDJ8+8mNpjiWyqMDFUet4Re/YkJDQHxJsIhmulIOkR1T4M3y99DA/08HuZuo8hv4CylXMOjCcA8aqLABUmjE22gDffIimynu+1V+gW9iP9RqUNQ9Pozgs/f6qOZ5O5kXVUaL9+MzGA6uHauw3Mz3wQGFqac/+vYJurmVDR9iYQgj9VlO5+3b1MZlpoqtgOeY866Nz1aJHdEPQTOp0pG7nPpGcE2QQGhN1+3JFb6Z8zS9PUfrYs/b+q5scBUGNV3McBbrQaXWyBRTJVthsxPjnhU0/BIo/5XDEVQ2F4ojEShkf0nwxjn0vfv8w2usDww6/7kCgYP1dTC3UYfaMGkAH6BjVjssVTx4XqUNC/k/F8gUTaxeFF4HL/Re12pdmH4rnUxyUmD5ggCwGvfJB6YVLrPzuUwXtRKdhjUphPki7uEj3+h3oUzy93XqozDJwVHaEYKwXBBMDI9Cl/X/plpWxNk8EtHk4DhpjrKrGjzST9/maircPf55dtMoRcD+wgebhl3ShnsaYXzEFKvyHR/OBlLcFeQVduK9VKD/yiOlWlRz8Xz4fep8xQqdDQ0dY3jIDUyPSj/UzeBx16o2hCkaRRctPKFRKR9161je0yxabBV0nRcEA9r5SRt1D3T9MqfkuEKQrCR9BbKtHrmPnt7rqyZCFsWimn5hvOSR9EAGLf7W5VOTYmw6aCUUajT+3KtMGabVDo6BKQ1sXPAluzvnJXi7wTIcR3ZqXPoV28d8h5WxkQb72yz//So= x-ms-office365-filtering-correlation-id: 0d2e73c2-cbcc-47fd-76c3-08d4d9cf7044 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(2017082002075)(300000503095)(300135400095)(2017052603031)(201703131423075)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:MWHPR12MB1439; x-ms-traffictypediagnostic: MWHPR12MB1439: x-exchange-antispam-report-test: UriScan:; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(6041248)(20161123562025)(20161123558100)(20161123560025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(2016111802025)(20161123564025)(6072148)(6043046)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR12MB1439; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR12MB1439; x-forefront-prvs: 0387D64A71 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(39410400002)(39450400003)(39840400002)(39400400002)(199003)(189002)(4326008)(7696004)(39060400002)(53936002)(74316002)(25786009)(189998001)(38730400002)(2900100001)(105586002)(5890100001)(229853002)(8936002)(575784001)(86362001)(77096006)(6246003)(106356001)(68736007)(6506006)(97736004)(93886004)(3660700001)(5660300001)(54356999)(76176999)(3846002)(50986999)(6116002)(102836003)(8676002)(99286003)(55016002)(66066001)(9686003)(8666007)(33656002)(81166006)(81156014)(54906002)(305945005)(478600001)(551934003)(2906002)(14454004)(7416002)(3280700002)(101416001)(7736002)(6436002)(2950100002); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR12MB1439; H:MWHPR12MB1600.namprd12.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: chelsio.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: chelsio.com X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Aug 2017 17:53:52.8262 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 065db76d-a7ae-4c60-b78a-501e8fc17095 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1439 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Okay, here you go.  As you can tell, it's almost a trivial copy of the cxgb4 patch.     By the way, I realized that we have yet another hole which is likely not to be fixable.  If we're dealing with a problematic Root Complex, and we instantiate Virtual Functions and attach them to a Virtual Machine along with an NVMe device which can deal with Relaxed Ordering TLPs, the VF driver in the VM will be able to tell that it shouldn't attempt to send RO TLPs to the RC because it will see the state of its own PCIe Capability Device Control[Relaxed Ordering Enable] (a copy of the setting in the VF's corresponding PF), but it won't be able to change that and send non-RO TLPs to the RC, and RO TLPs to the NVMe device.  Oh well.   I sure wish that the Intel guys would pop up with a hidden register change for these problematic Intel RCs that perform poorly with RO TLPs.  Their silence has been frustrating. Casey ---------- cxgb4vf Ethernet driver now queries PCIe configuration space to determine if it can send TLPs to it with the Relaxed Ordering Attribute set. diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h index 109bc63..08c6ddb 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h +++ b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h @@ -408,6 +408,7 @@ enum { /* adapter flags */ USING_MSI = (1UL << 1), USING_MSIX = (1UL << 2), QUEUES_BOUND = (1UL << 3), + ROOT_NO_RELAXED_ORDERING = (1UL << 4), }; /* diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c index ac7a150..59e7639 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c @@ -2888,6 +2888,24 @@ static int cxgb4vf_pci_probe(struct pci_dev *pdev, */ adapter->name = pci_name(pdev); adapter->msg_enable = DFLT_MSG_ENABLE; + + /* If possible, we use PCIe Relaxed Ordering Attribute to deliver + * Ingress Packet Data to Free List Buffers in order to allow for + * chipset performance optimizations between the Root Complex and + * Memory Controllers. (Messages to the associated Ingress Queue + * notifying new Packet Placement in the Free Lists Buffers will be + * send without the Relaxed Ordering Attribute thus guaranteeing that + * all preceding PCIe Transaction Layer Packets will be processed + * first.) But some Root Complexes have various issues with Upstream + * Transaction Layer Packets with the Relaxed Ordering Attribute set. + * The PCIe devices which under the Root Complexes will be cleared the + * Relaxed Ordering bit in the configuration space, So we check our + * PCIe configuration space to see if it's flagged with advice against + * using Relaxed Ordering. + */ + if (!pcie_relaxed_ordering_supported(pdev)) + adapter->flags |= ROOT_NO_RELAXED_ORDERING; + err = adap_init0(adapter); if (err) goto err_unmap_bar; diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c index e37dde2..05498e7 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c @@ -2205,6 +2205,7 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, struct port_info *pi = netdev_priv(dev); struct fw_iq_cmd cmd, rpl; int ret, iqandst, flsz = 0; + int relaxed = !(adapter->flags & ROOT_NO_RELAXED_ORDERING); /* * If we're using MSI interrupts and we're not initializing the @@ -2300,6 +2301,8 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, cpu_to_be32( FW_IQ_CMD_FL0HOSTFCMODE_V(SGE_HOSTFCMODE_NONE) | FW_IQ_CMD_FL0PACKEN_F | + FW_IQ_CMD_FL0FETCHRO_V(relaxed) | + FW_IQ_CMD_FL0DATARO_V(relaxed) | FW_IQ_CMD_FL0PADEN_F); /* In T6, for egress queue type FL there is internal overhead