From patchwork Wed Aug 2 17:53:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Casey Leedom X-Patchwork-Id: 9877329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id CA02C602BC for ; Wed, 2 Aug 2017 17:55:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCCDE28815 for ; Wed, 2 Aug 2017 17:55:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C140F2881A; Wed, 2 Aug 2017 17:55:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id AA41B28831 for ; Wed, 2 Aug 2017 17:55:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pkaKb1Eg1uiXJq1T9C2cKH55DSapV5I6eq0GOM0eulo=; b=rLkBp3L5dflPfV AElS1pYY2XsYZfjJZx7TeRU7GzQ9lnaqC1aqR0apcgNkB/i6pyj7RpSVB/uWoDwyigaVAWvz/Rw8G J+IxLomAoz4x/9c7kbOSWZPBbl/4wSoq+Ls/lL7Yy8Ro7GEWSaV/48GyYg6VaGYqSSja7ybs5Ir45 Z/einQAf3Eof8H6SlysK/EX4aEabgw5noKkfososh/4/HOmINj9AGpyO3jupeDDrvK4z51Gl+G3Ut 9ysxqv5Qk+lNIYJJTbN0Z7/vU7MiivNujEuci2dhXyAsveUkaY0Ox/6h9qNaCw8sFJ/Th1Vgwfhin bcEf1mpy6XAEhVjmkXBA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dcxqi-0005cm-DL; Wed, 02 Aug 2017 17:54:24 +0000 Received: from mail-by2nam01on0092.outbound.protection.outlook.com ([104.47.34.92] helo=NAM01-BY2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dcxqe-0005VM-2x for linux-arm-kernel@lists.infradead.org; Wed, 02 Aug 2017 17:54:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chelsious.onmicrosoft.com; s=selector1-chelsio-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=xPU3T4lkoOMiQE35MFlrsYE3BlgNcIk1pjZDQ2ydVIA=; b=h90OGfGEquACVCAAL0J+m1l+US8Dty9JayPPUqXAXaSxEIk/lJFXqvjz/gNp8yGEKSEAR2D2Yl1RundgwSwls4yoYxylnnB3/kzL4D6Lj0MHS50uFPIQ/791gLsPk2Q+nIFMTXrarOxTEQZ7trnO1zx5e9LllRBDsVqB/mpsc58= Received: from MWHPR12MB1600.namprd12.prod.outlook.com (10.172.56.13) by MWHPR12MB1439.namprd12.prod.outlook.com (10.169.206.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1304.22; Wed, 2 Aug 2017 17:53:53 +0000 Received: from MWHPR12MB1600.namprd12.prod.outlook.com ([10.172.56.13]) by MWHPR12MB1600.namprd12.prod.outlook.com ([10.172.56.13]) with mapi id 15.01.1304.023; Wed, 2 Aug 2017 17:53:53 +0000 From: Casey Leedom To: Ding Tianhong , Alexander Duyck Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Topic: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Index: AQHS++OWrGKhofvtIUCheprjRIXqU6JSQPmAgABHnoCAANEhgIAL8fcAgAPZDQCAA1q+dYAAcmoAgAEXs4CAAJoQgIAI088D Date: Wed, 2 Aug 2017 17:53:52 +0000 Message-ID: References: <1499955692-26556-1-git-send-email-dingtianhong@huawei.com> <1499955692-26556-3-git-send-email-dingtianhong@huawei.com> <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> , <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> In-Reply-To: <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=leedom@chelsio.com; x-originating-ip: [12.32.117.8] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; MWHPR12MB1439; 7:4JoNg7hSERFnFyifwBH4B8vE58er7tHSovI5D3gnb9JdEbjF4sk8oDJ8+8mNpjiWyqMDFUet4Re/YkJDQHxJsIhmulIOkR1T4M3y99DA/08HuZuo8hv4CylXMOjCcA8aqLABUmjE22gDffIimynu+1V+gW9iP9RqUNQ9Pozgs/f6qOZ5O5kXVUaL9+MzGA6uHauw3Mz3wQGFqac/+vYJurmVDR9iYQgj9VlO5+3b1MZlpoqtgOeY866Nz1aJHdEPQTOp0pG7nPpGcE2QQGhN1+3JFb6Z8zS9PUfrYs/b+q5scBUGNV3McBbrQaXWyBRTJVthsxPjnhU0/BIo/5XDEVQ2F4ojEShkf0nwxjn0vfv8w2usDww6/7kCgYP1dTC3UYfaMGkAH6BjVjssVTx4XqUNC/k/F8gUTaxeFF4HL/Re12pdmH4rnUxyUmD5ggCwGvfJB6YVLrPzuUwXtRKdhjUphPki7uEj3+h3oUzy93XqozDJwVHaEYKwXBBMDI9Cl/X/plpWxNk8EtHk4DhpjrKrGjzST9/maircPf55dtMoRcD+wgebhl3ShnsaYXzEFKvyHR/OBlLcFeQVduK9VKD/yiOlWlRz8Xz4fep8xQqdDQ0dY3jIDUyPSj/UzeBx16o2hCkaRRctPKFRKR9161je0yxabBV0nRcEA9r5SRt1D3T9MqfkuEKQrCR9BbKtHrmPnt7rqyZCFsWimn5hvOSR9EAGLf7W5VOTYmw6aCUUajT+3KtMGabVDo6BKQ1sXPAluzvnJXi7wTIcR3ZqXPoV28d8h5WxkQb72yz//So= x-ms-office365-filtering-correlation-id: 0d2e73c2-cbcc-47fd-76c3-08d4d9cf7044 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(2017082002075)(300000503095)(300135400095)(2017052603031)(201703131423075)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:MWHPR12MB1439; x-ms-traffictypediagnostic: MWHPR12MB1439: x-exchange-antispam-report-test: UriScan:; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(6041248)(20161123562025)(20161123558100)(20161123560025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(2016111802025)(20161123564025)(6072148)(6043046)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:MWHPR12MB1439; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:MWHPR12MB1439; x-forefront-prvs: 0387D64A71 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(39410400002)(39450400003)(39840400002)(39400400002)(199003)(189002)(4326008)(7696004)(39060400002)(53936002)(74316002)(25786009)(189998001)(38730400002)(2900100001)(105586002)(5890100001)(229853002)(8936002)(575784001)(86362001)(77096006)(6246003)(106356001)(68736007)(6506006)(97736004)(93886004)(3660700001)(5660300001)(54356999)(76176999)(3846002)(50986999)(6116002)(102836003)(8676002)(99286003)(55016002)(66066001)(9686003)(8666007)(33656002)(81166006)(81156014)(54906002)(305945005)(478600001)(551934003)(2906002)(14454004)(7416002)(3280700002)(101416001)(7736002)(6436002)(2950100002); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR12MB1439; H:MWHPR12MB1600.namprd12.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: chelsio.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: chelsio.com X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Aug 2017 17:53:52.8262 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 065db76d-a7ae-4c60-b78a-501e8fc17095 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1439 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170802_105420_207314_B124FE15 X-CRM114-Status: UNSURE ( 9.07 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "mark.rutland@arm.com" , "gabriele.paoloni@huawei.com" , "asit.k.mallick@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "linuxarm@huawei.com" , Sinan Kaya , "ashok.raj@intel.com" , "helgaas@kernel.org" , "jeffrey.t.kirsher@intel.com" , "linux-pci@vger.kernel.org" , Ganesh GR , "Bob.Shaw@amd.com" , "patrick.j.cramer@intel.com" , Alex Williamson , "bhelgaas@google.com" , Michael Werner , "linux-arm-kernel@lists.infradead.org" , "amira@mellanox.com" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "David.Laight@aculab.com" , "Suravee.Suthikulpanit@amd.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "l.stach@pengutronix.de" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Okay, here you go.  As you can tell, it's almost a trivial copy of the cxgb4 patch.     By the way, I realized that we have yet another hole which is likely not to be fixable.  If we're dealing with a problematic Root Complex, and we instantiate Virtual Functions and attach them to a Virtual Machine along with an NVMe device which can deal with Relaxed Ordering TLPs, the VF driver in the VM will be able to tell that it shouldn't attempt to send RO TLPs to the RC because it will see the state of its own PCIe Capability Device Control[Relaxed Ordering Enable] (a copy of the setting in the VF's corresponding PF), but it won't be able to change that and send non-RO TLPs to the RC, and RO TLPs to the NVMe device.  Oh well.   I sure wish that the Intel guys would pop up with a hidden register change for these problematic Intel RCs that perform poorly with RO TLPs.  Their silence has been frustrating. Casey ---------- cxgb4vf Ethernet driver now queries PCIe configuration space to determine if it can send TLPs to it with the Relaxed Ordering Attribute set. diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h index 109bc63..08c6ddb 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h +++ b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h @@ -408,6 +408,7 @@ enum { /* adapter flags */ USING_MSI = (1UL << 1), USING_MSIX = (1UL << 2), QUEUES_BOUND = (1UL << 3), + ROOT_NO_RELAXED_ORDERING = (1UL << 4), }; /* diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c index ac7a150..59e7639 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c @@ -2888,6 +2888,24 @@ static int cxgb4vf_pci_probe(struct pci_dev *pdev, */ adapter->name = pci_name(pdev); adapter->msg_enable = DFLT_MSG_ENABLE; + + /* If possible, we use PCIe Relaxed Ordering Attribute to deliver + * Ingress Packet Data to Free List Buffers in order to allow for + * chipset performance optimizations between the Root Complex and + * Memory Controllers. (Messages to the associated Ingress Queue + * notifying new Packet Placement in the Free Lists Buffers will be + * send without the Relaxed Ordering Attribute thus guaranteeing that + * all preceding PCIe Transaction Layer Packets will be processed + * first.) But some Root Complexes have various issues with Upstream + * Transaction Layer Packets with the Relaxed Ordering Attribute set. + * The PCIe devices which under the Root Complexes will be cleared the + * Relaxed Ordering bit in the configuration space, So we check our + * PCIe configuration space to see if it's flagged with advice against + * using Relaxed Ordering. + */ + if (!pcie_relaxed_ordering_supported(pdev)) + adapter->flags |= ROOT_NO_RELAXED_ORDERING; + err = adap_init0(adapter); if (err) goto err_unmap_bar; diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c index e37dde2..05498e7 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c @@ -2205,6 +2205,7 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, struct port_info *pi = netdev_priv(dev); struct fw_iq_cmd cmd, rpl; int ret, iqandst, flsz = 0; + int relaxed = !(adapter->flags & ROOT_NO_RELAXED_ORDERING); /* * If we're using MSI interrupts and we're not initializing the @@ -2300,6 +2301,8 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, cpu_to_be32( FW_IQ_CMD_FL0HOSTFCMODE_V(SGE_HOSTFCMODE_NONE) | FW_IQ_CMD_FL0PACKEN_F | + FW_IQ_CMD_FL0FETCHRO_V(relaxed) | + FW_IQ_CMD_FL0DATARO_V(relaxed) | FW_IQ_CMD_FL0PADEN_F); /* In T6, for egress queue type FL there is internal overhead