From patchwork Thu Sep 7 07:40:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jan Glauber X-Patchwork-Id: 9941687 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9660E6035F for ; Thu, 7 Sep 2017 07:41:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8489C20001 for ; Thu, 7 Sep 2017 07:41:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 78A082853B; Thu, 7 Sep 2017 07:41:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7F53720001 for ; Thu, 7 Sep 2017 07:41:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YcOPn/a2Avky4CKkQ/W5lItCWfVVB6MVpsYQ8uv6ktE=; b=jniwjvORv1EST5 i18SSVfeONr8nGzmVisKZK+aCP9L/YhJLrHnHYfBBX3luv99csNWBOhg4YabbIpN/WagM48CFsiw8 LN6GrGMmz49BJIVdFaMwtZ0fWYymiAq4G8d3agwGt8oyQ2kwkZRwIUW7TXB32zWPHlgWbKJNICdaI l81nohNqHttOk3J0jm2zGRWLGP1BXjV//MCJNSOoQqUFAKD6pO/R4Z86R8Ldid1Ej8hiO+glWOp3+ fkBveLCaPVlxz3O5Dwjgv4D7Dh2tRHSm8zvqHQW5NJ/geMimbsBqgKGzkFYB4KKk9YXyAmz40UL5r tVJXWOQhdYUBOwg+LQJg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dprQg-0007ns-RR; Thu, 07 Sep 2017 07:40:50 +0000 Received: from mail-bn3nam01on0064.outbound.protection.outlook.com ([104.47.33.64] helo=NAM01-BN3-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dprQc-0007Vu-Nu for linux-arm-kernel@lists.infradead.org; Thu, 07 Sep 2017 07:40:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=3yrnQxM5sccC1LmFSYongBTfLtOZPpa4aDxN2GwMQpA=; b=KIgwzsSnmFrFC+ZyRpQsTxQAl4qfPwo180dTTTv9SyTu1mDqRhwg1LDGIynTuvABKKsvROzEyuBwCIbaJIqucYwWKvffH5hHsiRQ3NN4ihc7nsF+7BgNn8MfuOrp2gNPDzHl2a+T4kjTjBSHbJz3JPOCiechw1cBqkiuLYigRFQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jan.Glauber@cavium.com; Received: from hc (46.223.66.108) by BN3PR07MB2577.namprd07.prod.outlook.com (10.167.4.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.13.10; Thu, 7 Sep 2017 07:40:21 +0000 Date: Thu, 7 Sep 2017 09:40:11 +0200 From: Jan Glauber To: Alex Williamson Subject: Re: [PATCH v3 3/3] PCI: Avoid slot reset for Cavium cn8xxx root ports Message-ID: <20170907074011.GA13490@hc> References: <20170830142454.10971-1-jglauber@cavium.com> <20170830142454.10971-4-jglauber@cavium.com> <20170830084012.19d91759@w520.home> <20170831094052.GA15906@hc> <20170831100130.5c8a922e@w520.home> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20170831100130.5c8a922e@w520.home> User-Agent: Mutt/1.5.21 (2010-09-15) X-Originating-IP: [46.223.66.108] X-ClientProxiedBy: DB6PR0801CA0044.eurprd08.prod.outlook.com (10.169.219.12) To BN3PR07MB2577.namprd07.prod.outlook.com (10.167.4.154) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 574473a8-79b2-40ee-9489-08d4f5c3b2c4 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:BN3PR07MB2577; X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2577; 3:Gxq0DF3LHMbKfor2Rp/xruN+vUyy9J+AmkkdVqqwYeNLj8h0KTwWfqnHoiQO6JxEYI274P5CAT6ZVnbUdA1rPbPUt/ev5AFHswzP5Bomo4xMNLSPHewW9bS5TcJhsrWaDjE2qQx5xDnCQXCipt8ZFSX7tIFf7X7jF3PTCCmmZYZuK2zSCpmSR63EGnckYVt5cUKpy2SznRuywukNCFXPevrrUV2UNwLMdLo9pJoRJj8LVvqw1+m1lsgWgJtMeP69; 25:xv9BQtn1hz4C2+sZKCpjN4K/R9rfOfWRIelES3mRacT2aoQUI4S7971r0aykDnaTe2h1j6bzoALwE3LJKnN1imEecN0hBaLEl78/zE7T3+ljprj7wQKKCjczLsK6vv1WXpvA5H6auKMY5gnJhpAaKcZc2EuFTqKevma8M4MZKnQQ/ovYGIy9rBjpuec2gjumuYG0mtBplkjRyd/uhCUD+aJj4k91aH0K7WylxlrdTIDo6O9sjU6pQrj8wrCxS3091y60+y3U3/ZfRn6kiiQDZgTjjsxEC4aGcromdPnvlaA2JqbG4Wwca+jh4xpxnKGoLtfDim9RB1cWpLDfXTTtXA==; 31:C6OueiD3GrWkuhTVTUqO8VlhAFN4p8TaGsxCU2pk7niYR7Fm1lY8A78mNk7yzCd7ondc1qMVwED2KGYUuSeFpSNef/V2RV4am/PkV2QJ8sNTpIRhC8KlYc9ByDBqCavOKJxcKMw4Uucz/LJDp0UtUtAsXSwYfuVuMpBMMKpImoFdyjB1oXTjHvrv3wREvyupBxeUFmk2h1ALxX05IyJ+UWmVFTkob8nJFgo43vqRTvw= X-MS-TrafficTypeDiagnostic: BN3PR07MB2577: X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2577; 20:xsBVSvhppOglN0ZskgPQAe4lgcuCHPa2uA2g//ty2fDOh2GxiqAQMfEO2u4zbflgqOBFvzJhIEXnqu3oaPRNYcTsD4TTogjxG1JJ8KIi/p2mTN6MVHU1XzzexX1sEc+1C/W8qdurC2ajejCAmSKhnjIkYuo0TU9nVSsvSBVa3bSAPnzwApOKhlfifNybu9trbkjbxoANvQr4uPvil6yLTf9z8DrQl2lJSIDltYo188hlefVEj6LveX6f8y/eCrUzCWoDXk/hR/ZOVFE3alRvaqfrCSmtTk3MeRnvAwDmyvYQTFmbzu0NxhkYin6d9l4L2/1t89b5DH/VKyks4NnwVwPH0DCMB1hmVddwhojqa9P7M7C0iw86UBqQj0nwlfUzEmIro7XljJF2eYKjEnF8C5w6pyM7TllOEqRbFWcZ1d8gtYdwtrlKP7iRyMBY3j7CwsWQK/1ZTgDAh32KH2gYQ/GIXxCK8ef368Olrraxi8eCAauNQ6hww7m+05JvwcWFtGM3I47NnuZQjJH/7NReqmywQ1SpwlMAudfS1BlrEf1NpwhQPvvGaA+9l1TVQ+7q6CaLAvBJyfFCs8R9Y35pLLWLY513Syy2UAFBOQ/KZvw=; 4:fulRdMmI8B592jTp/6d+6ppNix27wAF9DnV5lyOsoUZQx8GCLBr841vU0p8RNoKF6iaWCOLSNR85WqunwGpq1/Reh5evUQlUC6za6WSUo166dmy3HxwLNhNixxK7axEVHJglOQuTOCCxKaemX1twZtarUKd282QQpFeVMqqDy+R6ev1QHD5mANUSbkeiI637YJm/B3muv1EDpZU+9PLgCP31djDusHkFAjzB9ej9MtJrHJfuoiC5W/B0u2FL85AU X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(100000703101)(100105400095)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123564025)(20161123555025)(20161123558100)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:BN3PR07MB2577; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:BN3PR07MB2577; X-Forefront-PRVS: 04238CD941 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(189002)(199003)(24454002)(52314003)(42882006)(2950100002)(6916009)(189998001)(9686003)(54906002)(53936002)(55016002)(6246003)(110136004)(6496005)(229853002)(3846002)(6116002)(1076002)(6666003)(23726003)(83506001)(4001350100001)(93886005)(97736004)(50986999)(33716001)(76176999)(2906002)(5660300001)(106356001)(25786009)(105586002)(4326008)(8936002)(81156014)(81166006)(478600001)(7736002)(8676002)(305945005)(101416001)(50466002)(54356999)(42186005)(68736007)(66066001)(33656002)(47776003)(72206003)(18370500001)(32563001); DIR:OUT; SFP:1101; SCL:1; SRVR:BN3PR07MB2577; H:hc; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR07MB2577; 23:vQbsmhCcvr+SIIu/kS+rs+fCksP6kVZdNIgnvKy9h?= =?us-ascii?Q?f7gRkQkwthMNka3LwgIHad6I9TfN09P/lmPXTlPd4MrjS53wn6Z6bCCtxqj0?= =?us-ascii?Q?uKvDlQuo7+obrirSXK4UaHvsR0qL8t060BQy4xIc2IzPd2/HJBixPxMz7rP8?= =?us-ascii?Q?Z/S57r6zSRyYIcJkHCcJh5oX25zV9VJ0QHjBaBnzqOHqWfuO641SRmOHeOHl?= =?us-ascii?Q?zvYtnbrlgcZ1VYzvk1gC36d4dKipdFMFXnEDu1IUUWvHhb5HOI/0SZNmZ30I?= =?us-ascii?Q?8+6ANnlF4cajIGP04PFK9o6RXKigZhE+7USJgsS2htMGhrEfUNVJGDavQQ3x?= =?us-ascii?Q?mShLm1XvGx8hkXD7XzoYaV2zh1XsrhTowGF8lZ1YMCTBPTeZ3NOJbbHFeW25?= =?us-ascii?Q?K/663fFVpm/LyP89DiGZuQ2Aa1xPh80Ahz/9JzgzRlnlCdTPhw7y2DM7e5Vi?= =?us-ascii?Q?9TzH7mi/KQxdzMPBmdiaWcFTuwjNzw4JpAmoJrDc+JTVWSfztuhTz9GWZnMG?= =?us-ascii?Q?YLaxpNL1N5NGZdCbJD1ZPBzVbEEIX2SDLrRyf7ydpr0v11gYpvSn4O4vb3mD?= =?us-ascii?Q?DANx1Q2kV2x+002q8A6sy5YzWwUkYL+tZrntTHpYx6CT78VT4oXI1ErcSfCJ?= =?us-ascii?Q?FtDGOM/7ynZ1bhVYfmqriTV9f9W/H0mK6a8vQN8QNzSVPUwmy5rIhevw72yW?= =?us-ascii?Q?QMqrVTA5JqROW8iwN2NOmtppUSxMxTVFht8iueDOE0zT+lKDrO4NUKCBT1qv?= =?us-ascii?Q?ugesMp/jvXjHoEXy6AMQvca7V5bv59+woR+JFCXiXVqWXKYJxz7+ObhE0vAj?= =?us-ascii?Q?MiuNOMraDGa1Iy8f6GTSB31oogxrRPKW/wfw4UPON+08jb6jgFrHMMDNrQiv?= =?us-ascii?Q?zcX6cEaIfOuikVG3M1+gwnoPJ7YtWFYmez7vqtAQGUONm0y4IeJGOpLxaFl3?= =?us-ascii?Q?OHAMIGU6zBrS3Dihfuq1Ei/+97IswX53hpsIJsMzDz/67E3FqKYAYCyTCKEF?= =?us-ascii?Q?gvAWZ98UGXcyB+tUAay3l8kfQNxu1DUJsLDX43BqL13ikRVR4Z8BwLQ9OFjL?= =?us-ascii?Q?WwMVYTWCy871vZQ1TPojEbHcZ59m2XW6lZGaxg+rXngYXW/ASsTsvuSeWy8n?= =?us-ascii?Q?3L7g9zlae/xD0uk6qOCzcQd/T9v5lsucjUwpqFjFXOWztVNLWsScN5UTOk1+?= =?us-ascii?Q?qzGAPAMUm51WaRz41TCHeRUaxZzYEgYZ1c+FoalGCkwLQQjTyBtsHgSi7LxI?= =?us-ascii?Q?5woU5U/13QAIPRwPYw=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR07MB2577; 6:rnQeHVaJVYuvRMoPmy+U71blM5qAcarnHMmFupNLCQMoqcJftNZp8kJ20Xq8CC46noVreHjlhqg8UuN29ijwrliIjuzQ+SCSUUR1vLa0l7ehH97p7mdpQ3WEgY1/Tx4siDYYf3+tTPcpEIHmt3e4bqeslfr+8jiyuLMKy+UEeyMb+yXiPtV2kPhuPv3DZPSZF29CRfPaqFivEy3Q6wO2ArpsRl9gX+mqof5/eSNJJxjeco+YwTdnl8ycFjMJzD3xzXg6prFRrw0DEUCJeCWn2FjYLXYmDurJ20/SsjKdc3R6t6ikYBmEQKkcnMHf80h3svxtK7Nu39PD5Apj/5DWRg==; 5:qLJrDn4+v7LR5A2ej3C9fme1wURyKgAG6lU5z6HpM/OMJTiOQnIbXpxWtzg8rlt88IPRsGBTniH6nPHptD74rFJEY0qBYjUAP0q2ilvMDx/2XXPB0ryGYHcy+b8YuPiP+z4FcewBN6SbLdLygyW7bw==; 24:FlT0/ugqXs0v4si99bntS5s8alz8Rfl2Z70Qc8g6JYYqB/27NQLyb4rvKpvjdrDb4T7sobjM4ZzcoskXvnOiaks7JZH0g5dGOPihV70KlIE=; 7:xa+DyQYerawFM9SRK7YHUSfITEmrYpdguzt2fZ9F340zwZacDact4fCJXKqUNieUDt6LtE+Uvt684wCstIpR7iac8aFjhQVP/M6Q+hF3QDmp4Q/wwanuFzuLR2ZDoVwMxLclEm4bOksCr7n64J3yuPB+jYA4yXYy9iFi29lpZr5NF4j55O2aol/NornKv7aJJ8xglT+D35++1KP/0MqtBt46SbUG3/CzY5yUcA0iV1o= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Sep 2017 07:40:21.4612 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR07MB2577 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170907_004046_946033_85EC172A X-CRM114-Status: GOOD ( 27.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, david.daney@cavium.com, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Robert Richter , Jon Masters , Bjorn Helgaas , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Aug 31, 2017 at 10:01:30AM -0600, Alex Williamson wrote: > On Thu, 31 Aug 2017 11:40:52 +0200 > Jan Glauber wrote: > > > On Wed, Aug 30, 2017 at 08:40:12AM -0600, Alex Williamson wrote: > > > On Wed, 30 Aug 2017 16:24:54 +0200 > > > Jan Glauber wrote: > > > > > > > Root ports of cn8xxx do not function after a slot reset when used with > > > > some e1000e and LSI HBA devices. Add a quirk to prevent slot reset on > > > > these root ports. > > > > > > > > Signed-off-by: Jan Glauber > > > > --- > > > > drivers/pci/quirks.c | 16 ++++++++++++++++ > > > > 1 file changed, 16 insertions(+) > > > > > > > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > > > > index 85191b8..6679971 100644 > > > > --- a/drivers/pci/quirks.c > > > > +++ b/drivers/pci/quirks.c > > > > @@ -845,6 +845,22 @@ static void quirk_cavium_sriov_rnm_link(struct pci_dev *dev) > > > > DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_CAVIUM, 0xa018, quirk_cavium_sriov_rnm_link); > > > > #endif > > > > > > > > +/* > > > > + * Root port on some Cavium CN8xxx chips do not successfully complete > > > > + * a bus reset when used with certain types of child devices. Config > > > > + * space access to the child may quit responding. Flag all devices under > > > > + * the secondary bus as non-resettable. > > > > + */ > > > > +static void quirk_CN8xxx_secondary_bus(struct pci_dev *dev) > > > > +{ > > > > + struct pci_dev *pdev; > > > > + > > > > + dev_warn(&dev->dev, "Cavium CN8xxx quirk detected; reset for devices on secondary bus disabled\n"); > > > > + list_for_each_entry(pdev, &dev->subordinate->devices, bus_list) > > > > + pdev->dev_flags |= PCI_DEV_FLAGS_NO_BUS_RESET; > > > > +} > > > > +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_CAVIUM, 0xa100, quirk_CN8xxx_secondary_bus); > > > > + > > > > /* > > > > * Some settings of MMRBC can lead to data corruption so block changes. > > > > * See AMD 8131 HyperTransport PCI-X Tunnel Revision Guide > > > > > > > > > This doesn't seem reliable, doesn't the user just need to remove and > > > reprobe the slot and the device would re-appear without this flag set? > > > > No, I tried before to disable the slot with "echo 0 > /sys/bus/pci/slots/3/power" > > but that does not work as it is not supported. > > > > I'm not familiar with the quirk types, would another one be better > > suited here (even if we don't have the problem you descibed)? > > The scenario I'm mentioning is to "echo 1 > /sys/bus/pci/devices/ device under the slot>/remove", then "echo > > /sys/bus/pci/rescan". This would break the ordering implicit in using > a fixup defined for the root port. It seems like it'd make a lot more > sense to add a test on the parent bridge more similar to how the bus > reset works. It's not the subordinate devices imposing the > no-bus-reset flag, it's the bridge device and the objects and code > should support and reflect that. Thanks, Doing "echo > /sys/bus/pci/rescan" after the remove did not work for me, but maybe the format of the device address needs to be different. Anyway, the sequence echo 1 > /sys/bus/pci/devices//remove echo 1 > /sys/bus/pci/rescan still triggers the panic as you mentioned above. I agree that the subordinate devices are not causing the issue, still I need to make pci_slot_resetable() return false in our case. So what if we add an additional check like: --Jan diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index fdf65a6..389db4b 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4389,6 +4389,9 @@ static bool pci_slot_resetable(struct pci_slot *slot) { struct pci_dev *dev; + if (slot->bus->self & PCI_DEV_FLAGS_NO_BUS_RESET) + return false; + list_for_each_entry(dev, &slot->bus->devices, bus_list) { if (!dev->slot || dev->slot != slot) continue;