From patchwork Tue Oct 31 03:16:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Parav Pandit X-Patchwork-Id: 10033567 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 93658602B9 for ; Tue, 31 Oct 2017 03:16:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 85C97288CE for ; Tue, 31 Oct 2017 03:16:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 78A5A288D1; Tue, 31 Oct 2017 03:16:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C91C9288CE for ; Tue, 31 Oct 2017 03:16:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752003AbdJaDQr (ORCPT ); Mon, 30 Oct 2017 23:16:47 -0400 Received: from mail-eopbgr10061.outbound.protection.outlook.com ([40.107.1.61]:29563 "EHLO EUR02-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751900AbdJaDQq (ORCPT ); Mon, 30 Oct 2017 23:16:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=5CpjPo5KFsMjo1Y/i59rAo9QDTPCm7N8Qb+6DWnys1U=; b=ve1kh1G+948YNzRlhy3iKA81qnCyV8tYQIb+i6nH3wcDs0gy0KiAh9YWMKGMdsjXZao1U1pEkFPwFapxiMiJye7qlgGwEvvaohvyECWSUl4M+f/7Y4rhFn1nOoQkwFK6I/Ymzy3BocMYt7ujsfHY97dab46qHAKjr5Dzk54P7EY= Received: from VI1PR0502MB3008.eurprd05.prod.outlook.com (10.175.21.22) by HE1PR0501MB2265.eurprd05.prod.outlook.com (10.168.34.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.178.6; Tue, 31 Oct 2017 03:16:42 +0000 Received: from VI1PR0502MB3008.eurprd05.prod.outlook.com ([fe80::3c23:f8da:c50f:8b36]) by VI1PR0502MB3008.eurprd05.prod.outlook.com ([fe80::3c23:f8da:c50f:8b36%13]) with mapi id 15.20.0178.012; Tue, 31 Oct 2017 03:16:42 +0000 From: Parav Pandit To: Jason Gunthorpe , Chris Blake CC: Leon Romanovsky , "linux-rdma@vger.kernel.org" , Daniel Jurgens Subject: RE: 4.13 ib_mthca NULL pointer dereference with OpenSM Thread-Topic: 4.13 ib_mthca NULL pointer dereference with OpenSM Thread-Index: AQHTTn5NqoU3R8GyaUiIjUEMVV+dc6L7NmQAgAAGv4CAAMTZAIAAve0AgABJRACAADvywA== Date: Tue, 31 Oct 2017 03:16:42 +0000 Message-ID: References: <20171029191114.GO16127@mtr-leonro.local> <20171030071956.GU16127@mtr-leonro.local> <20171030230156.GA4081@ziepe.ca> In-Reply-To: <20171030230156.GA4081@ziepe.ca> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=parav@mellanox.com; x-originating-ip: [68.203.16.89] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; HE1PR0501MB2265; 6:IAA/jXOmLfxXe+HYpI2vstAgrj2pkRLuTA8NrEARFx6TCDJxgcD0+q6zkbKK5RTbxIFQNpTcjGA/mv/9+aneZg+DsrS5XObvKx3c6hS4718jM+n/6wgeCSu66e2E9yAMDnuVby3Og+94ilv2Jg7V8kM1OsbuS76+j8BEMBlzfmjNR/fyNALC1Lv36ndah7pqbOlZ6rbqdSpSx5MsCkoGkS5a0GJtDnmg2HxiumH0Vn24JyjHzfqHCmLKbIsBdyxcFVdTH91Y+95UIlVsVouHuvfC6OnTNIhaIyXu1Dxbt2yeL38joFfnS565FwzHOFV8qNgDKilAY39qA/Iv8ZJYMiN8786fG1xMEQHQvgOLZPI=; 5:A1pFL0lTN+cuhAQEh3ddgDzLbfPkPzWj5gq+08yGYim5aKMir/0NRm1jZk2uI6MwkWXuC5mbZIbjRnrF9HY/rIHVAyrFIICuugdh7JBdQGu818y9NYrlJXOYJESNnuvVhtyn9Wt1bj2Nk/X+19sY4KmAq//tBM5ncrSYfKpp+3c=; 24:YVvH1piuvEfREFoG6NeukkqtUt6+ffDlcKai+2BgoaEa1yuYocOp4mgjQUUIWoct5+aXXYxz3ingtM139+UUP+nnYAStWBdNYXp70MFcvcU=; 7:cRHUy5MaASuCRZjjAc2paIMItDsoGo70GF2RXDJro+bZD5M+L4Fa6fdqeEm5qK572NgYZidHx3gXOQ41Rj4wrXiirnrUrqFezdAbaeaAXllU0jyyFEDdv16+KjyypIC67bACHMLume+XqBfTCcgqSAYUpLazwkNmpNL/AfGYF69+toExfw65rAwBr++vV/AQ2t2QGXgI6tTNAxV+UkRUT73QmV9biOaMdhJmsHUlHlhQEh2SXmKqF5L60GntM7fN x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-ms-office365-filtering-correlation-id: 1aeeea5e-56c9-4b7a-d052-08d5200dcf23 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(48565401081)(4534020)(4602075)(2017052603199); SRVR:HE1PR0501MB2265; x-ms-traffictypediagnostic: HE1PR0501MB2265: x-exchange-antispam-report-test: UriScan:(192374486261705)(9452136761055); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(10201501046)(3231020)(93006095)(93001095)(3002001)(6055026)(6041248)(20161123558100)(20161123560025)(20161123564025)(20161123555025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR0501MB2265; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR0501MB2265; x-forefront-prvs: 04772EA191 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39860400002)(346002)(376002)(24454002)(189002)(13464003)(199003)(107886003)(189998001)(54356999)(97736004)(106356001)(2906002)(7696004)(110136005)(54906003)(2950100002)(316002)(105586002)(68736007)(5660300001)(229853002)(86362001)(6436002)(5250100002)(9686003)(39060400002)(3846002)(102836003)(6116002)(81166006)(25786009)(3660700001)(8936002)(478600001)(6506006)(33656002)(66066001)(93886005)(81156014)(7736002)(3280700002)(14454004)(50986999)(2900100001)(8676002)(305945005)(53936002)(76176999)(99286003)(74316002)(55016002)(4326008)(101416001)(6246003)(53546010); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0501MB2265; H:VI1PR0502MB3008.eurprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1aeeea5e-56c9-4b7a-d052-08d5200dcf23 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Oct 2017 03:16:42.3281 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2265 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Jason, > -----Original Message----- > From: Jason Gunthorpe [mailto:jgg@ziepe.ca] > Sent: Monday, October 30, 2017 6:02 PM > To: Chris Blake > Cc: Leon Romanovsky ; linux-rdma@vger.kernel.org; Parav > Pandit > Subject: Re: 4.13 ib_mthca NULL pointer dereference with OpenSM > > On Mon, Oct 30, 2017 at 01:39:42PM -0500, Chris Blake wrote: > > On Mon, Oct 30, 2017 at 2:19 AM, Leon Romanovsky > wrote: > > > > > > Can you please try to set CONFIG_SECURITY_INFINIBAND=n and see if it > > > helps? > > > > > > Thanks > > > > > > > Hello Leon, > > > > I went ahead and set CONFIG_SECURITY_INFINIBAND=n in my kernel, and so > > far the issue seems resolved. I will run this for a week or so and > > will get back to you, but things are looking promising. :) > > I certainly don't expect this setting to break any drivers.. > I looked the back trace - happening in freeing ib_free_recv_mad(). It doesn't look a driver issue certainly. Post_send failure seems to indicate that security enforcement checks likely would have failed on QP0/1. I tried ib_ipoib and rping with 4.13.10 and ConnectX4 but that didn't help with reproduction. I tried injecting error locally on recv mad, based on suspect and I was able to crash a host and with below patch I was able to avoid it. I am yet to review my below patch with Dan as he did most security dev, but I suspect this might be the cause where rmpp list is not initialized and mad processing is continued when security check fails. Let see if Chris has same issue or different one. Chris, Can you try below patch and see if that avoids the crash? More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c index f8f53bb..cb91245 100644 --- a/drivers/infiniband/core/mad.c +++ b/drivers/infiniband/core/mad.c @@ -1974,14 +1974,15 @@ static void ib_mad_complete_recv(struct ib_mad_agent_private *mad_agent_priv, unsigned long flags; int ret; + INIT_LIST_HEAD(&mad_recv_wc->rmpp_list); ret = ib_mad_enforce_security(mad_agent_priv, mad_recv_wc->wc->pkey_index); if (ret) { ib_free_recv_mad(mad_recv_wc); deref_mad_agent(mad_agent_priv); + return; } - INIT_LIST_HEAD(&mad_recv_wc->rmpp_list); list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list); -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org