From patchwork Wed Feb 1 15:17:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Gurtovoy X-Patchwork-Id: 9549943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7E74660425 for ; Wed, 1 Feb 2017 15:18:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BB1228442 for ; Wed, 1 Feb 2017 15:18:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5D3A428445; Wed, 1 Feb 2017 15:18:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1677F28442 for ; Wed, 1 Feb 2017 15:18:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751195AbdBAPSp (ORCPT ); Wed, 1 Feb 2017 10:18:45 -0500 Received: from mail-eopbgr30042.outbound.protection.outlook.com ([40.107.3.42]:51120 "EHLO EUR03-AM5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751096AbdBAPSo (ORCPT ); Wed, 1 Feb 2017 10:18:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=nVQ1fO9RHzqIyx9rCXVQZVqpq8zeh2Kbo6n+3AG5lwQ=; b=IdOd1UeUeH7oof4xyxDI7wKt6OH+gF9Tg0k9WSUFIj6ja7E+sdZVOPuSeXBy8OBDg0hCZmRmVJsCjlLhdyeezM7QMf/Tbbx/3v5+xS8aDor7MOba72Mwoz+qlD6nNeIJRt41zbXvhJKQW39tzmR6HWTVw7sGSG4hqI/RmyHXEbA= Received: from HE1PR05CA0081.eurprd05.prod.outlook.com (10.164.28.49) by DB5PR05MB1894.eurprd05.prod.outlook.com (10.166.173.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.860.13; Wed, 1 Feb 2017 15:18:40 +0000 Received: from DB3FFO11FD045.protection.gbl (2a01:111:f400:7e04::125) by HE1PR05CA0081.outlook.office365.com (2a01:111:e400:59a1::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.874.12 via Frontend Transport; Wed, 1 Feb 2017 15:18:40 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; chelsio.com; dkim=none (message not signed) header.d=none; chelsio.com; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD045.mail.protection.outlook.com (10.47.217.76) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.874.2 via Frontend Transport; Wed, 1 Feb 2017 15:18:40 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Wed, 1 Feb 2017 17:18:38 +0200 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4 via Frontend Transport; Wed, 1 Feb 2017 17:18:38 +0200 Received: from [10.223.3.90] (10.223.3.90) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.301.0; Wed, 1 Feb 2017 17:17:39 +0200 Subject: Re: iSER fails to release rdma resources (WRs) if iw_cxgb4 is unloaded while IO is in progress To: Raju Rangoju , Sagi Grimberg , "linux-rdma@vger.kernel.org" References: CC: SWise OGC , Potnuri Bharat Teja From: Max Gurtovoy Message-ID: <49223bb6-e2a1-eee7-cf4e-701957e6727f@mellanox.com> Date: Wed, 1 Feb 2017 17:17:39 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: X-Originating-IP: [10.223.3.90] X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:193.47.165.134; IPV:NLI; CTRY:IL; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(7916002)(39450400003)(39850400002)(39410400002)(39840400002)(2980300002)(438002)(199003)(189002)(377454003)(24454002)(83506001)(38730400001)(76176999)(229853002)(512944002)(270700001)(4610100001)(54356999)(81166006)(568964002)(50986999)(64126003)(356003)(512874002)(81156014)(8936002)(6306002)(4326007)(84326002)(8676002)(77096006)(6116002)(2950100002)(3846002)(92566002)(54906002)(2906002)(33646002)(966004)(345774005)(2476003)(189998001)(31696002)(36756003)(86362001)(305945005)(2501003)(7736002)(5001770100001)(31686004)(5890100001)(65956001)(106466001)(65806001)(626004)(4001350100001)(5660300001)(65826007)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR05MB1894; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:mail13.mellanox.com; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD045; 1:AY9tlpUdMC6u6Om6PjRblOIHH0eWRBhT7b+V2LWrjit2XHt3+5zqGTYOoWcyOiv0+7s8KAJ2fX3WkhwnWNW/uUb4qhcnUNvS+Kpv16JRX+uu8dCHKYdG14YZo/9ynbFwlAb6tj5Q/BY19LIglQ9slmfb7C4Z17npqPsCpxn/hUNkz4y352kQfxZvnvffQq5TNGyvmMv5f+GG2VxpFRuUkzpuOd1tuA2v6L+KoRkPeWBwbCkOCIrcmIBWky1P1rKK1J+dJhmzv96wgbDWLVpRVn+auSU4PMRDT/4P4oATb0e1aTT6Zk3TyWGyVggokELltj/9//o4xBJTIgkGAKhDtcd1i/be+1n4E+XrXLQ/TJcPd5GgUyxLcK8i/8QxcyPBDfyebgSryo65zazH5Pa82zkfVuhgiHtGn5+mN+HUIcdo4XDC3Fnf2QHOMhZBbflscq/xOtV3kKq48WCIHVssGzHNgKkEhF+na5MXtjjHnLI= X-MS-Office365-Filtering-Correlation-Id: 6f2ab3ae-16e3-4adc-3d85-08d44ab59a37 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(8251501002); SRVR:DB5PR05MB1894; X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1894; 3:GBq5wHH2qJNLBoPeXvCjPqKVpr/kt6HumiLJE9slsggWQ027CjJhbsI5hBnwqa4vk6jMja7YiBqDHhcE2xYGEnt0pBComZ3o96z4hFQwTJrhp3rfNZCFtYTH205kbKV7FQ8rQiFnVePFWa5MCGtqC0X61ktvUuVGGZPMBpJ3QkXHgg9FhrKxbmf/KthQMkCO4vK2+VhAbBsYvET8I6RFCBUInGakVmiGaFw3DF5hLBNimKFxqIzGjBF9Pte5jiS47OOrA6Hp0dl2rRKSiRjY1y/0h0aC3ssGE2dGp217z2dEfmRSoFakYuMAKbwAQiqIw8mTRM2vpLo9GGkwoLgK0cQrU5V5hFzpnn/OqAqF+7TfAb1wn6d6s4+v+SBlid/YYRAxTxGpI9WKkMLr0Stzeg==; 25:tstzkZOalZsdk2q7ZwHUKLt2NpkfA6gBY3nwYQqZKIiYAJAKlTtUw5dyfamgAxDaAYOESDjOtiZm3OEoPq+6px+oGz9oNKUguw1yjsx2Oi+8ITLWRU0BPRu+6ydsgGxBLn399RH6uC1Q5nYYxWxIsb147WNtOmZ1766CuvZu7Xtua1Dkx9IaWKDq39xgReEcryJTDRUg2k2/vlZ7OLgHw3Qks55JBgt+hr+09SKh+KHC9xUznr51c94sYDbH5WJ4JQGdfq6Sgi3QPrteiDl+mC8sy8yRw1sAiEoqAV9xK6jPb+a2h+F7VTkqVasSLnP9Brtatb3lHCgsbFRbGSYzJFqgN0/xt4gK2YRvqs02SvqdY/c0WPj2hxVj3hcSnwTJFv9Va83bJxMTfxOON7i40JsjZOq5QO5MJcQmLT7E7IGx5QvzcE9EQccWOuBFuIC96uftV4Vgrls2HUoqhtlsNJmK13sUsJG+bhdJK/rcH5nHGgpXwm/WY+DLiYLV3x3b X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1894; 31:kQMWRRsOzVyRupej3NtTUdcdnNAQhe88UJMd0X/5Bc8eBbUR/ez+tpRUmL3tGnaLuHRmGh+VOjSaSkUJ5Rlo1v+jMZau2zmWnIG5jW/kzoitkSwL2YLzMQXaqjR8Zxe2FIzez9BSs7JWlK6ZnPEygKy/i+q3E5D70vFmcWxNhLqp6qXX+sQj2D5MQhzTakvJ06z+fOtUe+wdz/ztmQpHSFp7VGGtnme+UXS32figNt1rgKBwPLeTNWx8TtkdqaJrfzSJhhmYbG8wWU1KEkFdvzUiNn2HZ0f3EO+oaQILF3Y3PMY1KwcH6wQ3eXV3iC/XXmZ8PLTeJF7AJJbAeqr8hA==; 20:rcfQObLPgMXdCrE1u7LXCgUlXWUe8e0g2qQ3CymN7fXhUjhnEM9wz8cIfXseyyfxcMN/EdOK1LK6a0T77XwFFhGApHr57/hiMbYCvkWDB3u5gsveRtmzHyTpZS/DjhzEekw15+ByZ2TrpY+deu/bU978VQyUZdPewNkTwliwKjHAlCPgbxIH5flUPzLB1/+LoW1yqatO4VM93IBUgd0jOdPqoe7jW9wiHPqT1tKzTXJX+GMF375erlNWru8gznUXS6dEibBLr5GlUUTW+FFKyHrige9EgfwU0JGHksv4KaZduRrZOMO0e+1LfJh8FsJDBjr/GShO3vEo1QISRkx4/wQPhvnle5G2NBDxKHm6jIUW0gis+KXWUALjAXZEXmw0i9Xd7kU8Mfqce048/3nTfD4TfsKqkmmUBlOqf+YECESjoAi7lWWidTR2MRhV78iR3wUBdQrOwLiE7qAhpnfts3Ea/Avxr+QDMu+6h13SL2Y5Fx5xx8bJSNRkpfSYvNsf X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(9452136761055); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(102415395)(6040375)(601004)(2401047)(13016025)(8121501046)(5005006)(13018025)(3002001)(10201501046)(6055026)(6041248)(20161123555025)(20161123558025)(20161123560025)(20161123564025)(20161123562025)(6072148); SRVR:DB5PR05MB1894; BCL:0; PCL:0; RULEID:; SRVR:DB5PR05MB1894; X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1894; 4:RnOZY+L1GjtDyoSprgjhyODj02krH1hA+ZSGsL96qb5rWaETAjABTkQz9fS+TDmp9q8guf+8Denn8GrlI/T1cmhGj+TQME2LNMRJY8IS0BQ4D10e1/F/BN3mJqAQ8Sl9SPyCv8HpfyTy4GJkCRWXMX1uEJE+alzIjA/SQVcuw0rFOIKF7S40dHEERY1FNXe0fh7l2bK/2OIqMnIt0x4CuLcOYOaf0TQYY+cnzu9gb19Rbs4UxJ8FYy7Tt24TslfFSz+S1RkngpFX7EfX/+/nsv1YIJRxDtQT9v/SyvfomGCeg5HKTQwwne9QQxWU1IqzvXF9j6FwmCl7to549XnCHLE51YZgyNEfCeUvpNYbTDiMeBjUMjvMKUuLnMiyZZbFHfAaH39cjXmmmcNADRlY+IRjY23xBzvBZ8hIqNBFOZiFfpOKnfbQQSeX4RlPE9OZtDW3JHGscp7BKSC2zeL1YCtY60M96skozKhqjAlNpMoo+hfaAG3bS3jYnzRBE/kPaDNyNaiychyd6XsSOZgl/zpp2Rw3dM9pzgm56IagqxYSyqKzSqNkI7LrbeiRUAtJ0vSmWnJtruu6E1YEYFs/r57bqMLzYfIYbKgL2kxmhlkv/zp9wDmVyntCeNGKBBIxxfPZxrQS3SI1VO8gFXsxbKZqt2ogkA2oRW5BD4q9IDVaq/NPmQjbWUGqvFifJJ+W8GrBlsp3WgCf+kNkP8KUDNOgJU84qG7PE60aOZ+KB4g= X-Forefront-PRVS: 0205EDCD76 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB5PR05MB1894; 23:XnDjvEUJhHk3CizEj0gKN80B7XsMO2nnmQDQXpA5A?= =?us-ascii?Q?jrwbb20aItH3alUnE8op3bpDDabIWSIUqIr3FdXQlQs65IAa5V+gG5zwAi/Z?= =?us-ascii?Q?9EwJHtxTZuRv+RBHM/MkuJ6izypqxyCYrfRNM4o+SBK8GsTag+4YoVoO5oOh?= =?us-ascii?Q?7ZSypAPOnwVDl6f/R20MAR6+Ail+2vYkBAMNhpphwt9gZo8wUXcfOmqqJ2AQ?= =?us-ascii?Q?llwMFpW8ZxHrrVfblI/i6Mk5jiqhzsJu+6Jon+gArmucwvHREPomimGUrrY4?= =?us-ascii?Q?8YhRXXD1PHBJGQgG6DJY4AkHuyBugSeHbI4LMKDEitRjgDCk94JpoEbZ6R1f?= =?us-ascii?Q?EPln0KPFw0NKg+TPc51S3i3fhTAEaXhF9Wt3OS9epLHudhuTEgVHH5D5mYzQ?= =?us-ascii?Q?8REohuuYrv6QRyTsVOD8AMJPd5t2Q2c9mwlMfFKvXkH6Y6AYLQ9EHT5Qy00w?= =?us-ascii?Q?mDjlJ84BUIATCYpcurG9HHmaSBFAEdOxyyER8qYnmiJ5h/sycw0gs04jttED?= =?us-ascii?Q?SvI/CNQeMWw02Rstcvl5gY00COnWOsMExw8HXJKu97wZtGDnb2AyKm3IKsrc?= =?us-ascii?Q?v2XBFClqhafyynGIMzwA48JvDAlx4y6Cy/9PidJZL22QWlKIFF2hSW81VepA?= =?us-ascii?Q?74aoxnBlIjj4pbBje7G7nHe4a0rT3fNmiMBIMUpCM1j7L1uk5g9jYPLQXYfG?= =?us-ascii?Q?OkZj866XFCjmhaM/jG1VDfTeDJrT/XQxaK2Tu/rYxEWzFwXx2HBvGBjvxhNX?= =?us-ascii?Q?1+hjCvNgNfwdZvtrMWUQKlTYeeJboQGe3FXCVI9qNY6Hff4FwMDH9nh812xy?= =?us-ascii?Q?8IhlivJCfISL5W4BGA3RGaKnd/gCMqYBoCaLnwQd93wKcCi3yXSWSUJi1UBm?= =?us-ascii?Q?7/nKe/P/UVc0X+GHLNe60cb1HJ3Rwzz1Lab1zheTLAcAhPe4yzx3Ry/0M551?= =?us-ascii?Q?GIkyla63hqdGI4Sfw7NGE9Wn3bkf2rdie6oc7+YZTiCxTLa9X05kDi/2yMbx?= =?us-ascii?Q?ijhVKykEybRrSFGh+XgbjuHaE7tFIdK0eU6JXQIKC9cMtf7Fbn01BL+qPzIk?= =?us-ascii?Q?cCVybNBg6ulkBgHGRVHDQeSHdHdOdgFwpPPir1sdjpKx3+/2V1JqN9ADsbre?= =?us-ascii?Q?AxJXBkRUjaaT6ufQSiZGnlvmOhnBDaGGkyPCF3rUiSTiabBtMB93FZQkdGIq?= =?us-ascii?Q?7UVDfzLaV1BKr2OJEzEgoEH3D1Mxg0SPBQ2y0wjHVQc7rGOMS8GcXvokYyTY?= =?us-ascii?Q?4ZcyOKNrSXcJSVcYJ0rUnmBnlIsv6R2T/sA+j2BgQUdyQAdT8viVkhB8WnhV?= =?us-ascii?Q?bd4PTbvd8UKOtOaJbDEyK1/1hJKKeFyO93rlaLrlCr33qHP1rqu7lbzy0dd/?= =?us-ascii?Q?vfrUuHT1i28A2zv4L/Il4sFT4gV5Yu3p5OpR7suqsFxSdUlh47oxADjxVuAT?= =?us-ascii?Q?fJ72lVBJOKpqhhFFmVFH75SicJ4hp5ySjKxz7YXGPr390gyGbwdCbHaZhAoI?= =?us-ascii?Q?NGsG58MLqjcxQ=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1894; 6:SuV6KQrKspphkdt59nJ5Jsxx4T+a3qAUI9ySraThS2SHA72UdacjWjeaokVuaX2lLFE8aBBPldxN4uJ8fI2Yx4RYlouewHs1LWIy6LE2Pw0n4BLehVcFZX9mqKBbH22WZoCrYA5UP0lE6uoDsLO+5dY7IyEjqrxho7IGzO/20reizYmv92AdwZoNyDJSy9dMjJripu9wZzH/mPZXTF+LHkHgUFxXdZBLrQwFjgc6UUtwjiXntjHmvg0XOU/nRVfjMpLOSOaYaOkI91XbDTygXi5LA3Hbm0V0GWjg/hy4F/1A2jf29hu0Woz1uRIA51QaKOnR27nDmIL8xwD2+yDMa1qnl7+V1aO/uZodhlZkmnbPqt4ZPr2UQAPjGIwegIrDuNEqA+th+1HnNXNbD0XfD9ZtQ8C4es/q+g6hEsxwNXfTJIVed1uBawko7jXnFEif; 5:frjFqzWWNDVhqJAK7+Ff3PwODF0uoc2iznPSsNEY7I+hEKUHYaXlOgJvV5lOAtrNiIeBPiUIiQpluSgucXgYwnt68cQyypSh0A2+80psyPmztrlXmghMFKvENwYMfD6Bzj5BoA+rQcvXzBOc7sQH4A==; 24:9/chL38/pe3OWuym1ZO6DsrANFVOFQ7SXPxjLatOuVkulj8VQCQEGXQ00N3lggTUWpXtkYswDR+k59reSBQr+LetKEAj4Yj8MgrnTF/z8qc= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1894; 7:EUqnuTD5p8+8dhs7Lx2Fte9EqxS7oMZx+XfrYJsipDuepPiofdSM1iv+DlzvLIxO2+iQWW+rauCphFuBK9IBevsJSowiO0sBFPKaAy3iKfuyQTzUue8pMw1Bql7BYmAoOTF+ya/XXWs8qCHmYZMnQ67IP9ZMADzVhAQQ32CI19vAco1b7BsMke4ZSwFEsyU2XMzVdpGK32DaJHk6Pip9yzJvcpAtaXLuKHVYbYqB2PterKtvgAZaDlE+FA9gUV9YrznDp/VsquoFtmLvaUCbPG+4IDx0nDfq5hxfPQyc+zgqFVIAWVkVPqMXMrLEebkIYtDmk2bwvr6um3LhLIpPEfw/ZBqHL7PGvt69tjEL4tYy0QBnZNSr3Bw5sp5yCgKUI/P0hxv3W+3saYiI4O9K6EMsRd6LsfZCU/WseseOjaj1ZsfEfV+x9/Tj+5QhogUCXxmzftnjNsSWMyELvnYKrX2sKyoQiBp7AutX6SLvhDffZ0mq/R+0TrpNMFZN8Cq8tfkhVnwzRYGbzX0o7eijp1n/q3QXuatFpr+6vDLxW4rcXZADCoatqrf0kseAEP+C X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2017 15:18:40.0590 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1894 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP hi Raju, please apply the attached patch I want to push soon (still haven't find the chance to test it). I'm not sure it will solve your problem but let's try it. thanks, Max. On 2/1/2017 11:08 AM, Raju Rangoju wrote: > > Hello Sagi, > > I intermittently see an issue with iser when unloading the iw_cxgb4 module while traffic is running. Apparently the rdma resources are not getting released when the iser receives RDMA_CM_EVENT_DEVICE_REMOVAL event while the IO in progress. iser_cma_handler() upon receiving the DEVICE_REMOVAL event, destroys the device by calling iser_cleanup_handler(). iser_free_ib_conn_res() destroys the qp and calls iser_free_fastreg_pool() to free the Memory Regions in the fastreg_pool list, and then it calls ib_dealloc_pd. > > Issue: > > iSCSI uses its .xmit_task and .cleanup_task callbacks to get/put MRs from iser fr_pool(fastreg_pool) during the normal IO, at this point if the DEVICE_REMOVAL event is received, iser_cma_handler()->iser_cleanup_handler() it simply releases the available MRs in the fr_pool list (some MRs may have been moved to running task list) and eventually calls ib_dealloc_pd, which ends up hitting kernel panic as some registered MRs are not freed up. > > iser_free_fastreg_pool() complains about the registered regions; "pool still has %d regions registered" > > Trace: > > iser: iser_free_fastreg_pool: pool still has 1 regions registered > iser: iser_device_try_release: device ffff880508660080 refcount 0 > iw_cxgb4:c4iw_destroy_cq ib_cq ffff8803f3addc00 > iw_cxgb4:c4iw_wait_for_reply add wr_waitp ffffc9000dd83a28 > ------------[ cut here ]------------ > WARNING: CPU: 7 PID: 14790 at drivers/infiniband/core/verbs.c:305 ib_dealloc_pd+0x87/0xd0 [ib_core] > Modules linked in: rdma_ucm ib_uverbs iw_cxgb4(OE-) autofs4 target_core_iblock target_core_file target_core_pscsi target_core_mod bnx2fc fcoe libfcoe 8021q libfc garp stp llc scsi_transport_fc cpufreq_ondemand be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb3i libcxgbi iw_cxgb3 cxgb3 mdio libcxgb ib_iser rdma_cm ib_cm iw_cm ib_core configfs ipv6 crc_ccitt iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi uinput ppdev iTCO_wdt iTCO_vendor_support serio_raw pcspkr parport_pc parport tpm_infineon sg i2c_i801 i2c_core lpc_ich mfd_core e1000e acpi_cpufreq i7core_edac edac_core ioatdma dca ext4(E) mbcache(E) jbd2(E) sd_mod(E) pata_acpi(E) ata_generic(E) ata_piix(E) floppy(E) cxgb4(OE) ptp(E) pps_core(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) > CPU: 7 PID: 14790 Comm: rmmod Tainted: G OE 4.10.0-rc4+ #22 > Hardware name: Supermicro X8ST3/X8ST3, BIOS 2.0 07/29/10 > Call Trace: > dump_stack+0x51/0x78 > __warn+0xfd/0x120 > warn_slowpath_null+0x1d/0x20 > ib_dealloc_pd+0x87/0xd0 [ib_core] > ? ib_unregister_event_handler+0x6d/0x80 [ib_core] > ? mutex_lock+0x16/0x40 > iser_device_try_release+0x81/0x120 [ib_iser] > ? iser_free_rx_descriptors+0xd3/0xf0 [ib_iser] > iser_free_ib_conn_res+0x75/0xb0 [ib_iser] > iser_cleanup_handler+0x41/0x70 [ib_iser] > iser_cma_handler+0x1c9/0x220 [ib_iser] > cma_remove_id_dev+0x8f/0xa0 [rdma_cm] > cma_process_remove+0x127/0x170 [rdma_cm] > ? kobject_cleanup+0x82/0x1b0 > ? kobject_release+0xd/0x10 > cma_remove_one+0x6f/0x90 [rdma_cm] > ib_unregister_device+0xe7/0x190 [ib_core] > c4iw_unregister_device+0x79/0x90 [iw_cxgb4] > c4iw_remove+0x45/0x6c [iw_cxgb4] > c4iw_exit_module+0x31/0x75 [iw_cxgb4] > SyS_delete_module+0x183/0x1d0 > ? syscall_trace_enter+0x154/0x1f0 > ? SyS_munmap+0x6e/0x90 > do_syscall_64+0x6c/0x160 > entry_SYSCALL64_slow_path+0x25/0x25 > RIP: 0033:0x37d22e8ee7 > RSP: 002b:00007ffedd1877b8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 > RAX: ffffffffffffffda RBX: 00007ffedd1877c0 RCX: 00000037d22e8ee7 > RDX: 00007ffedd1877af RSI: 0000000000000880 RDI: 00007ffedd1877c0 > RBP: 00007ffedd187810 R08: 00007f0120b48700 R09: 0000000000000100 > R10: 0000000000000011 R11: 0000000000000206 R12: 0000000000000880 > R13: 00007ffedd188735 R14: 0000000000000000 R15: 0000000000000001 > ---[ end trace 9bdbdddd5759d7e6 ]--- > > > Steps to reproduce: > 1. Bring up the iser target setup > 2. Bring up the iser initiator setup > 3. From DUT(initiator) login to all the Targets and start IOzone traffic on all the mounted luns. > 4. Now unload iw_cxgb4 module on the iser initiator setup. > > > This is a generic issue, seen with other vendors also. > > Could you give me a few pointers on how to debug it further to address this issue? > I am happy to provide any details further. > > Thank you for any help you can provide, > -Raju > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > From c1ec2dc3660fbd3e99dfc4aa3d766a526205aa6d Mon Sep 17 00:00:00 2001 From: Max Gurtovoy Date: Wed, 1 Feb 2017 13:09:48 +0200 Subject: [PATCH 1/1] IB/iser: access active_qps field atomically Signed-off-by: Max Gurtovoy --- drivers/infiniband/ulp/iser/iser_verbs.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c index 6a9d1cb..30b622f 100644 --- a/drivers/infiniband/ulp/iser/iser_verbs.c +++ b/drivers/infiniband/ulp/iser/iser_verbs.c @@ -597,7 +597,9 @@ static void iser_free_ib_conn_res(struct iser_conn *iser_conn, iser_conn, ib_conn->cma_id, ib_conn->qp); if (ib_conn->qp != NULL) { + mutex_lock(&ig.connlist_mutex); ib_conn->comp->active_qps--; + mutex_unlock(&ig.connlist_mutex); rdma_destroy_qp(ib_conn->cma_id); ib_conn->qp = NULL; }