From patchwork Thu Mar 22 20:04:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10302205 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id ADAC860216 for ; Thu, 22 Mar 2018 20:04:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97E53286BE for ; Thu, 22 Mar 2018 20:04:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 87A842895B; Thu, 22 Mar 2018 20:04:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C926A286BE for ; Thu, 22 Mar 2018 20:04:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751771AbeCVUEg (ORCPT ); Thu, 22 Mar 2018 16:04:36 -0400 Received: from mail-he1eur01on0089.outbound.protection.outlook.com ([104.47.0.89]:34064 "EHLO EUR01-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751731AbeCVUEe (ORCPT ); Thu, 22 Mar 2018 16:04:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=m4LpL7XnOzBOtqc0tvYP9qIv+uS2gQt4PVGVhGVAh58=; b=WbS1zO9SpzpNyWl1OaVpIaIp9uzBaFd7lM0BEdpi7hizZnHpeWhZxxF1Ty5iIZtvyncPeOuBmgIlXOlyIG3Gbx4BBDqCwKMELlmYegPdsYr5OxSJIXZnFdXQtpVYCIj1aZI1iO46RcqhvwJGfOJDT1pUXfXH80T9x2ALR66ke2Q= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; Received: from mlx.ziepe.ca (174.3.196.123) by DB7PR05MB4458.eurprd05.prod.outlook.com (2603:10a6:5:1b::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.588.14; Thu, 22 Mar 2018 20:04:31 +0000 Received: from jgg by mlx.ziepe.ca with local (Exim 4.86_2) (envelope-from ) id 1ez6Rj-0007ye-MJ; Thu, 22 Mar 2018 14:04:23 -0600 Date: Thu, 22 Mar 2018 14:04:23 -0600 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org, Leon Romanovsky , Parav Pandit , Mark Bloch Cc: Dmitry Vyukov , syzbot , danielj@mellanox.com, dledford@redhat.com, Johannes Berg , syzkaller-bugs@googlegroups.com Subject: [PATCH rdma-rc] RDMA/rdma_cm: Fix use after free race with process_one_req Message-ID: <20180322200423.GA30368@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [174.3.196.123] X-ClientProxiedBy: AM5PR0102CA0002.eurprd01.prod.exchangelabs.com (2603:10a6:206::15) To DB7PR05MB4458.eurprd05.prod.outlook.com (2603:10a6:5:1b::23) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 66528ebc-bc16-4683-23fe-08d59030203c X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DB7PR05MB4458; X-Microsoft-Exchange-Diagnostics: 1; DB7PR05MB4458; 3:bNMS2bW1sMqdp/kdcFA5QNDhr4Prs6Yd7hKIB+kd+PsG+fi4PQd0t0eQMvf/Lrpk5vrU1/GeC0nDyYeyXvTvslHiszb2IQSjLDoZP3Lw0U9i3wgW0Gv1jgwBttjhVgHn+RKmFsyvzxLqImhixdl8+qjdESKY/3j7eqjhPdHVzbo99KFQoxLHXgB5ry/JWgqy6THu0rapXFkguPwAofXduKDrZ8mhvagBQz+z0K0mj2JHg0CETrIjeqMfUTuCxJIy; 25:GZPbPNiii7pPtLMSgPtxhmlfnMcO8jCuiQfzzON9j8cxXJ3TnVf7rJm+UpMCiEF5YVjIKd8jgRrmHUIbER1PZzw2Vw3wrZPwY6HgtsLCQot/b2+v8Jplz7JBsQx+4ca+OJJ4sVZp3amYZENUAnUIteF7pwiuK5puWqg+LsAMg+9fABEfrkOeLP5rEOJHcVmoMcw17jGkfUDCQcBqBUUspN+EngnDwahbfOV967zmfKVwsap7h+FQJARrbCFYEPq9DYVgDzXkwrg70bW9hCRqUCGhpjHqgn+PzW/csor27TbfYHjXXYhp5VlHyBp5/CwHGRIjMtSw58I1nMpb+/n2Ig==; 31:r6WPciUuxeVBp6pe+CCacYCorsH/owpo/Ou2eiiPkruQU1Iw/PihjEBnp4wFBy2k/bMKPIZGeyjtEpOMrISs0Lag65Vrs87ngGjOLpqyLfz7Kw3f8P9UzihCT9QdqdeZCZVx1/LR+kfdctpWqswMTNWl/DADQW+M8cnsB0PkThlb+tG0h6Mtvmmkr2zbjSKL2BcV66RPOajVkGZXmUxjgmYhOHcZ/cvXZxKCRWc9D9A= X-MS-TrafficTypeDiagnostic: DB7PR05MB4458: X-Microsoft-Exchange-Diagnostics: 1; DB7PR05MB4458; 20:hXVWrd+LGwCdGdLu8ByA3nTcrkP8SyNH4cK0neFsp7qbvAqbOaGfKgRCnaa+kuZOlMJfQJW9RjtoFmleBjHieSpdYJZTgsGb/g5JdwGrTGYvEkTvzGgYCQZajQrSnXdVqKfCqqCwNmPTa9a5c+9sd9L4cmjfJ1xfNG57jDNaUlevnBr+v2i+zOsVa48Lv8NxA1f9THSJyaE1obCApipWydPXkvfI/8slUl6gbVHToLhctigwStakGzEF/XI1o5wjspNOK5/aq36WOx3Nfb+SXh4CMaCX3Z+wTcDAL9FitkgiJY/as7xfsPfXS/x7qL8cO0YGf2mkwpc8/l+Pp7jXfpJhivXVSTCl5dQytIdT+lIhydnOx3cgeDJjf6a/Px9j3yMcPGC9pXRoaOdRlnc3qusFre41QW7MOZnGDvuCF6IDCZbh0SOky5MpOecNOFZs3TplXATDxcc9dyIQA22L3jMi3d05eRFBhA48vU6NyuuJxcjf1XxDHPkOBUhgD/6E; 4:nxPx04g3LkPfQv2RUy6R9C71QPToCnHcvL7gI266rpXMujYM9nEoxllCm9khOoOHLAvLlf68sk5qR5XN34w4CIsOxSlDn2V3aVNokwrQduY93kiuvPZ6lt9Qv6qyos/QnNdPBHXNlIGOEE5IIH1qQYvqyGn7dK0Odj/o+PaX5rm4GfJZ62KjnZcCuMMwOQyv1kbkGchB+XKGlZ3FaNWvsa2KxWHzvDsdBXC1nmzveK6ITWPgNa98Annqb257y/KglgXqIRLN3gS94TL7HVROUQ== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(3002001)(3231221)(944501327)(52105095)(93006095)(93001095)(6055026)(6041310)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(20161123560045)(6072148)(201708071742011); SRVR:DB7PR05MB4458; BCL:0; PCL:0; RULEID:; SRVR:DB7PR05MB4458; X-Forefront-PRVS: 0619D53754 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(376002)(396003)(39860400002)(39380400002)(346002)(366004)(189003)(199004)(51234002)(105586002)(46656002)(66066001)(106356001)(110136005)(1076002)(54906003)(316002)(58126008)(23726003)(33896004)(68736007)(59450400001)(386003)(6636002)(3846002)(52116002)(47776003)(5660300001)(2906002)(97736004)(6116002)(16586007)(478600001)(186003)(36756003)(8936002)(81156014)(57986006)(122856001)(8676002)(81166006)(575784001)(305945005)(83796002)(50466002)(26005)(86362001)(33656002)(7736002)(69596002)(9746002)(4326008)(9786002)(53936002)(9686003)(18370500001)(24400500001)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR05MB4458; H:mlx.ziepe.ca; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DB7PR05MB4458; 23:qQQOWmZcCGnPDj4/lU6XSiRiYNOxQvEAFVKM3JSTf?= =?us-ascii?Q?wLpbdl0+z3uAf87oZFtlgZAk2Vx2xt94PABZ/yRI7NTX8nTDOZR1kCwg0nc7?= =?us-ascii?Q?XNjVDytzNCs7fxP47JbpmeouqatExfdyIA0W8075adsfkLNqHBGW8D6SvcmR?= =?us-ascii?Q?/zB3lz7vWgiyjp7ysJIhN11JukTtbl2HaD6N9a7DcD/bWTr98FbN2NlRf42D?= =?us-ascii?Q?e4UCnAcJz/sgnOOT763cCSsWvTcpBYh0XqnENtDDJMUADP76c88V5XxDDz27?= =?us-ascii?Q?zJfAcuREq+rK49zNUsM9txfNDkR0gGPOYjXgv3IncLP0m4ij4vxGtqDqnU1v?= =?us-ascii?Q?BMb5+bU2XhGWA5ehjmrskznL5EmEb8opQzB2tAE9n2x9g/Vy063mFxaWyX5s?= =?us-ascii?Q?Pn30W5N7aTMgSrCD4GrrajwyUm8f+F3Cf16R/wvu9lmahawkUZ4yKP/SNqtq?= =?us-ascii?Q?j9ourNYGGRUck0ugl03dh0fpK0/iRDw5bGEBkYesXCjkdqvdTMsCSeSyETtc?= =?us-ascii?Q?IWdtnQwocrLlyO6wPjXafBZ14B1zNyOSfNYJKFwP8hRXK9KzzJAQ3/gPoZoX?= =?us-ascii?Q?DEX03wU+cTXornesqItx1XubybTT21SlP8uYsAo66+HgTCamhPlT4cwU3Ww+?= =?us-ascii?Q?WrGql7cL30400cBNVF6I06v9gnY4V+MDfW5ew6KZ9kLdFGYieRqzzz4wUeYf?= =?us-ascii?Q?RdFISoiu/NuW6zQspS8DHCzHvcoD4yt5RhsYbc9qKR1TXaM9hfAVsNX0lQ6S?= =?us-ascii?Q?EFko1MbjXCFnsy7WnHyG9f/CuZ0Vo83Phjn1W3SOZNizrup9H7ncUaXKAZ1g?= =?us-ascii?Q?46o4OCrFu3UBzOAHYKKiQgftW/5Kp8en/B0z+1rsXArDcSQXbeIpOnAGGQ9y?= =?us-ascii?Q?/do0bp4RRNo4CqTNPHsuUNATDqQKA6q/4juyP6zpDoDxgspG2p2KbGqN+i+5?= =?us-ascii?Q?y3nxsW5uy21U9RhDtB50U8riGCPqSufThUMFSAaOXA23Kw4fn08HR1tePpa+?= =?us-ascii?Q?cGNzQZgpUEbLhQCMGicbPKd7J+wfmWEK4s6rLKEMTCuUWYFTgWd3V7qId5dJ?= =?us-ascii?Q?kyzKnZfDJ+wneAyh0NXedFbuYnyvpu0U371p+NHWy1vZC9nCj35dOx8I9+gQ?= =?us-ascii?Q?eOiVPV+CWuQXQfE9E+xMF897+V5uPrjyBLCSFX6XS//Nr7dhFxOwtqXOveyL?= =?us-ascii?Q?63bxoIpM6uwNbB2TlmwJFguTSpknZZGVdsmUvv6FoVRsmnQL3DgrAFIUvwjo?= =?us-ascii?Q?r8LbYeWMl0h0qIMVkoOeW35aGbUMrtwv8kR1LtPQHYb5vFw7opfaPNU+88xR?= =?us-ascii?Q?RHud/VAlJVRU90AZMPXvKIlnhjYU8YA5TTo5IuQyJu2?= X-Microsoft-Antispam-Message-Info: bEgK1A3d2ghhjtw4H9Ri2cETf67mbon9meydzJ4LbLHNGZCbh7+HT5vD2bIfXfnFc0d53TRE59Q8nSAcd6UPE7Cb3xr0A4mrIpfUU2eVCZWOkGYliKg4NvQw5NQILSg5hicq6b0xfdYgK11mSqV49dgjn5S7mOd45Q37G5Ofoqw65i76BXq1JE8/3tkGegXl X-Microsoft-Exchange-Diagnostics: 1; DB7PR05MB4458; 6:zggJJrD7Oi/+NNL+0BGefVdoV2i8zLcZHvsXv90WuAtRhRQbO17Ew+Spsc04pd35ap5YmpToauoU/zfIX7G0isLT8I52kzz9WQXmwOQDByu9yuX+1GVCGMebIZTEinY79gaBzD9VBJwSV490Anx1qcITaLD/O9HN2WUjf8XI16YF50gNsZuh52EsiqDXMzqYArf+M+g0QYj8TulveJpsIeD0kNwJr9YJxqlGWn33Ozc4qJ/qrkdCgIVumOkQO+u5Cs6dsLJPpfKr3ihR3eWmlGBkQbYR0dWNz59H6Eq61bxBr/YxmZC8IDNpmJCryA4cnvsJdoS84GUh4268cFXvEtFiH+zrniwpLH7Xf6Jmm3M=; 5:09LtpyAUiFOYyWK3qFnD6P8byvjOlf7UeZeXkcm7izdwHtq4LMxux/Gk5QNEN/OXyI3AqasGSbvD6+is+SZNCk7rLt5mIywA11kPloTE0X+IcIKfBBElsQvZfItjWrezrJEgMTLNrHfPv6cQ4dCq54nyzYDbfmQ1xsgzG1x3jSw=; 24:OSIvevS7uaQWdie+49FUQ28AlQ61GjfxHsr5/k7Vz/QaiGVvhI8CDu5l2GfjlMB5k4VQv9JpA3GaHpwtWKxxCPF2GrwputLv0kLhPmd7Lmw=; 7:/HVqb9razfA9Er1HY5MwZ4XMpaIavoDKKNS117GqebnIYxbqdN7oGLmgoimYy9MANo1jaMa+w6BG2CKh8YkIvS/KTMZVFdi+mcoRT7eR+3T/NmO5f2urGgJN9FBMeID1FxwAoQy0vfiCWeFL7E3e1lVBpI4l4OguHlXwjjyZ0lTxl1Xm2oiAOMmaTN2hpfRS59ktwJjeBcFf68Gg7xFiXz8B2/Dwby0ERhDmiaN2yE0iGyJSNkLrcgrckbq/GME4 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2018 20:04:31.4560 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 66528ebc-bc16-4683-23fe-08d59030203c X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR05MB4458 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP process_one_req() can race with rdma_addr_cancel(): CPU0 CPU1 ==== ==== process_one_work() debug_work_deactivate(work); process_one_req() rdma_addr_cancel() mutex_lock(&lock); set_timeout(&req->work,..); __queue_work() debug_work_activate(work); mutex_unlock(&lock); mutex_lock(&lock); [..] list_del(&req->list); mutex_unlock(&lock); [..] // ODEBUG explodes since the work is still queued. kfree(req); Causing ODEBUG to detect the use after free: ODEBUG: free active (active state 0) object type: work_struct hint: process_one_req+0x0/0x6c0 include/net/dst.h:165 WARNING: CPU: 0 PID: 79 at lib/debugobjects.c:291 debug_print_object+0x166/0x220 lib/debugobjects.c:288 kvm: emulating exchange as write Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 79 Comm: kworker/u4:3 Not tainted 4.16.0-rc6+ #361 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: ib_addr process_one_req Call Trace: __dump_stack lib/dump_stack.c:17 [inline] dump_stack+0x194/0x24d lib/dump_stack.c:53 panic+0x1e4/0x41c kernel/panic.c:183 __warn+0x1dc/0x200 kernel/panic.c:547 report_bug+0x1f4/0x2b0 lib/bug.c:186 fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178 fixup_bug arch/x86/kernel/traps.c:247 [inline] do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296 do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315 invalid_op+0x1b/0x40 arch/x86/entry/entry_64.S:986 RIP: 0010:debug_print_object+0x166/0x220 lib/debugobjects.c:288 RSP: 0000:ffff8801d966f210 EFLAGS: 00010086 RAX: dffffc0000000008 RBX: 0000000000000003 RCX: ffffffff815acd6e RDX: 0000000000000000 RSI: 1ffff1003b2cddf2 RDI: 0000000000000000 RBP: ffff8801d966f250 R08: 0000000000000000 R09: 1ffff1003b2cddc8 R10: ffffed003b2cde71 R11: ffffffff86f39a98 R12: 0000000000000001 R13: ffffffff86f15540 R14: ffffffff86408700 R15: ffffffff8147c0a0 __debug_check_no_obj_freed lib/debugobjects.c:745 [inline] debug_check_no_obj_freed+0x662/0xf1f lib/debugobjects.c:774 kfree+0xc7/0x260 mm/slab.c:3799 process_one_req+0x2e7/0x6c0 drivers/infiniband/core/addr.c:592 process_one_work+0xc47/0x1bb0 kernel/workqueue.c:2113 worker_thread+0x223/0x1990 kernel/workqueue.c:2247 kthread+0x33c/0x400 kernel/kthread.c:238 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:406 Fixes: 5fff41e1f89d ("IB/core: Fix race condition in resolving IP to MAC") Reported-by: syzbot+3b4acab09b6463472d0a@syzkaller.appspotmail.com Signed-off-by: Jason Gunthorpe --- drivers/infiniband/core/addr.c | 9 +++++++++ 1 file changed, 9 insertions(+) Leon, I took a look at this last bug you noted so we can get cleaned up for the next kernel release. I didn't repo it, but I did confirm the C repo is calling rdma_addr_cancel, so I think this is very likely to be the bug.. Parav/Mark: Does this make sense? I'll hang on to this till Monday so you guys can check it. diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c index b0a52c99620834..826b1edd9b58b6 100644 --- a/drivers/infiniband/core/addr.c +++ b/drivers/infiniband/core/addr.c @@ -586,6 +586,15 @@ static void process_one_req(struct work_struct *_work) list_del(&req->list); mutex_unlock(&lock); + /* + * Although the work will normally have been canceled by the + * workqueue, it can still be requeued as long as it is on the + * req_list, so it could have been requeued before we grabbed &lock. + * We need to cancel it after it is removed from req_list to really be + * sure it is safe to free. + */ + cancel_delayed_work(&req->work); + req->callback(req->status, (struct sockaddr *)&req->src_addr, req->addr, req->context); put_client(req->client);