From patchwork Fri May 11 07:48:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 10393393 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D7AE4601A0 for ; Fri, 11 May 2018 07:49:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7420628875 for ; Fri, 11 May 2018 07:49:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6845728E16; Fri, 11 May 2018 07:49:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E34528875 for ; Fri, 11 May 2018 07:49:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752596AbeEKHtI (ORCPT ); Fri, 11 May 2018 03:49:08 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:58184 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752294AbeEKHtH (ORCPT ); Fri, 11 May 2018 03:49:07 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w4B7kQVP013098; Fri, 11 May 2018 07:48:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=subject : to : cc : references : from : message-id : date : mime-version : in-reply-to : content-type : content-transfer-encoding; s=corp-2017-10-26; bh=12VGZeBZN+PSxTY2VDJpm870fZXz4qblRq+HfqF/KnM=; b=qvnCWhkN1j2zaYm5a9k/ZSDOFXVyr0CJa3bvIF2WIbN3Qb8jfMCl0azo6CGJ3sPL1+qS OX9U1kJ5K+zpyHIdjHqU0AQ1HaqNRXhVYAKJK8tGXvAOMfn17WaYHIIAui8vZCJ71kJd IlE2Mc49r+DW6aqTUF+qzxmsZtlSIepQJxM30hWltW1j6azURlDOyC26F2f5iGLRS5lf w8tHvFOghJULtu3QJ+GBzs28zkSZ2q4qJNiAkCoMHb6DnXSyp+DoDyms3l3/+KSjw7Hj P6P/RsVHvCaYyO6gfiME7GeJbD9QBL+9HGxEGSCysg2HKzc4IFQWf42ZAMljyppk2tkL 7A== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2hvth92xdf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 May 2018 07:48:59 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id w4B7mwBo029800 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 May 2018 07:48:58 GMT Received: from abhmp0007.oracle.com (abhmp0007.oracle.com [141.146.116.13]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id w4B7mvbR015725; Fri, 11 May 2018 07:48:57 GMT Received: from [10.182.71.69] (/10.182.71.69) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 11 May 2018 00:48:57 -0700 Subject: Re: KASAN: null-ptr-deref Read in rds_ib_get_mr To: DaeRyong Jeong , santosh.shilimkar@oracle.com, davem@davemloft.net Cc: netdev@vger.kernel.org, linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com, linux-kernel@vger.kernel.org, byoungyoung@purdue.edu, kt0755@gmail.com References: <20180511052056.GA10547@dragonet.kaist.ac.kr> From: Yanjun Zhu Organization: Oracle Corporation Message-ID: Date: Fri, 11 May 2018 15:48:50 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <20180511052056.GA10547@dragonet.kaist.ac.kr> Content-Language: en-US X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8889 signatures=668698 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1805110074 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 2018/5/11 13:20, DaeRyong Jeong wrote: > We report the crash: KASAN: null-ptr-deref Read in rds_ib_get_mr > > Note that this bug is previously reported by syzkaller. > https://syzkaller.appspot.com/bug?id=0bb56a5a48b000b52aa2b0d8dd20b1f545214d91 > Nonetheless, this bug has not fixed yet, and we hope that this report and our > analysis, which gets help by the RaceFuzzer's feature, will helpful to fix the > crash. > > This crash has been found in v4.17-rc1 using RaceFuzzer (a modified > version of Syzkaller), which we describe more at the end of this > report. Our analysis shows that the race occurs when invoking two > syscalls concurrently, bind$rds and setsockopt$RDS_GET_MR. > > > Analysis: > We think the concurrent execution of __rds_rdma_map() and rds_bind() > causes the problem. __rds_rdma_map() checks whether rs->rs_bound_addr is 0 > or not. But the concurrent execution with rds_bind() can by-pass this > check. Therefore, __rds_rdmap_map() calls rs->rs_transport->get_mr() and > rds_ib_get_mr() causes the null deref at ib_rdma.c:544 in v4.17-rc1, when > dereferencing rs_conn. > > > Thread interleaving: > CPU0 (__rds_rdma_map) CPU1 (rds_bind) > // rds_add_bound() sets rs->bound_addr as none 0 > ret = rds_add_bound(rs, sin->sin_addr.s_addr, &sin->sin_port); > if (rs->rs_bound_addr == 0 || !rs->rs_transport) { > ret = -ENOTCONN; /* XXX not a great errno */ > goto out; > } > if (rs->rs_transport) { /* previously bound */ > trans = rs->rs_transport; > if (trans->laddr_check(sock_net(sock->sk), > sin->sin_addr.s_addr) != 0) { > ret = -ENOPROTOOPT; > // rds_remove_bound() sets rs->bound_addr as 0 > rds_remove_bound(rs); > ... > trans_private = rs->rs_transport->get_mr(sg, nents, rs, > &mr->r_key); > (in rds_ib_get_mr()) > struct rds_ib_connection *ic = rs->rs_conn->c_transport_data; > > > Call sequence (v4.17-rc1): > CPU0 > rds_setsockopt > rds_get_mr > __rds_rdma_map > rds_ib_get_mr > > > CPU1 > rds_bind > rds_add_bound > ... > rds_remove_bound > > > Crash log: > ================================================================== > BUG: KASAN: null-ptr-deref in rds_ib_get_mr+0x3a/0x150 net/rds/ib_rdma.c:544 > Read of size 8 at addr 0000000000000068 by task syz-executor0/32067 > > CPU: 0 PID: 32067 Comm: syz-executor0 Not tainted 4.17.0-rc1 #1 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 > Call Trace: > __dump_stack lib/dump_stack.c:77 [inline] > dump_stack+0x166/0x21c lib/dump_stack.c:113 > kasan_report_error mm/kasan/report.c:352 [inline] > kasan_report+0x140/0x360 mm/kasan/report.c:412 > check_memory_region_inline mm/kasan/kasan.c:260 [inline] > __asan_load8+0x54/0x90 mm/kasan/kasan.c:699 > rds_ib_get_mr+0x3a/0x150 net/rds/ib_rdma.c:544 > __rds_rdma_map+0x521/0x9d0 net/rds/rdma.c:271 > rds_get_mr+0xad/0xf0 net/rds/rdma.c:333 > rds_setsockopt+0x57f/0x720 net/rds/af_rds.c:347 > __sys_setsockopt+0x147/0x230 net/socket.c:1903 > __do_sys_setsockopt net/socket.c:1914 [inline] > __se_sys_setsockopt net/socket.c:1911 [inline] > __x64_sys_setsockopt+0x67/0x80 net/socket.c:1911 > do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 > entry_SYSCALL_64_after_hwframe+0x49/0xbe > RIP: 0033:0x4563f9 > RSP: 002b:00007f6a2b3c2b28 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 > RAX: ffffffffffffffda RBX: 000000000072bee0 RCX: 00000000004563f9 > RDX: 0000000000000002 RSI: 0000000000000114 RDI: 0000000000000015 > RBP: 0000000000000575 R08: 0000000000000020 R09: 0000000000000000 > R10: 0000000020000140 R11: 0000000000000246 R12: 00007f6a2b3c36d4 > R13: 00000000ffffffff R14: 00000000006fd398 R15: 0000000000000000 > ================================================================== I made this raw patch. If you can reproduce this bug, please make tests with it. Thanks a lot. > > = About RaceFuzzer > > RaceFuzzer is a customized version of Syzkaller, specifically tailored > to find race condition bugs in the Linux kernel. While we leverage > many different technique, the notable feature of RaceFuzzer is in > leveraging a custom hypervisor (QEMU/KVM) to interleave the > scheduling. In particular, we modified the hypervisor to intentionally > stall a per-core execution, which is similar to supporting per-core > breakpoint functionality. This allows RaceFuzzer to force the kernel > to deterministically trigger racy condition (which may rarely happen > in practice due to randomness in scheduling). > > RaceFuzzer's C repro always pinpoints two racy syscalls. Since C > repro's scheduling synchronization should be performed at the user > space, its reproducibility is limited (reproduction may take from 1 > second to 10 minutes (or even more), depending on a bug). This is > because, while RaceFuzzer precisely interleaves the scheduling at the > kernel's instruction level when finding this bug, C repro cannot fully > utilize such a feature. Please disregard all code related to > "should_hypercall" in the C repro, as this is only for our debugging > purposes using our own hypervisor. > --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/rds/ib_rdma.c b/net/rds/ib_rdma.c index e678699..2228b50 100644 --- a/net/rds/ib_rdma.c +++ b/net/rds/ib_rdma.c @@ -539,11 +539,17 @@ void rds_ib_flush_mrs(void)  void *rds_ib_get_mr(struct scatterlist *sg, unsigned long nents,                     struct rds_sock *rs, u32 *key_ret)  { -       struct rds_ib_device *rds_ibdev; +       struct rds_ib_device *rds_ibdev = NULL;         struct rds_ib_mr *ibmr = NULL; -       struct rds_ib_connection *ic = rs->rs_conn->c_transport_data; +       struct rds_ib_connection *ic = NULL;         int ret; +       if (rs->rs_bound_addr == 0) { +               ret = -EPERM; +               goto out; +       } + +       ic = rs->rs_conn->c_transport_data;         rds_ibdev = rds_ib_get_device(rs->rs_bound_addr);         if (!rds_ibdev) {                 ret = -ENODEV;