From patchwork Thu Jun 23 22:29:26 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Greear X-Patchwork-Id: 914372 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.4) with ESMTP id p5NMTTcq028053 for ; Thu, 23 Jun 2011 22:29:29 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760076Ab1FWW32 (ORCPT ); Thu, 23 Jun 2011 18:29:28 -0400 Received: from mail.candelatech.com ([208.74.158.172]:59902 "EHLO ns3.lanforge.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753179Ab1FWW31 (ORCPT ); Thu, 23 Jun 2011 18:29:27 -0400 Received: from [192.168.100.195] (firewall.candelatech.com [70.89.124.249]) (authenticated bits=0) by ns3.lanforge.com (8.14.2/8.14.2) with ESMTP id p5NMTRCx014567 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 23 Jun 2011 15:29:27 -0700 Message-ID: <4E03BE46.2040405@candelatech.com> Date: Thu, 23 Jun 2011 15:29:26 -0700 From: Ben Greear Organization: Candela Technologies User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100430 Fedora/3.0.4-2.fc11 Thunderbird/3.0.4 MIME-Version: 1.0 To: linux-nfs@vger.kernel.org Subject: Use-after-free in hacked 2.6.38.8 kernel. Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Thu, 23 Jun 2011 22:29:30 +0000 (UTC) 2.6.38.8 kernel, with our NFS bind-source-IP patches and some other stuff, including a tainting module (though that module isn't active in this test). I'm also running the patch I posted a few days ago that explicitly un-links the xpt_ready list: Test is to create 200 unique mounts (using unique srcaddr) and mount/run-file-io-traffic/unmount them every 15 seconds. It hit this bug after about 5 hours. I'm going to try to figure this out, but any help is appreciated! ============================================================================= BUG kmalloc-64: Poison overwritten ----------------------------------------------------------------------------- INFO: 0xffff8800c6da9dd0-0xffff8800c6da9e03. First byte 0x48 instead of 0x6b INFO: Allocated in nfs_get_lock_context+0xa4/0x179 [nfs] age=60 cpu=2 pid=9218 INFO: Freed in nfs_put_lock_context+0x3f/0x44 [nfs] age=70 cpu=0 pid=8543 INFO: Slab 0xffffea0002b7fcf8 objects=30 used=26 fp=0xffff8800c6da9dd0 flags=0x200000000000c1 INFO: Object 0xffff8800c6da9dd0 @offset=3536 fp=0xffff8800c6da9d48 Bytes b4 0xffff8800c6da9dc0: fe b7 0f 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ??......ZZZZZZZZ Object 0xffff8800c6da9dd0: 48 90 b9 b3 00 88 ff ff 6b 6b 6b 6b 6b 6b 6b 6b H.??..??kkkkkkkk Object 0xffff8800c6da9de0: 06 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ......kkkkkkkkkk Object 0xffff8800c6da9df0: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b ........kkkkkkkk Object 0xffff8800c6da9e00: f3 ff ff ff 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 ????kkkkkkkkkkk? Redzone 0xffff8800c6da9e10: bb bb bb bb bb bb bb bb ???????? Padding 0xffff8800c6da9e50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Pid: 9019, comm: btserver Tainted: P 2.6.38.8+ #9 Call Trace: [] ? print_trailer+0x12e/0x137 [] ? check_bytes_and_report+0xb9/0xfd [] ? nfs_get_lock_context+0x94/0x179 [nfs] [] ? check_object+0xb5/0x192 [] ? nfs_get_lock_context+0xa4/0x179 [nfs] [] ? alloc_debug_processing+0x79/0xf2 [] ? __slab_alloc+0x337/0x375 [] ? nfs_get_lock_context+0xa4/0x179 [nfs] [] ? nfs_get_lock_context+0x28/0x179 [nfs] [] ? nfs_get_lock_context+0xa4/0x179 [nfs] [] ? kmem_cache_alloc_trace+0x76/0xef [] ? sub_preempt_count+0x92/0xa6 [] ? nfs_get_lock_context+0xa4/0x179 [nfs] [] ? nfs_file_direct_write+0x1ab/0x752 [nfs] [] ? pollwake+0x0/0x4f [] ? get_parent_ip+0x11/0x41 [] ? __slab_free+0x86/0xf1 [] ? fsnotify_put_event+0x63/0x67 [] ? trace_hardirqs_on+0xd/0xf [] ? nfs_file_write+0x5d/0x169 [nfs] [] ? do_sync_write+0xc6/0x103 [] ? security_file_permission+0x29/0x2e [] ? vfs_write+0xa9/0x105 [] ? fget_light+0x35/0x94 [] ? sys_write+0x45/0x6c [] ? system_call_fastpath+0x16/0x1b FIX kmalloc-64: Restoring 0xffff8800c6da9dd0-0xffff8800c6da9e03=0x6b FIX kmalloc-64: Marking all objects used ============================================================================= BUG kmalloc-64: Redzone overwritten ----------------------------------------------------------------------------- INFO: 0xffff8800c6da9e10-0xffff8800c6da9e17. First byte 0xbb instead of 0xcc INFO: Allocated in nfs_get_lock_context+0xa4/0x179 [nfs] age=173 cpu=2 pid=9218 INFO: Freed in nfs_put_lock_context+0x3f/0x44 [nfs] age=172 cpu=0 pid=8543 INFO: Slab 0xffffea0002b7fcf8 objects=30 used=30 fp=0x (null) flags=0x20000000000081 INFO: Object 0xffff8800c6da9dd0 @offset=3536 fp=0xffff8800c6da9d48 Bytes b4 0xffff8800c6da9dc0: fe b7 0f 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ??......ZZZZZZZZ Object 0xffff8800c6da9dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object 0xffff8800c6da9de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object 0xffff8800c6da9df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object 0xffff8800c6da9e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk? Redzone 0xffff8800c6da9e10: bb bb bb bb bb bb bb bb ???????? Padding 0xffff8800c6da9e50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Pid: 13574, comm: mount.nfs Tainted: P 2.6.38.8+ #9 Call Trace: [] ? print_trailer+0x12e/0x137 [] ? check_bytes_and_report+0xb9/0xfd [] ? rpcb_create_local+0x6a/0x112 [sunrpc] [] ? rpcb_map_release+0x3f/0x44 [sunrpc] [] ? check_object+0x49/0x192 [] ? rpcb_map_release+0x3f/0x44 [sunrpc] [] ? free_debug_processing+0x7a/0x18e [] ? rpcb_map_release+0x3f/0x44 [sunrpc] [] ? __slab_free+0xdc/0xf1 [] ? rpcb_map_release+0x3f/0x44 [sunrpc] [] ? kfree+0x12e/0x166 [] ? rpcb_map_release+0x3f/0x44 [sunrpc] [] ? rpc_release_calldata+0x12/0x14 [sunrpc] [] ? rpc_free_task+0x59/0x61 [sunrpc] [] ? rpc_final_put_task+0x82/0x8a [sunrpc] [] ? rpc_do_put_task+0x2b/0x32 [sunrpc] [] ? rpc_put_task+0xb/0xd [sunrpc] [] ? rpcb_getport_async+0x564/0x5a5 [sunrpc] [] ? get_parent_ip+0x11/0x41 [] ? call_bind+0x70/0x75 [sunrpc] [] ? __rpc_execute+0x78/0x24b [sunrpc] [] ? wake_up_bit+0x20/0x25 [] ? rpc_execute+0x3d/0x42 [sunrpc] [] ? rpc_run_task+0xe3/0xef [sunrpc] [] ? rpc_call_sync+0x3f/0x60 [sunrpc] [] ? rpc_ping+0x42/0x58 [sunrpc] [] ? _raw_spin_unlock+0x45/0x52 [] ? rpc_create+0x493/0x50e [sunrpc] [] ? nfs_get_client+0x50/0x536 [nfs] [] ? nfs_create_rpc_client+0xb1/0xf6 [nfs] [] ? nfs_create_server+0x170/0x48e [nfs] [] ? trace_hardirqs_on+0xd/0xf [] ? nfs_get_sb+0x4e8/0x742 [nfs] [] ? vfs_kern_mount+0xea/0x1f6 [] ? do_kern_mount+0x48/0xd8 [] ? do_mount+0x708/0x770 [] ? alloc_pages_current+0xaa/0xcd [] ? sys_mount+0x83/0xbd [] ? system_call_fastpath+0x16/0x1b FIX kmalloc-64: Restoring 0xffff8800c6da9e10-0xffff8800c6da9e17=0xcc diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index ab86b79..178716f 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -901,6 +901,7 @@ void svc_delete_xprt(struct svc_xprt *xprt) spin_lock_bh(&serv->sv_lock); if (!test_and_set_bit(XPT_DETACHED, &xprt->xpt_flags)) list_del_init(&xprt->xpt_list); + list_del_init(&xprt->xpt_ready); /* * We used to delete the transport from whichever list * it's sk_xprt.xpt_ready node was on, but we don't actually