From patchwork Mon Oct 10 01:35:44 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Conole X-Patchwork-Id: 9368853 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5849E60487 for ; Mon, 10 Oct 2016 01:36:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B78028BBB for ; Mon, 10 Oct 2016 01:36:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2CA7D28BC6; Mon, 10 Oct 2016 01:36:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC51928BBB for ; Mon, 10 Oct 2016 01:36:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751487AbcJJBgT (ORCPT ); Sun, 9 Oct 2016 21:36:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45866 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750991AbcJJBgS (ORCPT ); Sun, 9 Oct 2016 21:36:18 -0400 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id EC32FC04D29A; Mon, 10 Oct 2016 01:35:49 +0000 (UTC) Received: from dhcp-25-97.bos.redhat.com (vpn-57-25.rdu2.redhat.com [10.10.57.25]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u9A1Zjpw031099 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 9 Oct 2016 21:35:46 -0400 From: Aaron Conole To: Florian Westphal Cc: Linus Torvalds , Al Viro , Andrew Morton , Jens Axboe , "Ted Ts'o" , Christoph Lameter , David Miller , Pablo Neira Ayuso , Linux Kernel Mailing List , linux-fsdevel , Network Development , NetFilter Subject: Re: slab corruption with current -git (was Re: [git pull] vfs pile 1 (splice)) References: <20161010005105.GA18349@breakpoint.cc> Date: Sun, 09 Oct 2016 21:35:44 -0400 In-Reply-To: <20161010005105.GA18349@breakpoint.cc> (Florian Westphal's message of "Mon, 10 Oct 2016 02:51:05 +0200") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1.50 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 10 Oct 2016 01:35:50 +0000 (UTC) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Florian Westphal writes: > Linus Torvalds wrote: >> On Sun, Oct 9, 2016 at 12:11 PM, Linus Torvalds >> wrote: >> > >> > Anyway, I don't think I can bisect it, but I'll try to narrow it down >> > a *bit* at least. >> > >> > Not doing any more pulls on this unstable base, I've been puttering >> > around in trying to clean up some stupid printk logging issues >> > instead. >> >> So I finally got a oops with slub debugging enabled. It doesn't really >> narrow things down, though, it kind of extends on the possible >> suspects. Now adding David Miller and Pablo, because it looks like it >> may be netfilter that does something bad and corrupts memory. > > Quite possible, the netns interactions are not nice :-/ > >> Without further ado, here's the new oops: >> >> general protection fault: 0000 [#1] SMP >> CPU: 7 PID: 169 Comm: kworker/u16:7 Not tainted >> 4.8.0-11288-gb66484cd7470 #1 >> Hardware name: System manufacturer System Product Name/Z170-K, BIOS > .. >> Call Trace: >> netfilter_net_exit+0x2f/0x60 >> ops_exit_list.isra.4+0x38/0x60 >> cleanup_net+0x1ba/0x2a0 >> process_one_work+0x1f1/0x480 >> worker_thread+0x48/0x4d0 >> ? process_one_work+0x480/0x480 > > .. > >> like it's a pointer loaded from a free'd allocation. >> >> The code disassembles to >> >> 0: 0f b6 ca movzbl %dl,%ecx >> 3: 48 8d 84 c8 00 01 00 lea 0x100(%rax,%rcx,8),%rax >> a: 00 >> b: 49 8b 5c c5 00 mov 0x0(%r13,%rax,8),%rbx >> 10: 48 85 db test %rbx,%rbx >> 13: 0f 84 cb 00 00 00 je 0xe4 >> 19: 4c 3b 63 40 cmp 0x40(%rbx),%r12 >> 1d: 48 8b 03 mov (%rbx),%rax >> 20: 0f 84 e9 00 00 00 je 0x10f >> 26: 48 85 c0 test %rax,%rax >> 29: 74 26 je 0x51 >> 2b:* 4c 3b 60 40 cmp 0x40(%rax),%r12 <-- trapping instruction >> 2f: 75 08 jne 0x39 >> 31: e9 ef 00 00 00 jmpq 0x125 >> 36: 48 89 d8 mov %rbx,%rax >> 39: 48 8b 18 mov (%rax),%rbx >> 3c: 48 85 db test %rbx,%rbx >> >> and that oopsing instruction seems to be the compare of >> "hooks_entry->orig_ops" from hooks_entry in this expression: >> >> if (hooks_entry && hooks_entry->orig_ops == reg) { >> >> so hooks_entry() is bogus. It was gotten from >> >> hooks_entry = nf_hook_entry_head(net, reg); >> >> but that's as far as I dug. And yes, I do have >> CONFIG_NETFILTER_INGRESS=y in case that matters. >> >> And all this code has changed pretty radically in commit e3b37f11e6e4 >> ("netfilter: replace list_head with single linked list"), and there >> was clearly already something wrong with that code, with commit >> 5119e4381a90 ("netfilter: Fix potential null pointer dereference") >> adding the test against NULL. But I suspect that only hid the "oops, >> it's actually not NULL, it loaded some uninitialized value" problem. >> >> Over to the networking guys.. Ideas? > > Sorry, not off the top of my head. > Pablo is currently travelling back home from netdev 1.2 in Tokyo, > I can help starting Wednesday when I am back. > > One shot in the dark (not even compile tested; wonder if we can end up > zapping bogus hook ...) > I was just about to build and test something similar: --- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/netfilter/core.c b/net/netfilter/core.c index c9d90eb..e84103f 100644 --- a/net/netfilter/core.c +++ b/net/netfilter/core.c @@ -189,7 +189,7 @@ void nf_unregister_net_hook(struct net *net, const struct nf_hook_ops *reg) unlock: mutex_unlock(&nf_hook_mutex); - if (!hooks_entry) { + if (!hooks_entry || hooks_entry->orig_ops != reg) { WARN(1, "nf_unregister_net_hook: hook not found!\n"); return; }