From patchwork Thu Sep 8 17:33:29 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaohua Li X-Patchwork-Id: 9321843 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 04B7D60752 for ; Thu, 8 Sep 2016 17:33:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 018942997A for ; Thu, 8 Sep 2016 17:33:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA59C2997C; Thu, 8 Sep 2016 17:33:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAAA12997A for ; Thu, 8 Sep 2016 17:33:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753270AbcIHRd4 (ORCPT ); Thu, 8 Sep 2016 13:33:56 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:49647 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753269AbcIHRdy (ORCPT ); Thu, 8 Sep 2016 13:33:54 -0400 Received: from pps.filterd (m0001255.ppops.net [127.0.0.1]) by mx0b-00082601.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id u88HWQl4030019; Thu, 8 Sep 2016 10:33:48 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : in-reply-to; s=facebook; bh=NUgeE9sqkEFlAhkDkGELZ1dTXByUeVhrMKWava+e/QU=; b=WohXoh23DLDuhTVuGag88pCrO7XvphGqCjXhXI6NrMvGihG0cBA5zeu5XhSfrlVjJIL/ 5jT1r2HQivFTPGaWL6hXlBPZs7BLtD1vjmd11UJbEzordrZmauuMigw3DGxtvrzQBefe IdWFXndmQUHNJ/FrELofIYNzOc9usX0tYpI= Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0b-00082601.pphosted.com with ESMTP id 25bc2qr6by-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 08 Sep 2016 10:33:48 -0700 Received: from NAM01-BN3-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.34) with Microsoft SMTP Server (TLS) id 14.3.294.0; Thu, 8 Sep 2016 13:33:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=465kHEK5sLyI3H82FEYqSPHP04e8qmQRGa47Il9RgQk=; b=AabpI8o7HLNQfsbieTxQhBLoo3C39oZYPCm4rQhh8oCbjaG6QTyBZ83g1SM6FNadqCh8Lk/+iFdGqD8hkgyo0CV8E9tBlDVi+oCQxT/f/CDuDvb6SMUpXjRjKKIqc5jwkjbHGVPNzzJ2rEOpOO5xSpIgUBwPYHOQsuoa9LQuqZs= Received: from shli-mbp.local (2620:10d:c090:200::3:8d55) by CO1PR15MB0933.namprd15.prod.outlook.com (10.166.29.143) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.609.9; Thu, 8 Sep 2016 17:33:37 +0000 Date: Thu, 8 Sep 2016 10:33:29 -0700 From: Shaohua Li To: Jens Axboe CC: Stefan Priebe - Profihost AG , , Subject: Re: kernel 4.8-rc5 kernel BUG at block/blk-core.c:2032! Message-ID: <20160908173328.GA58334@shli-mbp.local> References: <56e78c6a-5996-d106-2952-d985ceca9e34@profihost.ag> <7ad7efe2-ef46-e1d5-bb0d-c7ed0e8d47db@kernel.dk> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <7ad7efe2-ef46-e1d5-bb0d-c7ed0e8d47db@kernel.dk> User-Agent: Mutt/1.6.1 (2016-04-27) X-Originating-IP: [2620:10d:c090:200::3:8d55] X-ClientProxiedBy: MWHPR04CA0041.namprd04.prod.outlook.com (10.172.163.27) To CO1PR15MB0933.namprd15.prod.outlook.com (10.166.29.143) X-MS-Office365-Filtering-Correlation-Id: 46d1cdce-1dd3-4d79-7adb-08d3d80e4485 X-Microsoft-Exchange-Diagnostics: 1; CO1PR15MB0933; 2:2EGnLDf2wW04rR4SZ95iyfoq4LgbkutbJvFNX8jY+/I4t3+16XlYbbEaqDKx8WDFWeU1L+gJKYBusdxY6AysJ9ElK9P0wyO8n1RQ1DwHcfKntNgPY00L3VDN+CFzBfWU8v2ZEzSNcYey3TODZ1j9TZUdvGXYPVQmVOko65o9ts7EJrLwtSf/elkalj3W/+Cx; 3:XCiIv/Bss0abg94CECVnIns/g1mNQinpxzCc153CwPVKnBkdQQRp8uYCLGuYD3USOQlE+q+Fupnycscde1z/DA0jShF/kZLW5w/e4BezKbPoHomgci7sr6baZxKw8C8y; 25:qie+Ypl8bqdfT3o5yZqxDPxZ6a4wAm3+Nr+DZZ2bepAI4zWKb2JRNYeCMLlnSiziLp4XNwIqPxealy02d7y6mqEch9NX8pJ/oLzXr+KLepdjobnvfXnUtH/fuhp0NNU12rp6suTYkwtrJumYHYUK1sfhQj22BxYX1fE80Vmu2Y2J7iFYPcn0clJ+2O4DXToo38tX1ScjsFdLmgRRz+FeIQULmVCU615hF242wgrrL9dMsLj3IUftEBk0SqolNvd8csb7pFyRIfUKzFUu5CbMUbx2JAtYbWZpabrKCdAxaMmSE/EO3ydKybTnknRNogeGS6iFoaJePv8h1otredBwkVigeMrq/Gw8uTUIx0VI/xDcdIMl+6DH/O6TvjrBVRLmVN5F05tf7ylLWTorVad9N9HwRjR6UkDGO8lqT8YflFI= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CO1PR15MB0933; X-Microsoft-Exchange-Diagnostics: 1; CO1PR15MB0933; 31:EHnd84qx+PG0VGD6uv/ZxfqlXmuS2X3BGmvE69R5bNOfGKCfVBr7+/xuo9vBEe/rxu2Bx5bENMe8a6Wtj/vuJdpQWOrWTcm94yGIsqELw8LueNfdGkt3lruwei3hTPLPwKr67hQyNAPIFJ+AMtr8yIRgWAmOZVYohyu5MFx6bFVuguAcGILTTgczXKvPvMRRykR1kCBWenwG6j9tnC28+QjL72DO0xOSUVZ11W1oJ+I=; 20:P6bCk12EnH2aD3mqLTw5xDKd230nmWDnhsyS6kNP8qUiHTe1TuHhvmvNC3i/GNh+yqKxGCTo55NnGYu4XcpUTXNXhYrQg7jz0rOriM/J65IU9giufxuWAbehRcLREbhFi/iUhsz0/m/8/tK6whPNv6t9HXOE8NWQGqB18NnoLNo=; 4:U1Nz7jGcOoPPC/QmGburc/cWP0ky81VcevzSrgevLFHL1rk+5AZjTOhpOO1NQ93sPJ/oXUKvb3YMTQqhZBI9lkr7NqW0W3bAiEP4a6d+K1enSCWk2CsipkqbFqamqeDgpunI0ZS5FaB4MFNs1WarBwJXnVjophSMD1ydl5zWrRauE7XAxUPngteqGz01QfS2qsH9agBvSyWwy2frlaePhHwNi4vgjF8dsj+/pGw7xuFkxA90Uk2pwcQ0g9EUkXxK2Brl/uvL/88jRxCybWPXua9+uNQeHew+WHOJ/YfTOkBpyB+3TP5BWAEUHU3Nhaps+cG5aEskDcyUWf9/cwy6cw0m8/RwMqYcz6a/rVnWJCeyRCzZz74lLTC/4ySEdhr1fROz+oVe4OheyMWvweUvRg== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040176)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:CO1PR15MB0933; BCL:0; PCL:0; RULEID:; SRVR:CO1PR15MB0933; X-Forefront-PRVS: 00594E8DBA X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(4630300001)(6009001)(7916002)(189002)(377454003)(24454002)(199003)(50466002)(2950100001)(4326007)(305945005)(226693001)(76176999)(54356999)(7846002)(50986999)(5660300001)(106356001)(4001350100001)(97736004)(189998001)(110136002)(105586002)(7736002)(101416001)(42186005)(46406003)(98436002)(77096005)(92566002)(33656002)(23726003)(97756001)(9686002)(83506001)(1076002)(47776003)(6116002)(8676002)(19580395003)(586003)(81156014)(2906002)(81166006)(86362001)(575784001)(68736007)(18370500001)(3826002); DIR:OUT; SFP:1102; SCL:1; SRVR:CO1PR15MB0933; H:shli-mbp.local; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: fb.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CO1PR15MB0933; 23:ljYTLPZTnCfoFvy++BGJP9osrg/a3Pt0sKwUmMaYm?= =?us-ascii?Q?ES0Q9+yTQBEUDXTjYnRI/QIvauqe4QfQfOeuy2WHZ0+S3m7jwwZ9OaG982mH?= =?us-ascii?Q?rMtG+GYqWNEsm9lKDyQ8A6vFR28boixr4mIxqCkUyQmN1E1d8Yv0iw9tGvhK?= =?us-ascii?Q?1LCRWY/isHlBRer0tsYH0NVMaZJU5uLMUNvjjULq45hGpRt07eQr6+BMi4Yu?= =?us-ascii?Q?PqJW7Ze77RLoONKKYaVo4Rtjmvi2BJ+dNcaANscDt3u6jxbSl9AwQY/Bi7Ji?= =?us-ascii?Q?0smvawuWGcgoE1vWeJ15L4+HezZ/TuuGMr0uVViq423jeOwvvjEEX8+mw3ga?= =?us-ascii?Q?ij0OP2QOEuHKvOXOw5pNt5Xmovo22jHroVc/kIr6VYehT+Nu7I18VvZ1CR8M?= =?us-ascii?Q?wLfZs6JPMkuGaszT5gQm5lyoNbA49QbScHRWhuo2RPEgSoI7uDxfxZURb8bO?= =?us-ascii?Q?yuePZiofutKX7bHr+KqL5nQDQsA6+78ms6zZ0MQzzyVkEtn3E/R8CzR+WBBX?= =?us-ascii?Q?bglIXhjKSPi598BAlWKTZHm78yn/I7wOhsiEQQJb5OF/B6KsWcofHTKIo02M?= =?us-ascii?Q?nqVb9t7Xd3IfSbv/dn8Hdy1A4VuqYPMdzqYxhi6fCWe6n1vqoXwO0N3aBtyV?= =?us-ascii?Q?jIvFrou0Oikjp/hYPBoocr0UaBC3xz4gSDl3Is6J+jLdr+fMFu7TvDoNizZw?= =?us-ascii?Q?yE3K1hbisfr632fIp+a2vnDc3HOkf2duLMFu2cY+YyRklHGUC77WmzTw2SEP?= =?us-ascii?Q?rFZpFGNsWFw8c3S/flVmhZRuVHs8hoMRYaQAqV2q4W83F9gRV2fcYSHKSneL?= =?us-ascii?Q?25cF9LWmPVBRj66cdj/CHAPzX6iXBDuOKovm287Mbp6AFQxOQ4y2SWcbM3Yc?= =?us-ascii?Q?M6uYOieeOqqlQAjI4wctzZp6mxRkBL2LlFn/h+yUFQNaC64A+FzY9GFudv38?= =?us-ascii?Q?CtQ1n+6Zc85kaW1Nv/tLEcVoidrHWNt/c40vVgY6gYdZkGQyoDJGcMaRp4we?= =?us-ascii?Q?lz7xUPH22dmJoNf7TwkV4Fu/t+JdzFRQFyjfuwNEWIQHJA6nUsPx8uealmRo?= =?us-ascii?Q?Z4QVI67NFKuDTHEePB1/0ouWF/7nVO0MAqoT1l59tzjcIl/VT9ccqhj4fsbI?= =?us-ascii?Q?yOjxPK5eNNiVXW9tPUUG4gv9eFS6s5/5d+bO6F7ZDJOhUh9XtUdle2xqJCZB?= =?us-ascii?Q?4J8Io/F6ZDswamwaguCFVJAvkBMYOABm9HtWFpJDlNFNleCm1Qn69ejMA=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1; CO1PR15MB0933; 6:2fcHOl1pkmlSxXQ2C5yptwt22t0Wi/ixHvZU5Tu3PQvfoyYHjAxtAuLwbPWlVQvo2POErZNw0ckTuSUbN0nVIYELPgyTcaY7pDSDr9ycEpplp7+TPgxplu3n/A43m/caS98yfn4Yzy6yoP/adGX+UVvNNthsp7ui2qWheuVi9/p3gIZinCikp1EZIN7qGJ8FfRZy89cvU/nlDHx+WEeoikqjHiXOcFO11mud8LQV0uXWroUs6v0yigwq8ObTCSMeX2uNtI0K2GQQWoFHvveZbxyntkbFJaQ96aZe/SAIwZQ=; 5:MZas3sWJpp90cFvkG7yTpiM5CrL1ocMh1VllWk6ocF0BNG4RfHEAQ4/642AoT7guSKvaneGnw+6kfZOckWcQcin54XkDrrDu2UBQunG3v0qNVI7S7cNDwRRFZbz3maDZFA9a5VTjhg6D7reADeRZjQ==; 24:nyMqvPlNLbaRPW3gyGA3pKp5K5N/2FUY1VoTDIZyFl8+VVzhtZds3HzanNKfyzR432DJcQY7x2NusCaynO+YMsIrRVDhcIRochQyO0R4a0U=; 7:pUZwI/xjjZkTFcZ4awhNV9RXFCsAqkI+0Um+qd5flsxLKOWruoSJiZTeSAOvlw2TsBVr2WlfxDPvt8x7M2YCzbp9sFzLUx7kKUJ5F63CzDx0cMQvq2OnMBW/dJd3fEzy4B2ickL9+3EtgaGBVGxB/881WI0bn1XXW4iZQnpZcY19FEclDqDNtHA3qap8zWJFmTk0z/83IfPyB2JpygX+RIcY+c+wcgAVSS7Mn9GnyVqfEMlFNb/tPtrkj4O3tpqt SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CO1PR15MB0933; 20:BtdY8p9KJgx3Rh3ouiKr95hch7OpO0KkWAeqMh+7F12oPgXfgWdbL1slS7mW29qQo8YlqJPx0SVFsXEDxrAEl37a/vm1Cf1o9uXPDJU+IM81ziRSRiXPiGB8i5RBiTNLBOLxh9AipgFZilXk41wn+6COQQWMb7fsk5ZYcHfd5/I= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2016 17:33:37.7940 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR15MB0933 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-09-08_08:, , signatures=0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Sep 08, 2016 at 10:16:59AM -0600, Jens Axboe wrote: > On 09/08/2016 02:23 AM, Stefan Priebe - Profihost AG wrote: > > Hi, > > > > while trying Kernel 4.8-rc5 my raid5 breaks every few minutes. > > > > Trace: > > ------------[ cut here ]------------ > > kernel BUG at block/blk-core.c:2032! > > invalid opcode: 0000 [#1] SMP > > Modules linked in: netconsole ipt_REJECT nf_reject_ipv4 xt_multiport > > iptable_filter ip_tables x_tables 8021q garp bonding sb_edac edac_core > > x86_pkg_temp_thermal coretemp kvm_intel kvm i2c_i801 irqbypass i2c_smbus > > ipmi_si crc32_pclmul i2c_core ghash_clmulni_intel shpchp ipmi_msghandler > > button loop fuse btrfs dm_mod raid10 raid0 multipath linear raid456 > > async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq > > raid1 md_mod sg sd_mod ixgbe i40e mdio usbhid ehci_pci ehci_hcd ahci > > usbcore ptp libahci usb_common megaraid_sas pps_core > > CPU: 8 PID: 1105 Comm: md0_raid5 Not tainted 4.8.0-rc5-00003-g3abda5c #2 > > Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c 02/18/2015 > > task: ffff97de5e1e0000 task.stack: ffff97de597a0000 > > RIP: 0010:[] [] > > generic_make_request+0x1c0/0x1d0 > > RSP: 0018:ffff97de597a3aa0 EFLAGS: 00010286 > > RAX: ffff97de5e1e0000 RBX: ffff97dd227e5030 RCX: 0000000000000000 > > RDX: ffffffffc0000001 RSI: 0000000000000001 RDI: ffff97de5e7d9db8 > > RBP: ffff97de597a3ad8 R08: 0000000000000008 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ffffffff > > R13: ffff97de5aa20c00 R14: 00000000000002f0 R15: ffff97e65dce0e00 > > FS: 0000000000000000(0000) GS:ffff97e67f200000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > CR2: 00007f0e4e1ec000 CR3: 0000000078c06000 CR4: 00000000001406e0Stack: > > ffff97de597a3b50 0000000000001000 0000000000000000 ffff97dd227e4c80 > > ffff97de5aa20c00 00000000000002f0 ffff97e65dce0e00 ffff97de597a3ba0 > > ffffffffc02595db ffffffffc025e04b 00000001597a3b01 0000000200000006 > > Call Trace: > > [] ops_run_io+0x3bb/0x990 [raid456] > > [] ? raid_run_ops+0xefb/0x1520 [raid456] > > [] handle_stripe+0x9a6/0x2280 [raid456] > > [] ? default_wake_function+0x12/0x20 > > [] ? autoremove_wake_function+0x12/0x40 > > [] handle_active_stripes.isra.54+0x193/0x4b0 [raid456] > > [] ? __release_stripe+0x15/0x20 [raid456] > > [] raid5d+0x4a9/0x740 [raid456] > > [] ? init_timer_key+0xa0/0xa0 > > [] md_thread+0x12b/0x130 [md_mod] > > [] ? wait_woken+0x90/0x90 > > [] ? find_pers+0x70/0x70 [md_mod] > > [] kthread+0xdb/0x100 > > [] ret_from_fork+0x1f/0x40 > > [] ? kthread_park+0x60/0x60 > > Code: bd 70 08 00 00 f0 49 83 ad 70 08 00 00 01 74 05 e9 5a ff ff ff 41 > > ff 95 80 08 00 00 e9 4e ff ff ff 48 c7 40 08 00 00 00 00 eb 8c <0f> 0b > > 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 > > RIP [] generic_make_request+0x1c0/0x1d0 > > RSP > > ---[ end trace 457dbe5e9cdd3473 ]--- > > CC'ing Shaohua - this is: > > BUG_ON(bio->bi_next); > > which doesn't look healthy. Hi Stefan, does below patch help? Looks there is a race condition introduced recently. --- To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index b95c54c..ee7fc37 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -2423,10 +2423,10 @@ static void raid5_end_read_request(struct bio * bi) } } rdev_dec_pending(rdev, conf->mddev); + bio_reset(bi); clear_bit(R5_LOCKED, &sh->dev[i].flags); set_bit(STRIPE_HANDLE, &sh->state); raid5_release_stripe(sh); - bio_reset(bi); } static void raid5_end_write_request(struct bio *bi) @@ -2498,6 +2498,7 @@ static void raid5_end_write_request(struct bio *bi) if (sh->batch_head && bi->bi_error && !replacement) set_bit(STRIPE_BATCH_ERR, &sh->batch_head->state); + bio_reset(bi); if (!test_and_clear_bit(R5_DOUBLE_LOCKED, &sh->dev[i].flags)) clear_bit(R5_LOCKED, &sh->dev[i].flags); set_bit(STRIPE_HANDLE, &sh->state); @@ -2505,7 +2506,6 @@ static void raid5_end_write_request(struct bio *bi) if (sh->batch_head && sh != sh->batch_head) raid5_release_stripe(sh->batch_head); - bio_reset(bi); } static void raid5_build_block(struct stripe_head *sh, int i, int previous)