From patchwork Tue Jan 17 13:00:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 9520837 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 065BE6020B for ; Tue, 17 Jan 2017 13:02:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14B2B28546 for ; Tue, 17 Jan 2017 13:02:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0941B2854A; Tue, 17 Jan 2017 13:02:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8262F28546 for ; Tue, 17 Jan 2017 13:02:16 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v0HD0wPa055403; Tue, 17 Jan 2017 08:00:59 -0500 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v0HD0uxd015351 for ; Tue, 17 Jan 2017 08:00:56 -0500 Received: from mx1.redhat.com (ext-mx08.extmail.prod.ext.phx2.redhat.com [10.5.110.32]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v0HD0uOM014221 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 17 Jan 2017 08:00:56 -0500 Received: from mail-pf0-f171.google.com (mail-pf0-f171.google.com [209.85.192.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CD943C00F216 for ; Tue, 17 Jan 2017 13:00:50 +0000 (UTC) Received: by mail-pf0-f171.google.com with SMTP id 189so62230668pfu.3 for ; Tue, 17 Jan 2017 05:00:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=s8DjBdTnJbCbrBoLy+Pkbdrf3I+HWd8oiOBWwqvJTmE=; b=Yklttdhjo+b5uc6c6/P5gLQ4DvIAch/0Gfd5BYQ8nZiSOaJ0KfqzuB+HkJ9vl9Ui08 PF6844ThYQE+K3C2R9RXYw/ppuh62wqxFZ2noIu9Yp8u4i412XkilE/JXpiHLlSUBPZ8 6YVnTozJ3VBrokyA4WtAOZbCR0HpC35mmkFhW1WsSjdX354dBzBEkHCwLSW9aJZJOk4g Q6CqGZiiDKYUS/aQPkrODjih3ARy7lAvVtof0gUxSHARo4xJ7GhujY3kHBvoxPa28RW+ sBvE15TtkHP9vcFfPqkTPkkEEAzoawfn8uw4GObrGf7T7AvOMwz+NTXylnrdZp8uobyG JEKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=s8DjBdTnJbCbrBoLy+Pkbdrf3I+HWd8oiOBWwqvJTmE=; b=koL1E/kQ8Jr+2BenbVMAHAm0kYkMWrrxZnpTs0Ii1jZLBCiAMAH1BeWy24YVOe+mTG zrgL7Cfj4V/r9mX8k4Xa5h9XGDOEUOObANrYwk87rNa7rnTvXFmAbgu2v4SUawXvlUtk to2K/NEgBLEFifWVEHqR7Xz3w2X9vlfeBww9T222ufhwXw7codLzgmAsRZ0DFsBKIvRm ksrDOKMCfFNvXG7x4nvGo4o/qulnOvOSVHWz+MFjC54ajmJq7AdpPSWJBk1QgmTcXoI/ V4+QLfZvnWeGWenjoi4d/uSThqDFc5ZBR49Zmj+aJiWhUe+0PyTlFmFiexmtFp2pz7fd nWmw== X-Gm-Message-State: AIkVDXJrZwFLkHMxjOo3lh84h88SW/m8sbvijYE6dWWI4duVZZjEBYDb9n20xA73e6KhIg== X-Received: by 10.99.158.18 with SMTP id s18mr46023832pgd.88.1484658049022; Tue, 17 Jan 2017 05:00:49 -0800 (PST) Received: from [192.168.102.217] (50-233-49-230-static.hfc.comcastbusiness.net. [50.233.49.230]) by smtp.gmail.com with ESMTPSA id q190sm55757059pfb.51.2017.01.17.05.00.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Jan 2017 05:00:48 -0800 (PST) To: Hannes Reinecke References: <55b21574-baf3-25d9-a920-299fa2d04d72@kernel.dk> From: Jens Axboe Message-ID: <10ffc7ea-564f-8eb0-ee00-5c1277c7efc3@kernel.dk> Date: Tue, 17 Jan 2017 05:00:47 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <55b21574-baf3-25d9-a920-299fa2d04d72@kernel.dk> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Tue, 17 Jan 2017 13:00:50 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Tue, 17 Jan 2017 13:00:50 +0000 (UTC) for IP:'209.85.192.171' DOMAIN:'mail-pf0-f171.google.com' HELO:'mail-pf0-f171.google.com' FROM:'axboe@kernel.dk' RCPT:'' X-RedHat-Spam-Score: -0.267 (BAYES_50, DCC_REPUT_00_12, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, RCVD_IN_SORBS_SPAM, SPF_PASS) 209.85.192.171 mail-pf0-f171.google.com 209.85.192.171 mail-pf0-f171.google.com X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Scanned-By: MIMEDefang 2.78 on 10.5.110.32 X-loop: dm-devel@redhat.com Cc: "linux-block@vger.kernel.org" , device-mapper development , Christoph Hellwig Subject: Re: [dm-devel] kernel oops with blk-mq-sched latest X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Virus-Scanned: ClamAV using ClamSMTP On 01/17/2017 04:47 AM, Jens Axboe wrote: > On 01/17/2017 12:57 AM, Hannes Reinecke wrote: >> Hi Jens, >> >> I gave your latest patchset from >> >> git.kernel.dk/linux-block blk-mq-sched >> >> I see a kernel oops when shutting down: >> >> [ 2132.708929] systemd-shutdown[1]: Detaching DM devices. >> [ 2132.965107] BUG: unable to handle kernel NULL pointer dereference at >> 00000000 >> 00000001 >> [ 2133.037182] IP: dd_merged_requests+0x6/0x60 >> [ 2133.077816] PGD 0 >> [ 2133.077818] >> [ 2133.113087] Oops: 0000 [#1] SMP >> [ list of modules removed ] >> [ 2133.925265] CPU: 20 PID: 1 Comm: systemd-shutdow Not tainted >> 4.10.0-rc4+ #543 >> [ 2133.990034] Hardware name: HP ProLiant DL380p Gen8, BIOS P70 09/18/2013 >> [ 2134.050522] task: ffff88042d614040 task.stack: ffffc90003150000 >> [ 2134.106915] RIP: 0010:dd_merged_requests+0x6/0x60 >> [ 2134.150593] RSP: 0018:ffffc90003153b18 EFLAGS: 00010002 >> [ 2134.198740] RAX: ffffffff81cc6de0 RBX: ffff8804296d5040 RCX: >> 0000000000000001 >> [ 2134.262708] RDX: 0000000000000001 RSI: 0000000000000001 RDI: >> ffff8804296d5040 >> [ 2134.326987] RBP: ffffc90003153b30 R08: 0000000000000000 R09: >> 0000000000000000 >> [ 2134.391054] R10: 0000000000000000 R11: 0001f8180001f815 R12: >> 0000000000000000 >> [ 2134.456095] R13: ffff8804296d5040 R14: ffff8804099801f0 R15: >> 0000000000000004 >> [ 2134.521196] FS: 00007fd64d3bf840(0000) GS:ffff88042f900000(0000) >> knlGS:0000000000000000 >> [ 2134.595178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 2134.648637] CR2: 0000000000000001 CR3: 000000081b892000 CR4: >> 00000000000406e0 >> [ 2134.713349] Call Trace: >> [ 2134.737168] ? elv_drain_elevator+0x29/0xa0 >> [ 2134.775821] __blk_drain_queue+0x52/0x1a0 >> [ 2134.812473] blk_queue_bypass_start+0x6e/0xa0 >> [ 2134.854009] blkcg_deactivate_policy+0x30/0xf0 >> [ 2134.894993] blk_throtl_exit+0x34/0x50 >> [ 2134.929450] blkcg_exit_queue+0x35/0x40 >> [ 2134.965089] blk_release_queue+0x33/0xe0 >> [ 2135.001364] kobject_cleanup+0x63/0x170 >> [ 2135.037412] kobject_put+0x25/0x50 >> [ 2135.068622] blk_cleanup_queue+0x198/0x260 >> [ 2135.107780] cleanup_mapped_device+0xb5/0xf0 [dm_mod] >> [ 2135.154741] __dm_destroy+0x1a5/0x290 [dm_mod] >> [ 2135.195009] dm_destroy+0x13/0x20 [dm_mod] >> [ 2135.232149] dev_remove+0xde/0x120 [dm_mod] >> [ 2135.270184] ? dev_suspend+0x210/0x210 [dm_mod] >> [ 2135.311478] ctl_ioctl+0x20b/0x510 [dm_mod] >> [ 2135.349680] ? terminate_walk+0xc3/0x140 >> [ 2135.385442] ? kmem_cache_free+0x10/0x260 >> [ 2135.422315] dm_ctl_ioctl+0x13/0x20 [dm_mod] >> >> Known issue? > > Try with this, looks like we're calling the old bypass path for the new > path, which is now triggering because q->elevator is true. Should have grepped, there's one more path that uses the old bypass path for q->mq_ops, potentially. This one handles that one, too. diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 8ba0af780e88..2630f64bed19 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1223,7 +1223,11 @@ int blkcg_activate_policy(struct request_queue *q, if (blkcg_policy_enabled(q, pol)) return 0; - blk_queue_bypass_start(q); + if (q->mq_ops) { + blk_mq_freeze_queue(q); + blk_mq_quiesce_queue(q); + } else + blk_queue_bypass_start(q); pd_prealloc: if (!pd_prealloc) { pd_prealloc = pol->pd_alloc_fn(GFP_KERNEL, q->node); @@ -1261,7 +1265,10 @@ int blkcg_activate_policy(struct request_queue *q, spin_unlock_irq(q->queue_lock); out_bypass_end: - blk_queue_bypass_end(q); + if (q->mq_ops) + blk_mq_unfreeze_queue(q); + else + blk_queue_bypass_end(q); if (pd_prealloc) pol->pd_free_fn(pd_prealloc); return ret; @@ -1284,7 +1291,12 @@ void blkcg_deactivate_policy(struct request_queue *q, if (!blkcg_policy_enabled(q, pol)) return; - blk_queue_bypass_start(q); + if (q->mq_ops) { + blk_mq_freeze_queue(q); + blk_mq_quiesce_queue(q); + } else + blk_queue_bypass_start(q); + spin_lock_irq(q->queue_lock); __clear_bit(pol->plid, q->blkcg_pols); @@ -1304,7 +1316,11 @@ void blkcg_deactivate_policy(struct request_queue *q, } spin_unlock_irq(q->queue_lock); - blk_queue_bypass_end(q); + + if (q->mq_ops) + blk_mq_unfreeze_queue(q); + else + blk_queue_bypass_end(q); } EXPORT_SYMBOL_GPL(blkcg_deactivate_policy); diff --git a/block/elevator.c b/block/elevator.c index d14cb87e6564..464372840774 100644 --- a/block/elevator.c +++ b/block/elevator.c @@ -613,6 +613,9 @@ void elv_drain_elevator(struct request_queue *q) { static int printed; + if (WARN_ON_ONCE(q->mq_ops)) + return; + lockdep_assert_held(q->queue_lock); while (q->elevator->type->ops.sq.elevator_dispatch_fn(q, 1))