From patchwork Fri Feb 12 08:24:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 8288651 Return-Path: X-Original-To: patchwork-linux-block@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id A8F7A9F3CD for ; Fri, 12 Feb 2016 08:27:10 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 87507203DA for ; Fri, 12 Feb 2016 08:27:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3EBE12024F for ; Fri, 12 Feb 2016 08:27:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751636AbcBLI04 (ORCPT ); Fri, 12 Feb 2016 03:26:56 -0500 Received: from mail-pa0-f54.google.com ([209.85.220.54]:32955 "EHLO mail-pa0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751484AbcBLI0U convert rfc822-to-8bit (ORCPT ); Fri, 12 Feb 2016 03:26:20 -0500 Received: by mail-pa0-f54.google.com with SMTP id fl4so31469935pad.0; Fri, 12 Feb 2016 00:26:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references :organization:mime-version:content-type:content-transfer-encoding; bh=WVRcGECPiNimysF72/FyaPLEyb2wCjW1+8RFGRiTWfg=; b=n4ZWiJl0jvS/6ZNf6bcXZZMDLj9YRicBh7lETClt0LyFg85COchA9aDYQhPwyTt/1C HbUjkcidlWJNl1kLhmUd79qG74hV6KGHY+qGTbc3taYCGVtDwyOX917Tnj2xXtUoD0HS jwlaag/vC1QUZdP7nlPUf6keOR/lmVOM3kd9TYgk8q9E2L/WTnNhyA+fLNOucZsmhn7A fM6uGGomEVzuOmiGO9Mm18jJkq5kt2RD7st7bLW3RYqrnfiPcVpMeZ7CqBaQpfxhbwmU /ZvOwLo+uqptv5RJVood/NZ2AsoGXTApcppvtDVsmDs13VPCgC4xrl8C2Shqsr2bUoI4 SREw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-type :content-transfer-encoding; bh=WVRcGECPiNimysF72/FyaPLEyb2wCjW1+8RFGRiTWfg=; b=Ke++TUCiPKTgSe305nl31tnJZ2alfNgWDhHXw6GHJ1+xOYIXp/CZ3M6a3esCXADhRS fsvtjizEe49Exot/o28hqDpvmIGs5sFh2I4HivDLar0UU/Ie6ojG7bYr1Kvc7HuK3ufb O63JVrYuN4uunl/vZkCA25HZ4kOMLJA+6K1nDeVdWM4RHtfo9dSY3I/Wm6maxzJjs+Rr l7IR6otb+JH0fW++mb4pY5tRCDunQGo06rMoV9UM6cZ5wO/ZEuilJQwbwngJinYr/aZV NRYfdn6e2f286PBW5UR0+u3oya/EKFX6z2lMlA/6wVYnfFy5KGtTobDGRxynx17wZQzo sn8g== X-Gm-Message-State: AG10YOSGJGVvX7mtgElgvcQoAF0qzylnLmeJkzRPXjwSc9zN+JJK8Uyu1FIzSrq6OZ+Rtw== X-Received: by 10.66.102.104 with SMTP id fn8mr237313pab.129.1455265580231; Fri, 12 Feb 2016 00:26:20 -0800 (PST) Received: from tom-T450 ([116.251.208.106]) by smtp.gmail.com with ESMTPSA id cq4sm17417748pad.28.2016.02.12.00.26.17 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Fri, 12 Feb 2016 00:26:20 -0800 (PST) Date: Fri, 12 Feb 2016 16:24:35 +0800 From: Ming Lei To: Sasha Levin Cc: keith.busch@intel.com, Jens Axboe , Christoph Hellwig , jonathan.derrick@intel.com, LKML , linux-block@vger.kernel.org, tom.leiming@gmail.com Subject: Re: blk: accessing invalid memory with "blk-mq: dynamic h/w context count" Message-ID: <20160212162435.1e809790@tom-T450> In-Reply-To: <56BD7088.1020908@oracle.com> References: <56BD7088.1020908@oracle.com> Organization: Ming X-Mailer: Claws Mail 3.9.3 (GTK+ 2.24.23; x86_64-pc-linux-gnu) MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, 12 Feb 2016 00:41:28 -0500 Sasha Levin wrote: > Hi all, > > I've started seeing the following errors on boot: > > [6035791.296570] ================================================================== > [6035791.297467] BUG: KASAN: slab-out-of-bounds in loop_init_request+0x19c/0x1c0 at addr ffff880052e5c190 > [6035791.298355] Write of size 8 by task swapper/0/1 > [6035791.298842] ============================================================================= > [6035791.299751] BUG kmalloc-512 (Tainted: G W ): kasan: bad access detected > [6035791.300736] ----------------------------------------------------------------------------- > [6035791.300736] > [6035791.301696] Disabling lock debugging due to kernel taint > [6035791.302220] INFO: Slab 0xffffea00014b9700 objects=32 used=32 fp=0x (null) flags=0x1fffff80004080 > [6035791.303218] INFO: Object 0xffff880052e5c000 @offset=0 fp=0x (null) > [6035791.303218] > [6035791.304047] Object ffff880052e5c000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.304955] Object ffff880052e5c010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.305970] Object ffff880052e5c020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.306916] Object ffff880052e5c030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.307908] Object ffff880052e5c040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.308903] Object ffff880052e5c050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.309959] Object ffff880052e5c060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.310896] Object ffff880052e5c070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.311849] Object ffff880052e5c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.312784] Object ffff880052e5c090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.313734] Object ffff880052e5c0a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.314646] Object ffff880052e5c0b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.315567] Object ffff880052e5c0c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.316519] Object ffff880052e5c0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.317475] Object ffff880052e5c0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.318461] Object ffff880052e5c0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.319428] Object ffff880052e5c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.320548] Object ffff880052e5c110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.321680] Object ffff880052e5c120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.322585] Object ffff880052e5c130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.323587] Object ffff880052e5c140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.324574] Object ffff880052e5c150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.325505] Object ffff880052e5c160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.326449] Object ffff880052e5c170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.327412] Object ffff880052e5c180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.328329] Object ffff880052e5c190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.329200] Object ffff880052e5c1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.330117] Object ffff880052e5c1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.331000] Object ffff880052e5c1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.331949] Object ffff880052e5c1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.332888] Object ffff880052e5c1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.333886] Object ffff880052e5c1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [6035791.334813] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G B W 4.5.0-rc3-next-20160211-sasha-00028-g542d18e-dirty #2898 > [6035791.335884] 1ffff1000a714ed2 00000000534d57fe ffff8800538a7718 ffffffffa34d4a15 > [6035791.336796] ffffffff00000000 fffffbfff5eec534 0000000041b58ab3 ffffffffaefba520 > [6035791.337631] ffffffffa34d489f 00000000534d57fe ffff880184220000 ffffffffaefd813f > [6035791.338458] Call Trace: > [6035791.338756] dump_stack (lib/dump_stack.c:53) > [6035791.340573] print_trailer (mm/slub.c:661) > [6035791.341117] object_err (mm/slub.c:668) > [6035791.341738] kasan_report_error (include/linux/kasan.h:28 mm/kasan/report.c:170 mm/kasan/report.c:237) > [6035791.344327] __asan_report_store8_noabort (mm/kasan/report.c:259 mm/kasan/report.c:285) > [6035791.345775] loop_init_request (drivers/block/loop.c:1699) > [6035791.347753] blk_mq_realloc_hw_ctxs (block/blk-mq.c:1722 block/blk-mq.c:1981) > [6035791.351966] blk_mq_init_allocated_queue (block/blk-mq.c:2027) > [6035791.355528] blk_mq_init_queue (block/blk-mq.c:1944) > [6035791.356081] loop_add (drivers/block/loop.c:1749) > [6035791.358663] loop_init (drivers/block/loop.c:2006 (discriminator 3)) > [6035791.362708] do_one_initcall (init/main.c:788) > [6035791.363968] kernel_init_freeable (init/main.c:853 init/main.c:861 init/main.c:879 init/main.c:1004) > [6035791.366040] kernel_init (init/main.c:932) > [6035791.366573] ret_from_fork (arch/x86/entry/entry_64.S:383) > [6035791.367782] Memory state around the buggy address: > [6035791.368247] ffff880052e5c080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [6035791.368968] ffff880052e5c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fc fc > [6035791.369852] >ffff880052e5c180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > [6035791.370635] ^ > [6035791.371015] ffff880052e5c200: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > [6035791.371816] ffff880052e5c280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > > Bisection pointed to: > > commit 868f2f0b72068a097508b6e8870a8950fd8eb7ef > Author: Keith Busch > Date: Thu Dec 17 17:08:14 2015 -0700 > > blk-mq: dynamic h/w context count Hi Sasha, It should be about timing of setting q->mq_ops, and I believe the following patch may fix the issue, could you give a test? Thanks, --- From 299dfbd27a4ede53104608b07669041d202afe1f Mon Sep 17 00:00:00 2001 From: Ming Lei Date: Fri, 12 Feb 2016 15:27:00 +0800 Subject: [PATCH] blk-mq: mark request queue as mq asap Currently q->mq_ops is used widely to decide if the queue is mq or not, so we should set the 'flag' asap so that both block core and drivers can get the correct mq info. For example, commit 868f2f0b720(blk-mq: dynamic h/w context count) moves the hctx's initialization before setting q->mq_ops in blk_mq_init_allocated_queue(), then cause blk_alloc_flush_queue() to think the queue is non-mq and don't allocate command size for the per-hctx flush rq. This patches should fix the problem reported by Sasha. Cc: Keith Busch Reported-by: Sasha Levin Signed-off-by: Ming Lei --- block/blk-mq.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 645eb9e..f539a53 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2010,6 +2010,9 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set, struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, struct request_queue *q) { + /* mark the queue as mq asap */ + q->mq_ops = set->ops; + q->queue_ctx = alloc_percpu(struct blk_mq_ctx); if (!q->queue_ctx) return ERR_PTR(-ENOMEM); @@ -2032,7 +2035,6 @@ struct request_queue *blk_mq_init_allocated_queue(struct blk_mq_tag_set *set, q->nr_queues = nr_cpu_ids; - q->mq_ops = set->ops; q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT; if (!(set->flags & BLK_MQ_F_SG_MERGE))