From patchwork Thu Jul 27 12:54:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matan Barak X-Patchwork-Id: 9866773 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B756E60382 for ; Thu, 27 Jul 2017 12:54:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A93A028803 for ; Thu, 27 Jul 2017 12:54:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9DFBD2881B; Thu, 27 Jul 2017 12:54:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8F0428803 for ; Thu, 27 Jul 2017 12:54:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751481AbdG0MyJ (ORCPT ); Thu, 27 Jul 2017 08:54:09 -0400 Received: from mail-ua0-f195.google.com ([209.85.217.195]:34315 "EHLO mail-ua0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751452AbdG0MyJ (ORCPT ); Thu, 27 Jul 2017 08:54:09 -0400 Received: by mail-ua0-f195.google.com with SMTP id k43so12485592uaf.1 for ; Thu, 27 Jul 2017 05:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dev-mellanox-co-il.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=T+BylNpHXhMVhooEJrLYG03v5OFhOOueLq3gRrLCXCA=; b=u7qb8pqIvXyLxeYGdLR5W5biM6alYjBXu4YpCCM0ut19+GWN2ImKp+UpSqQQ9C6lZy iwW6pzbzRMRhCQx24KamSi02l5DH4asbuL/UrmGTGsm4TRrnVnB3Qn58vutjNoqQU7KR vpTvUt2hIrJpWqyaJa3mRV9esH6r6QHOxBdJqoRIwMTpInP5rjr7JgZwsbLRPdz7KeQh UAzSCMiVMztJCsgUW4Q4Rm6pL7+/o6N2bta6V4YO4Uk1sfzhvj+ATKunJdjpjW6XyS8J iPl3u+dCkHsuEVv8Rmlt9tZvt+GoA6e7f/L+iNyzdLPYHqOFq0pN5cSw7XSopsCutzrV wf6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=T+BylNpHXhMVhooEJrLYG03v5OFhOOueLq3gRrLCXCA=; b=IUMbozJtlUbja5DzrvouJA8SyYQSnwqW3pJ95wQV+qM7uMA+fJL6JOP0iwBUZSud09 /6K6hMcorl4F5gIjn3FsjENSo5BoucHIL63ntBsmaw7X9t2WkfSS8WmLEI0XX3GlElHZ UvNMmRJz+Qj0HYZVbDTa66fWdiO1/c8F/xy07alHxi3OLln//BYwAlqMiY76zd1SIu1w xlzNelbLVCGF4bVhgAxRJ+xHvyghu3KMw/nXyoxwlVnn6uvFtc+2l0Kk3vuaerPKgf85 mLJhhTwygoH2QSe5ClKL88fX/b+gcfFU2UDNiVW4Thb8Qd2oFji28VishG/XMYmCzkkX LJjg== X-Gm-Message-State: AIVw112lI/D0sR3SM+z9njQNKHXoZ9vxn7MPKsSoMC6Og9QoBBIgsh8X Nx0EkLUlK9kFGRpfgu/cgr7cIqOmzHrVt9k= X-Received: by 10.31.89.195 with SMTP id n186mr2547491vkb.5.1501160048258; Thu, 27 Jul 2017 05:54:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.37.38 with HTTP; Thu, 27 Jul 2017 05:54:07 -0700 (PDT) In-Reply-To: References: <20170724211606.GA1705@obsidianresearch.com> From: Matan Barak Date: Thu, 27 Jul 2017 15:54:07 +0300 Message-ID: Subject: Re: Kernel oops To: Jason Gunthorpe Cc: Doug Ledford , linux-rdma , Yishai Hadas Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Thu, Jul 27, 2017 at 2:46 PM, Matan Barak wrote: > On Tue, Jul 25, 2017 at 12:16 AM, Jason Gunthorpe > wrote: >> Matan, >> >> I suspect your reworking series broke hot removal, I get a kernel oops >> when removing modules on 4.13-rc2. >> >> I think it is some kind of race with uverbs being removed concurrently >> with user apps closing uverbs. In this case I removed umad first which >> caused systemd to forcibly shutdown srp_daemon, which happened >> concurrently with the rmmod of ib_uverbs. >> > > Hi, > > Thanks for informing :) > We'll try to debug that, but I expect we'll only get to that by end of > next week. > > Thanks, > Matan > Digging a bit, we found a fix that might be related to this issue. I would be happy if you could try that and report if it solved this problem. We plan to send it soon. commit 1d4ecbf034193f000fe6686586c40ab4b2a95da1 Author: Yishai Hadas Date: Thu Jul 27 15:49:00 2017 +0200 IB/uverbs: Fix device cleanup Uverbs device should be cleaned up only when there is no potential usage of. As part of ib_uverbs_remove_one which might be triggered upon reset flow the device reference count is decreased as expected and leave the final cleanup to the FDs that were opened. Current code increases reference count upon opening a new command FD and decreases it upon closing the file. The event FD is opened internally and rely on the command FD by taking on it a reference count. In case that the command FD was closed and just later the event FD we may ensure that the device resources as of srcu are still alive as they are still in use. Fixing the above by moving the reference count decreasing to the place where the command FD is really freed instead of doing that when it was just closed. Signed-off-by: Yishai Hadas Reviewed-by: Matan Barak if (file->ucontext) { @@ -940,7 +940,6 @@ static int ib_uverbs_close(struct inode *inode, struct file *filp) ib_uverbs_release_async_event_file); kref_put(&file->ref, ib_uverbs_release_file); - kobject_put(&dev->kobj); return 0; } >> [ 50.797421] general protection fault: 0000 [#1] SMP >> [ 50.798400] Modules linked in: ib_srp scsi_transport_srp scsi_mod rdma_cm ib_umad ib_cm ib_uverbs iw_cm mlx4_core ib_core [last unloaded: mlx4_ib] >> [ 50.800946] CPU: 0 PID: 235 Comm: srp_daemon Not tainted 4.13.0-rc2+ #2 >> [ 50.802178] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014 >> [ 50.803970] task: ffff88022e504b00 task.stack: ffffc90000178000 >> [ 50.805163] RIP: 0010:ib_uverbs_release_file+0x29/0x90 [ib_uverbs] >> [ 50.806428] RSP: 0018:ffffc9000017bd90 EFLAGS: 00010202 >> [ 50.807521] RAX: 0000000000000001 RBX: ffff88022dec56c0 RCX: 0000000000000001 >> [ 50.808898] RDX: 732f74696e752f31 RSI: ffff88022dec56c0 RDI: ffff88022cccc000 >> [ 50.810349] RBP: ffffc9000017bda0 R08: 000000002dec5801 R09: 0000000180150013 >> [ 50.811830] R10: ffffc9000017bd80 R11: ffff88022afe5200 R12: ffff88022dec56c0 >> [ 50.813260] R13: ffff88022dec56e8 R14: ffff88022dec5f70 R15: ffff88022edaa020 >> [ 50.814830] FS: 00007fb54d287700(0000) GS:ffff880236e00000(0000) knlGS:0000000000000000 >> [ 50.816401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 50.817602] CR2: 00007fb54bc4c8f8 CR3: 000000022dcd9000 CR4: 00000000000406b0 >> [ 50.818991] Call Trace: >> [ 50.819487] uverbs_close_fd+0x5f/0xa0 [ib_uverbs] >> [ 50.820416] ib_uverbs_comp_event_close+0xa4/0xb0 [ib_uverbs] >> [ 50.821539] __fput+0xd4/0x1d0 >> [ 50.822202] ____fput+0x9/0x10 >> [ 50.822890] task_work_run+0x79/0xa0 >> [ 50.823676] do_exit+0x362/0xa90 >> [ 50.824309] ? __do_page_fault+0x203/0x430 >> [ 50.825140] do_group_exit+0x42/0xb0 >> [ 50.825908] SyS_exit_group+0xf/0x10 >> [ 50.826630] entry_SYSCALL_64_fastpath+0x1a/0xa5 >> [ 50.827591] RIP: 0033:0x7fb54c73bb98 >> [ 50.828340] RSP: 002b:00007ffec1987788 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 >> [ 50.829900] RAX: ffffffffffffffda RBX: 0000006aa1fe2dc0 RCX: 00007fb54c73bb98 >> [ 50.831303] RDX: 0000000000000000 RSI: 000000000000003c RDI: 0000000000000000 >> [ 50.832793] RBP: 00007ffec1987780 R08: 00000000000000e7 R09: ffffffffffffff98 >> [ 50.834288] R10: 00007fb54a446640 R11: 0000000000000246 R12: 00007fb54be4dc50 >> [ 50.835802] R13: 00000000ffffffff R14: 00007ffec1987680 R15: 0000000000000006 >> [ 50.837254] Code: 1f 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 47 48 48 8d b8 10 01 00 00 e8 d4 f8 02 e1 48 8b 7b 48 48 8b 57 30 48 85 d2 74 0a <48> 83 ba b0 02 00 00 00 74 42 89 c6 48 81 c7 10 01 00 00 e8 df >> [ 50.841156] RIP: ib_uverbs_release_file+0x29/0x90 [ib_uverbs] RSP: ffffc9000017bd90 >> [ 50.842723] ---[ end trace 68785d98b53d9203 ]--- >> >> Jason Reviewed-by: Jason Gunthorpe Tested-by: Jason Gunthorpe --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c index a88d223..cb1729a 100644 --- a/drivers/infiniband/core/uverbs_main.c +++ b/drivers/infiniband/core/uverbs_main.c @@ -251,6 +251,7 @@ void ib_uverbs_release_file(struct kref *ref) if (atomic_dec_and_test(&file->device->refcount)) ib_uverbs_comp_dev(file->device); + kobject_put(&file->device->kobj); kfree(file); } @@ -918,7 +919,6 @@ static int ib_uverbs_open(struct inode *inode, struct file *filp) static int ib_uverbs_close(struct inode *inode, struct file *filp) { struct ib_uverbs_file *file = filp->private_data; - struct ib_uverbs_device *dev = file->device; mutex_lock(&file->cleanup_mutex);