| Message ID | 20170622141000.9899-2-leon@kernel.org (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
On Thu, Jun 22, 2017 at 05:10:00PM +0300, Leon Romanovsky wrote: > From: yonatanc <yonatanc@mellanox.com> > > The RXE coupled with dummy device causes to the kernel panic attached > below. The panic happens when ib_register_device tries to set dma_mask > by accessing a NULLed parent device. > > The RXE does not actually use DMA, so we can set the dma_mask > to architecture value. > > [16240.199689] RIP: 0010:ib_register_device+0x468/0x5a0 [ib_core] > [16240.205289] RSP: 0018:ffffc9000220fc10 EFLAGS: 00010246 > [16240.209909] RAX: 0000000000000024 RBX: ffff880220d1a2a8 RCX: 0000000000000000 > [16240.212244] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 > [16240.214385] RBP: ffffc9000220fcb0 R08: 0000000000000000 R09: 000000000000023f > [16240.254465] R10: 0000000000000007 R11: 0000000000000000 R12: 0000000000000000 > [16240.259467] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880220d1a2a8 > [16240.263314] FS: 00007fd8ecca0740(0000) GS:ffff8802364c0000(0000) knlGS:0000000000000000 > [16240.267292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [16240.273503] CR2: 0000000000000218 CR3: 00000002253ba000 CR4: 00000000000006e0 > [16240.277066] Call Trace: > [16240.281836] ? __kmalloc+0x26f/0x280 > [16240.286596] rxe_register_device+0x297/0x300 [rdma_rxe] > [16240.291377] rxe_add+0x535/0x5b0 [rdma_rxe] > [16240.297586] rxe_net_add+0x3e/0xc0 [rdma_rxe] > [16240.302375] rxe_param_set_add+0x65/0x144 [rdma_rxe] > [16240.307769] param_attr_store+0x68/0xd0 > [16240.311640] module_attr_store+0x1d/0x30 > [16240.316421] sysfs_kf_write+0x3a/0x50 > [16240.317802] kernfs_fop_write+0xff/0x180 > [16240.322989] __vfs_write+0x37/0x140 > [16240.328164] ? handle_mm_fault+0xce/0x240 > [16240.333340] vfs_write+0xb2/0x1b0 > [16240.335013] SyS_write+0x55/0xc0 > [16240.340632] entry_SYSCALL_64_fastpath+0x1a/0xa9 > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > Signed-off-by: Yonatan Cohen <yonatanc@mellanox.com> > Reviewed-by: Moni Shoua <monis@mellanox.com> > Signed-off-by: Leon Romanovsky <leon@kernel.org> > --- Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
Hi Yonatan and Leon, In one of my servers i got kernel oops also in ib_register_device when using dummy device (macvtap) with rxe so was blindly hoping this patch would solve it but is is not. Crash is in alloc_name somewhere in the "list_for_each_entry" loop, i think first line of it. Anyway, steps that i'm doing are: $ ip link add link eth0 name macvtap3 type macvtap mode bridge $ modprobe ib_core ib_umad rdma_ucm ib_uverbs rdma_rxe $ echo eth0 > /sys/module/rdma_rxe/parameters/add $ echo macvtap3 > /sys/module/rdma_rxe/parameters/add At this point the system crash. I'm using 4.12.0-rc6. This is 100% reproduced. Interesting thing is that i'm unable to reproduce it on my workstation. See the below kernel oops: BUG: unable to handle kernel paging request at ffffffffa073b6db [159135.410160] IP: report_bug+0x87/0x110 [159135.454889] PGD 1c0c067 [159135.454890] P4D 1c0c067 [159135.486112] PUD 1c0d063 [159135.517334] PMD c381c5067 [159135.548554] PTE 8000000c42ec7161 [159135.581852] [159135.640138] Oops: 0003 [#1] SMP [159135.678635] Modules linked in: crc32_generic(E) crc32_pclmul(E) rdma_rxe(E) udp_tunnel(E) ip6_udp_tunnel(E) ib_ipoib(E) rdma_ucm(E) ib_ucm(E) ib_uverbs(E) ib_umad(E) rdma_cm(E) ib_cm(E) iw_cm(E) mlx4_ib(E) ib_core(E) mlx4_en(E) mlx4_core(E) rds_tcp(E) rds(E) xt_REDIRECT(E) nf_nat_redirect(E) xt_nat(E) iptable_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack(E) iptable_filter(E) ip_tables(E) kvm_intel(E) kvm(E) irqbypass(E) macvtap(E) tap(E) macvlan(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) nfs(E) fscache(E) lockd(E) grace(E) sunrpc(E) bnx2fc(E) cnic(E) uio(E) fcoe(E) libfcoe(E) libfc(E) 8021q(E) scsi_transport_fc(E) mrp(E) garp(E) stp(E) llc(E) configfs(E) iTCO_wdt(E) iTCO_vendor_support(E) pcspkr(E) ipmi_ssif(E) ipmi_si(E) ipmi_msghandler(E) i2c_i801(E) lpc_ich(E) [159136.531229] mfd_core(E) ioatdma(E) i7core_edac(E) sg(E) acpi_cpufreq(E) igb(E) dca(E) i2c_algo_bit(E) i2c_core(E) ext4(E) mbcache(E) fscrypto(E) jbd2(E) sd_mod(E) ahci(E) libahci(E) ipv6(E) crc_ccitt(E) ptp(E) pps_core(E) megaraid_sas(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) [last unloaded: mlx4_core] [159136.866231] CPU: 3 PID: 3533 Comm: bash Tainted: G E 4.12.0-rc6.master.20170625.ol6.x86_64 #1 [159136.982848] Hardware name: Oracle Corporation SUN FIRE X4170 M2 SERVER /ASSY,MOTHERBOARD,X4170, BIOS 08140109 12/10/2014 [159137.121282] task: ffff881843225200 task.stack: ffffc9000e160000 [159137.193126] RIP: 0010:report_bug+0x87/0x110 [159137.244188] RSP: 0018:ffffc9000e163938 EFLAGS: 00010202 [159137.307715] RAX: 0000000000000001 RBX: ffffffffa071d4e1 RCX: 0000000000000907 [159137.394202] RDX: ffffffffa073b6d1 RSI: 0000000000000000 RDI: ffffffffa071d4e1 [159137.480687] RBP: ffffc9000e163958 R08: ffffffffa073cf80 R09: ffffc9000e163908 [159137.567175] R10: ffffc9000e1638d8 R11: 00000000000008c4 R12: 000000000000015a [159137.653660] R13: ffffffffa07376f8 R14: ffffc9000e163ac8 R15: ffff881843225200 [159137.740147] FS: 00007fe99f793700(0000) GS:ffff880c4fac0000(0000) knlGS:0000000000000000 [159137.838065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [159137.907828] CR2: ffffffffa073b6db CR3: 000000184199c000 CR4: 00000000000006e0 [159137.994315] Call Trace: [159138.024611] fixup_bug+0x2e/0x50 [159138.064251] do_trap+0x13f/0x190 [159138.103893] do_error_trap+0xbd/0x100 [159138.148745] ? ib_register_device+0x391/0x3a0 [ib_core] [159138.212282] ? kmalloc_order_trace+0x34/0xc0 [159138.264392] ? __kmalloc+0x1cd/0x1e0 [159138.308185] ? ttwu_do_activate+0x87/0xa0 [159138.357174] do_invalid_op+0x20/0x30 [159138.400968] invalid_op+0x1e/0x30 [159138.441654] RIP: 0010:ib_register_device+0x391/0x3a0 [ib_core] [159138.512454] RSP: 0018:ffffc9000e163b78 EFLAGS: 00010246 [159138.575987] RAX: 0000000000000000 RBX: ffff880c3f8ed000 RCX: 0000000000000000 [159138.662473] RDX: ffffffffa03d3050 RSI: 0000000000000000 RDI: ffff880c3f8ed000 [159138.748959] RBP: ffffc9000e163be8 R08: ffff880c4fadf0e0 R09: ffff880c42cfa360 [159138.835446] R10: ffffc9000e163718 R11: 0000000000000000 R12: 00000000000005dc [159138.921933] R13: 0000000000000009 R14: ffff8818431f38e0 R15: ffff881843af1a60 [159139.008430] rxe_register_device+0x315/0x3a0 [rdma_rxe] [159139.071963] rxe_add+0x64/0x70 [rdma_rxe] [159139.120950] ? dev_get_by_name_rcu+0x76/0xa0 [159139.173054] rxe_net_add+0x45/0xd0 [rdma_rxe] [159139.226193] ? _raw_spin_unlock_bh+0x1e/0x20 [159139.278299] rxe_param_set_add+0xb5/0x1b0 [rdma_rxe] [159139.338718] ? path_to_nameidata+0x40/0x60 [159139.388752] param_attr_store+0x64/0x90 [159139.435659] module_attr_store+0x25/0x30 [159139.483610] sysfs_kf_write+0x3e/0x40 [159139.528441] kernfs_fop_write+0x113/0x1b0 [159139.577430] __vfs_write+0x38/0xe0 [159139.619147] ? filp_close+0x65/0x90 [159139.661906] ? __getnstimeofday64+0x45/0xe0 [159139.712974] ? do_dup2+0x99/0xe0 [159139.752614] ? __sb_start_write+0x5e/0xc0 [159139.801602] vfs_write+0xc1/0x130 [159139.842280] ? __fdget+0x13/0x20 [159139.881916] SyS_write+0x56/0xc0 [159139.921557] do_syscall_64+0x7a/0x230 [159139.966390] ? do_page_fault+0x37/0x90 [159140.012261] entry_SYSCALL64_slow_path+0x25/0x25 Yuval On Thu, Jun 22, 2017 at 05:10:00PM +0300, Leon Romanovsky wrote: > From: yonatanc <yonatanc@mellanox.com> > > The RXE coupled with dummy device causes to the kernel panic attached > below. The panic happens when ib_register_device tries to set dma_mask > by accessing a NULLed parent device. > > The RXE does not actually use DMA, so we can set the dma_mask > to architecture value. > > [16240.199689] RIP: 0010:ib_register_device+0x468/0x5a0 [ib_core] > [16240.205289] RSP: 0018:ffffc9000220fc10 EFLAGS: 00010246 > [16240.209909] RAX: 0000000000000024 RBX: ffff880220d1a2a8 RCX: 0000000000000000 > [16240.212244] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009 > [16240.214385] RBP: ffffc9000220fcb0 R08: 0000000000000000 R09: 000000000000023f > [16240.254465] R10: 0000000000000007 R11: 0000000000000000 R12: 0000000000000000 > [16240.259467] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880220d1a2a8 > [16240.263314] FS: 00007fd8ecca0740(0000) GS:ffff8802364c0000(0000) knlGS:0000000000000000 > [16240.267292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [16240.273503] CR2: 0000000000000218 CR3: 00000002253ba000 CR4: 00000000000006e0 > [16240.277066] Call Trace: > [16240.281836] ? __kmalloc+0x26f/0x280 > [16240.286596] rxe_register_device+0x297/0x300 [rdma_rxe] > [16240.291377] rxe_add+0x535/0x5b0 [rdma_rxe] > [16240.297586] rxe_net_add+0x3e/0xc0 [rdma_rxe] > [16240.302375] rxe_param_set_add+0x65/0x144 [rdma_rxe] > [16240.307769] param_attr_store+0x68/0xd0 > [16240.311640] module_attr_store+0x1d/0x30 > [16240.316421] sysfs_kf_write+0x3a/0x50 > [16240.317802] kernfs_fop_write+0xff/0x180 > [16240.322989] __vfs_write+0x37/0x140 > [16240.328164] ? handle_mm_fault+0xce/0x240 > [16240.333340] vfs_write+0xb2/0x1b0 > [16240.335013] SyS_write+0x55/0xc0 > [16240.340632] entry_SYSCALL_64_fastpath+0x1a/0xa9 > > Fixes: 8700e3e7c485 ("Soft RoCE driver") > Signed-off-by: Yonatan Cohen <yonatanc@mellanox.com> > Reviewed-by: Moni Shoua <monis@mellanox.com> > Signed-off-by: Leon Romanovsky <leon@kernel.org> > --- > drivers/infiniband/sw/rxe/rxe_verbs.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c > index 83d709e74dfb..70fd060e30a7 100644 > --- a/drivers/infiniband/sw/rxe/rxe_verbs.c > +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c > @@ -1245,6 +1245,8 @@ int rxe_register_device(struct rxe_dev *rxe) > addrconf_addr_eui48((unsigned char *)&dev->node_guid, > rxe->ndev->dev_addr); > dev->dev.dma_ops = &dma_virt_ops; > + dma_coerce_mask_and_coherent(&dev->dev, > + dma_get_required_mask(dev->dev.parent)); > > dev->uverbs_abi_ver = RXE_UVERBS_ABI_VERSION; > dev->uverbs_cmd_mask = BIT_ULL(IB_USER_VERBS_CMD_GET_CONTEXT) > -- > 2.13.1 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Jun 27, 2017 at 1:04 PM, Yuval Shaia <yuval.shaia@oracle.com> wrote: > Hi Yonatan and Leon, > In one of my servers i got kernel oops also in ib_register_device when > using dummy device (macvtap) with rxe so was blindly hoping this patch > would solve it but is is not. > > Crash is in alloc_name somewhere in the "list_for_each_entry" loop, i think > first line of it. > > Anyway, steps that i'm doing are: > $ ip link add link eth0 name macvtap3 type macvtap mode bridge > $ modprobe ib_core ib_umad rdma_ucm ib_uverbs rdma_rxe > $ echo eth0 > /sys/module/rdma_rxe/parameters/add > $ echo macvtap3 > /sys/module/rdma_rxe/parameters/add > At this point the system crash. > > I'm using 4.12.0-rc6. > This is 100% reproduced. > Interesting thing is that i'm unable to reproduce it on my workstation. > thanks for the report Yuval You said that it happens in only one of your servers but not in your workstation, right? Did you try to reproduce with the above scenario on different servers with the same configuration? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 83d709e74dfb..70fd060e30a7 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1245,6 +1245,8 @@ int rxe_register_device(struct rxe_dev *rxe) addrconf_addr_eui48((unsigned char *)&dev->node_guid, rxe->ndev->dev_addr); dev->dev.dma_ops = &dma_virt_ops; + dma_coerce_mask_and_coherent(&dev->dev, + dma_get_required_mask(dev->dev.parent)); dev->uverbs_abi_ver = RXE_UVERBS_ABI_VERSION; dev->uverbs_cmd_mask = BIT_ULL(IB_USER_VERBS_CMD_GET_CONTEXT)