| Message ID | 161402344429.1980160.4798557236979159924.stgit@firesoul (mailing list archive) |
|---|---|
| State | RFC |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [RFC,net-next] mlx5: fix for crash on net-next | expand |
| Context | Check | Description |
|---|---|---|
| netdev/cover_letter | success | Link |
| netdev/fixes_present | success | Link |
| netdev/patch_count | success | Link |
| netdev/tree_selection | success | Clearly marked for net-next |
| netdev/subject_prefix | success | Link |
| netdev/cc_maintainers | warning | 2 maintainers not CCed: leon@kernel.org linux-rdma@vger.kernel.org |
| netdev/source_inline | success | Was 0 now: 0 |
| netdev/verify_signedoff | success | Link |
| netdev/module_param | success | Was 0 now: 0 |
| netdev/build_32bit | success | Errors and warnings before: 0 this patch: 0 |
| netdev/kdoc | success | Errors and warnings before: 0 this patch: 0 |
| netdev/verify_fixes | success | Link |
| netdev/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 32 lines checked |
| netdev/build_allmodconfig_warn | success | Errors and warnings before: 0 this patch: 0 |
| netdev/header_inline | success | Link |
| netdev/stable | success | Stable not CCed |
On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote: > Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17') > > There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c > where it seems that num_tc_x_num_ch can become zero in a modulo operation: > > if (unlikely(txq_ix >= num_tc_x_num_ch)) > txq_ix %= num_tc_x_num_ch; > > I think error were introduced in: > - 214baf22870c ("net/mlx5e: Support HTB offload") > > The modulo operation was introduced in: > - 145e5637d941 ("net/mlx5e: Add TX PTP port object support") > > The crash looks like this: > > [ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI > [ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575 > [ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016 > [ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core] > [ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f > [ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297 > [ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007 > [ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740 > [ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]--- > > This is an RFC, because I don't think this is a proper fix, but > at least it allows me to boot with mlx5 driver enabled. > > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> > --- Hi Jesper, Thanks for your report and patch. We'll check it and post the proper fix. I wonder what's special in your configuration / use case, as I'm not aware of such failure. Can you please share more info about the reproduction? Regards, Tariq
On Tue, 23 Feb 2021 10:28:35 +0200 Tariq Toukan <ttoukan.linux@gmail.com> wrote: > On 2/22/2021 9:50 PM, Jesper Dangaard Brouer wrote: > > Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17') > > > > There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c > > where it seems that num_tc_x_num_ch can become zero in a modulo operation: > > > > if (unlikely(txq_ix >= num_tc_x_num_ch)) > > txq_ix %= num_tc_x_num_ch; > > > > I think error were introduced in: > > - 214baf22870c ("net/mlx5e: Support HTB offload") > > > > The modulo operation was introduced in: > > - 145e5637d941 ("net/mlx5e: Add TX PTP port object support") > > > > The crash looks like this: > > > > [ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI > > [ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575 > > [ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016 > > [ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core] > > [ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f > > [ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297 > > [ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007 > > [ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740 > > [ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]--- > > > > This is an RFC, because I don't think this is a proper fix, but > > at least it allows me to boot with mlx5 driver enabled. > > > > Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> > > --- > > Hi Jesper, > > Thanks for your report and patch. > > We'll check it and post the proper fix. > I wonder what's special in your configuration / use case, as I'm not > aware of such failure. > Can you please share more info about the reproduction? The machine simply crash under boot, that is how I reproduced. You should look at why: num_tc_x_num_ch can be zero in: u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb, struct net_device *sb_dev) { [...] num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch); Here are some info on the mlx5 NIC I'm using: $ ethtool -i mlx5p1 driver: mlx5_core version: 5.11.0-net-next-alloc_pages_bul firmware-version: 16.23.1020 (MT_0000000009) expansion-rom-version: bus-info: 0000:03:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes 03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] Subsystem: Mellanox Technologies Device 0002 Physical Slot: 0 Flags: bus master, fast devsel, latency 0, IRQ 26 Memory at f0000000 (64-bit, prefetchable) [size=32M] Expansion ROM at fba00000 [disabled] [size=1M] Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [40] Power Management version 3 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [1c0] Secondary PCI Express Capabilities: [230] Access Control Services Capabilities: [320] Lane Margining at the Receiver <?> Capabilities: [370] Physical Layer 16.0 GT/s <?> Capabilities: [420] Data Link Feature <?> Kernel driver in use: mlx5_core Kernel modules: mlx5_core 03:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] Subsystem: Mellanox Technologies Device 0002 Physical Slot: 0 Flags: bus master, fast devsel, latency 0, IRQ 86 Memory at ee000000 (64-bit, prefetchable) [size=32M] Expansion ROM at fb900000 [disabled] [size=1M] Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [40] Power Management version 3 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [230] Access Control Services Capabilities: [420] Data Link Feature <?> Kernel driver in use: mlx5_core Kernel modules: mlx5_core
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c index bdbffe484fce..4dc26c1f2b53 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c @@ -131,6 +131,7 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb, int ch_ix; /* Sync with mlx5e_update_num_tc_x_num_ch - avoid refetching. */ + // XXX this reading of num_tc_x_num_ch looks strange num_tc_x_num_ch = READ_ONCE(priv->num_tc_x_num_ch); if (unlikely(dev->real_num_tx_queues > num_tc_x_num_ch)) { /* Order maj_id before defcls - pairs with mlx5e_htb_root_add. */ @@ -142,19 +143,21 @@ u16 mlx5e_select_queue(struct net_device *dev, struct sk_buff *skb, return txq_ix; } - if (unlikely(priv->channels.port_ptp)) + txq_ix = netdev_pick_tx(dev, skb, NULL); + + if (unlikely(priv->channels.port_ptp)) { if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) && mlx5e_use_ptpsq(skb)) return mlx5e_select_ptpsq(dev, skb); - txq_ix = netdev_pick_tx(dev, skb, NULL); /* Fix netdev_pick_tx() not to choose ptp_channel and HTB txqs. * If they are selected, switch to regular queues. * Driver to select these queues only at mlx5e_select_ptpsq() * and mlx5e_select_htb_queue(). */ - if (unlikely(txq_ix >= num_tc_x_num_ch)) - txq_ix %= num_tc_x_num_ch; + if (unlikely(txq_ix >= num_tc_x_num_ch)) + txq_ix %= num_tc_x_num_ch; + } } else { txq_ix = netdev_pick_tx(dev, skb, NULL); }
Net-next at commit d310ec03a34e ("Merge tag 'perf-core-2021-02-17') There is a divide error in drivers/net/ethernet/mellanox/mlx5/core/en_tx.c where it seems that num_tc_x_num_ch can become zero in a modulo operation: if (unlikely(txq_ix >= num_tc_x_num_ch)) txq_ix %= num_tc_x_num_ch; I think error were introduced in: - 214baf22870c ("net/mlx5e: Support HTB offload") The modulo operation was introduced in: - 145e5637d941 ("net/mlx5e: Add TX PTP port object support") The crash looks like this: [ 12.112849] divide error: 0000 [#1] PREEMPT SMP PTI [ 12.117727] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-net-next+ #575 [ 12.124677] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016 [ 12.132149] RIP: 0010:mlx5e_select_queue+0xd5/0x1e0 [mlx5_core] [ 12.138110] Code: 85 c0 75 2e 48 83 bb 08 57 00 00 00 75 5b 31 d2 48 89 ee 48 89 df e8 ba 3e 54 e1 0f b7 d0 41 39 d4 0f 8f 6b ff ff ff 89 d0 99 <41> f7 fc e9 60 ff ff ff 8b 96 8c 00 00 00 89 d1 c1 e9 10 39 c1 0f [ 12.156849] RSP: 0018:ffffc900001c0c10 EFLAGS: 00010297 [ 12.162065] RAX: 0000000000000004 RBX: ffff88810ff00000 RCX: 0000000000000007 [ 12.169188] RDX: 0000000000000000 RSI: ffff888107016400 RDI: ffff8881008dc740 [ 12.176313] RBP: ffff888107016400 R08: 0000000000000000 R09: 000000ncing: Fatal exception in interrupt ]--- This is an RFC, because I don't think this is a proper fix, but at least it allows me to boot with mlx5 driver enabled. Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com> --- drivers/net/ethernet/mellanox/mlx5/core/en_tx.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)