| Message ID | 1421957007-720-1-git-send-email-isubramanian@apm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Thu, 2015-01-22 at 12:03 -0800, Iyappan Subramanian wrote: > This patch fixes the following kernel crash, > > WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c() > Call trace: > > Software writes poison data into the descriptor bytes[15:8] and upon > receiving the interrupt, if those bytes are overwritten by the hardware with > the valid data, software also reads bytes[7:0] and executes receive/tx > completion logic. > > If the CPU executes the above two reads in out of order fashion, then the > bytes[7:0] will have older data and causing the kernel panic. We have to > force the order of the reads and thus this patch introduces read memory > barrier between these reads. > > Signed-off-by: Iyappan Subramanian <isubramanian@apm.com> > Signed-off-by: Keyur Chudgar <kchudgar@apm.com> > Tested-by: Mark Langsdorf <mlangsdo@redhat.com> > --- > drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c > index 83a5028..3622cdb 100644 > --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c > +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c > @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring, > if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc))) > break; > > + /* read fpqnum field after dataaddr field */ > + smp_rmb(); > if (is_rx_desc(raw_desc)) > ret = xgene_enet_rx_frame(ring, raw_desc); > else Reading your changelog, it looks like you need a plain rmb() here.
On Thu, Jan 22, 2015 at 2:50 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Thu, 2015-01-22 at 12:03 -0800, Iyappan Subramanian wrote: >> This patch fixes the following kernel crash, >> >> WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c() >> Call trace: > >> >> Software writes poison data into the descriptor bytes[15:8] and upon >> receiving the interrupt, if those bytes are overwritten by the hardware with >> the valid data, software also reads bytes[7:0] and executes receive/tx >> completion logic. >> >> If the CPU executes the above two reads in out of order fashion, then the >> bytes[7:0] will have older data and causing the kernel panic. We have to >> force the order of the reads and thus this patch introduces read memory >> barrier between these reads. >> >> Signed-off-by: Iyappan Subramanian <isubramanian@apm.com> >> Signed-off-by: Keyur Chudgar <kchudgar@apm.com> >> Tested-by: Mark Langsdorf <mlangsdo@redhat.com> >> --- >> drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c >> index 83a5028..3622cdb 100644 >> --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c >> +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c >> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring, >> if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc))) >> break; >> >> + /* read fpqnum field after dataaddr field */ >> + smp_rmb(); >> if (is_rx_desc(raw_desc)) >> ret = xgene_enet_rx_frame(ring, raw_desc); >> else > > Reading your changelog, it looks like you need a plain rmb() here. rmb() translates into dsb, which in arm64 serializes everything including instructions and thus expensive compared to dmb. Do you see any issue with smp_rmb() (which translates into dmb) ? > > >
On Mon, 2015-01-26 at 13:12 -0800, Iyappan Subramanian wrote: > On Thu, Jan 22, 2015 at 2:50 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > On Thu, 2015-01-22 at 12:03 -0800, Iyappan Subramanian wrote: > >> This patch fixes the following kernel crash, > >> > >> WARNING: CPU: 2 PID: 0 at net/ipv4/tcp_input.c:3079 tcp_clean_rtx_queue+0x658/0x80c() > >> Call trace: > > > >> > >> Software writes poison data into the descriptor bytes[15:8] and upon > >> receiving the interrupt, if those bytes are overwritten by the hardware with > >> the valid data, software also reads bytes[7:0] and executes receive/tx > >> completion logic. > >> > >> If the CPU executes the above two reads in out of order fashion, then the > >> bytes[7:0] will have older data and causing the kernel panic. We have to > >> force the order of the reads and thus this patch introduces read memory > >> barrier between these reads. > >> > >> Signed-off-by: Iyappan Subramanian <isubramanian@apm.com> > >> Signed-off-by: Keyur Chudgar <kchudgar@apm.com> > >> Tested-by: Mark Langsdorf <mlangsdo@redhat.com> > >> --- > >> drivers/net/ethernet/apm/xgene/xgene_enet_main.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c > >> index 83a5028..3622cdb 100644 > >> --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c > >> +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c > >> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring, > >> if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc))) > >> break; > >> > >> + /* read fpqnum field after dataaddr field */ > >> + smp_rmb(); > >> if (is_rx_desc(raw_desc)) > >> ret = xgene_enet_rx_frame(ring, raw_desc); > >> else > > > > Reading your changelog, it looks like you need a plain rmb() here. > > rmb() translates into dsb, which in arm64 serializes everything > including instructions and thus expensive compared to dmb. > > Do you see any issue with smp_rmb() (which translates into dmb) ? What happens if you compile a kernel with CONFIG_SMP=n ? Most drivers in drivers/net use rmb() in this case, not smp_rmb() or barrier()
On Mon, 2015-01-26 at 13:27 -0800, Eric Dumazet wrote: > What happens if you compile a kernel with CONFIG_SMP=n ? > > > Most drivers in drivers/net use rmb() in this case, not smp_rmb() or > barrier() Note that dma_rmb() was recently added as well.
From: Iyappan Subramanian <isubramanian@apm.com> Date: Mon, 26 Jan 2015 13:12:23 -0800 >>> @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring, >>> if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc))) >>> break; >>> >>> + /* read fpqnum field after dataaddr field */ >>> + smp_rmb(); >>> if (is_rx_desc(raw_desc)) >>> ret = xgene_enet_rx_frame(ring, raw_desc); >>> else >> >> Reading your changelog, it looks like you need a plain rmb() here. > > rmb() translates into dsb, which in arm64 serializes everything > including instructions and thus expensive compared to dmb. > > Do you see any issue with smp_rmb() (which translates into dmb) ? smp_rmb() is not appropriate. You're not serializing accesses between two cpus, you're serializing the cpu with the device.
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c index 83a5028..3622cdb 100644 --- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c +++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c @@ -369,6 +369,8 @@ static int xgene_enet_process_ring(struct xgene_enet_desc_ring *ring, if (unlikely(xgene_enet_is_desc_slot_empty(raw_desc))) break; + /* read fpqnum field after dataaddr field */ + smp_rmb(); if (is_rx_desc(raw_desc)) ret = xgene_enet_rx_frame(ring, raw_desc); else