| Message ID | 20191009170721.2978.128.stgit@manet.1015granger.net (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v1,1/6] xprtrdma: Add unique trace points for posting Local Invalidate WRs | expand |
Hi- > On Oct 9, 2019, at 1:07 PM, Chuck Lever <chuck.lever@oracle.com> wrote: > > When adding frwr_unmap_async way back when, I re-used the existing > trace_xprtrdma_post_send() trace point to record the return code > of ib_post_send. > > Unfortunately there are some cases where re-using that trace point > causes a crash. Instead, construct a trace point specific to posting > Local Invalidate WRs that will always be safe to use in that context, > and will act as a trace log eye-catcher for Local Invalidation. I forgot to add a cover letter to this series. The first patch fixes a crasher that shows up when the rpcrdma trace class is enabled. The remaining patches address issues with how the RPC/RDMA client handles MRs. See individual patch descriptions for details. > Fixes: 847568942f93 ("xprtrdma: Remove fr_state") > Fixes: d8099feda483 ("xprtrdma: Reduce context switching due ... ") > Signed-off-by: Chuck Lever <chuck.lever@oracle.com> > Tested-by: Bill Baker <bill.baker@oracle.com> > --- > include/trace/events/rpcrdma.h | 25 +++++++++++++++++++++++++ > net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- > 2 files changed, 27 insertions(+), 2 deletions(-) > > diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h > index 9dd7680..31681cb 100644 > --- a/include/trace/events/rpcrdma.h > +++ b/include/trace/events/rpcrdma.h > @@ -735,6 +735,31 @@ > ) > ); > > +TRACE_EVENT(xprtrdma_post_linv, > + TP_PROTO( > + const struct rpcrdma_req *req, > + int status > + ), > + > + TP_ARGS(req, status), > + > + TP_STRUCT__entry( > + __field(const void *, req) > + __field(int, status) > + __field(u32, xid) > + ), > + > + TP_fast_assign( > + __entry->req = req; > + __entry->status = status; > + __entry->xid = be32_to_cpu(req->rl_slot.rq_xid); > + ), > + > + TP_printk("req=%p xid=0x%08x status=%d", > + __entry->req, __entry->xid, __entry->status > + ) > +); > + > /** > ** Completion events > **/ > diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c > index 30065a2..9901a81 100644 > --- a/net/sunrpc/xprtrdma/frwr_ops.c > +++ b/net/sunrpc/xprtrdma/frwr_ops.c > @@ -570,7 +570,6 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) > */ > bad_wr = NULL; > rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); > - trace_xprtrdma_post_send(req, rc); > > /* The final LOCAL_INV WR in the chain is supposed to > * do the wake. If it was never posted, the wake will > @@ -583,6 +582,7 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) > > /* Recycle MRs in the LOCAL_INV chain that did not get posted. > */ > + trace_xprtrdma_post_linv(req, rc); > while (bad_wr) { > frwr = container_of(bad_wr, struct rpcrdma_frwr, > fr_invwr); > @@ -673,12 +673,12 @@ void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) > */ > bad_wr = NULL; > rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); > - trace_xprtrdma_post_send(req, rc); > if (!rc) > return; > > /* Recycle MRs in the LOCAL_INV chain that did not get posted. > */ > + trace_xprtrdma_post_linv(req, rc); > while (bad_wr) { > frwr = container_of(bad_wr, struct rpcrdma_frwr, fr_invwr); > mr = container_of(frwr, struct rpcrdma_mr, frwr); > -- Chuck Lever
diff --git a/include/trace/events/rpcrdma.h b/include/trace/events/rpcrdma.h index 9dd7680..31681cb 100644 --- a/include/trace/events/rpcrdma.h +++ b/include/trace/events/rpcrdma.h @@ -735,6 +735,31 @@ ) ); +TRACE_EVENT(xprtrdma_post_linv, + TP_PROTO( + const struct rpcrdma_req *req, + int status + ), + + TP_ARGS(req, status), + + TP_STRUCT__entry( + __field(const void *, req) + __field(int, status) + __field(u32, xid) + ), + + TP_fast_assign( + __entry->req = req; + __entry->status = status; + __entry->xid = be32_to_cpu(req->rl_slot.rq_xid); + ), + + TP_printk("req=%p xid=0x%08x status=%d", + __entry->req, __entry->xid, __entry->status + ) +); + /** ** Completion events **/ diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 30065a2..9901a81 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -570,7 +570,6 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) */ bad_wr = NULL; rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); - trace_xprtrdma_post_send(req, rc); /* The final LOCAL_INV WR in the chain is supposed to * do the wake. If it was never posted, the wake will @@ -583,6 +582,7 @@ void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) /* Recycle MRs in the LOCAL_INV chain that did not get posted. */ + trace_xprtrdma_post_linv(req, rc); while (bad_wr) { frwr = container_of(bad_wr, struct rpcrdma_frwr, fr_invwr); @@ -673,12 +673,12 @@ void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req) */ bad_wr = NULL; rc = ib_post_send(r_xprt->rx_ia.ri_id->qp, first, &bad_wr); - trace_xprtrdma_post_send(req, rc); if (!rc) return; /* Recycle MRs in the LOCAL_INV chain that did not get posted. */ + trace_xprtrdma_post_linv(req, rc); while (bad_wr) { frwr = container_of(bad_wr, struct rpcrdma_frwr, fr_invwr); mr = container_of(frwr, struct rpcrdma_mr, frwr);
When adding frwr_unmap_async way back when, I re-used the existing trace_xprtrdma_post_send() trace point to record the return code of ib_post_send. Unfortunately there are some cases where re-using that trace point causes a crash. Instead, construct a trace point specific to posting Local Invalidate WRs that will always be safe to use in that context, and will act as a trace log eye-catcher for Local Invalidation. Fixes: 847568942f93 ("xprtrdma: Remove fr_state") Fixes: d8099feda483 ("xprtrdma: Reduce context switching due ... ") Signed-off-by: Chuck Lever <chuck.lever@oracle.com> Tested-by: Bill Baker <bill.baker@oracle.com> --- include/trace/events/rpcrdma.h | 25 +++++++++++++++++++++++++ net/sunrpc/xprtrdma/frwr_ops.c | 4 ++-- 2 files changed, 27 insertions(+), 2 deletions(-)