| Message ID | 4DD42BCF.8070909@sandia.gov (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, 18 May 2011, Jim Schutt wrote: > Sage Weil wrote: > > > > > I pushed a patch to the msgr_race branch that catches all four cases (I > > think). Does the fix make sense given what you saw? > > Sorry, I haven't completed much testing; it took me a while > to figure out the fix needs this: > > diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c > index 9c0a9bd..b140dd3 100644 > --- a/net/ceph/messenger.c > +++ b/net/ceph/messenger.c > @@ -2013,6 +2013,7 @@ done: > mutex_unlock(&con->mutex); > done_unlocked: > con->ops->put(con); > + return; > > fault: > mutex_unlock(&con->mutex); > > > Still testing.... Good catch. Let us know how it goes! sge -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Sage, Sage Weil wrote: > On Wed, 18 May 2011, Jim Schutt wrote: >> Sage Weil wrote: >> >>> I pushed a patch to the msgr_race branch that catches all four cases (I >>> think). Does the fix make sense given what you saw? >> Sorry, I haven't completed much testing; it took me a while >> to figure out the fix needs this: >> >> diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c >> index 9c0a9bd..b140dd3 100644 >> --- a/net/ceph/messenger.c >> +++ b/net/ceph/messenger.c >> @@ -2013,6 +2013,7 @@ done: >> mutex_unlock(&con->mutex); >> done_unlocked: >> con->ops->put(con); >> + return; >> >> fault: >> mutex_unlock(&con->mutex); >> >> >> Still testing.... > > Good catch. Let us know how it goes! I've been testing commit a30413af363 from your msgr_race branch, and I think it is ready. In my testing of it I've found none of the signs that I recognize as indicators of the race we were trying to fix. However, with that issue fixed I'm now running into a different type of stall under a heavy write load. If the load is heavy enough to trigger that issue where heartbeat processing is delayed, and an osd is wrongly marked down, then I see a flurry of messages with bad tags. So far I can't generate a high enough load with much client debugging enabled, but what I have done is turn on client debugging after I see the "wrongly marked me down" from ceph -w, and the bad message tags in the osd logs. When I do that I see some clients with a few messages queued up to send. Sending of the first message starts as expected, but before it completes it stalls, as though the socket buffer fills up and isn't being emptied. Then the message times out, the socket is closed/reopened, and the partially-sent message is resent from the beginning. And then sending stalls again. I'll let you know when I've got some logs that suggest what the problem might be... FWIW, on the server side I'm running the stable branch (commit b6cccc741a3). -- Jim > > sge > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c index 9c0a9bd..b140dd3 100644 --- a/net/ceph/messenger.c +++ b/net/ceph/messenger.c @@ -2013,6 +2013,7 @@ done: mutex_unlock(&con->mutex); done_unlocked: con->ops->put(con); + return; fault: mutex_unlock(&con->mutex);