| Message ID | 20200529181440.2510.45116.stgit@manet.1015granger.net (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v1] NFS: Fix direct WRITE throughput regression | expand |
On Fri, 2020-05-29 at 14:14 -0400, Chuck Lever wrote: > I measured a 50% throughput regression for large direct writes. > > The observed on-the-wire behavior is that the client sends every > NFS WRITE twice: once as an UNSTABLE WRITE plus a COMMIT, and once > as a FILE_SYNC WRITE. > > This is because the nfs_write_match_verf() check in > nfs_direct_commit_complete() fails for every WRITE. > > Buffered writes use nfs_write_completion(), which sets req->wb_verf > correctly. Direct writes use nfs_direct_write_completion(), which > does not set req->wb_verf at all. This leaves req->wb_verf set to > all zeroes for every direct WRITE, and thus > nfs_direct_commit_completion() always sets > NFS_ODIRECT_RESCHED_WRITES. > > This fix appears to restore nearly all of the lost performance. > > Fixes: 1f28476dcb98 ("NFS: Fix O_DIRECT commit verifier handling") > Signed-off-by: Chuck Lever <chuck.lever@oracle.com> > --- > fs/nfs/direct.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c > index a57e7c72c7f4..d49b1d197908 100644 > --- a/fs/nfs/direct.c > +++ b/fs/nfs/direct.c > @@ -731,6 +731,8 @@ static void nfs_direct_write_completion(struct > nfs_pgio_header *hdr) > nfs_list_remove_request(req); > if (request_commit) { > kref_get(&req->wb_kref); > + memcpy(&req->wb_verf, &hdr->verf.verifier, > + sizeof(req->wb_verf)); > nfs_mark_request_commit(req, hdr->lseg, &cinfo, > hdr->ds_commit_idx); > } > Thanks Chuck! That's a no-brainer... Sorry for the screwup. Anna, are you taking this one?
diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index a57e7c72c7f4..d49b1d197908 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -731,6 +731,8 @@ static void nfs_direct_write_completion(struct nfs_pgio_header *hdr) nfs_list_remove_request(req); if (request_commit) { kref_get(&req->wb_kref); + memcpy(&req->wb_verf, &hdr->verf.verifier, + sizeof(req->wb_verf)); nfs_mark_request_commit(req, hdr->lseg, &cinfo, hdr->ds_commit_idx); }
I measured a 50% throughput regression for large direct writes. The observed on-the-wire behavior is that the client sends every NFS WRITE twice: once as an UNSTABLE WRITE plus a COMMIT, and once as a FILE_SYNC WRITE. This is because the nfs_write_match_verf() check in nfs_direct_commit_complete() fails for every WRITE. Buffered writes use nfs_write_completion(), which sets req->wb_verf correctly. Direct writes use nfs_direct_write_completion(), which does not set req->wb_verf at all. This leaves req->wb_verf set to all zeroes for every direct WRITE, and thus nfs_direct_commit_completion() always sets NFS_ODIRECT_RESCHED_WRITES. This fix appears to restore nearly all of the lost performance. Fixes: 1f28476dcb98 ("NFS: Fix O_DIRECT commit verifier handling") Signed-off-by: Chuck Lever <chuck.lever@oracle.com> --- fs/nfs/direct.c | 2 ++ 1 file changed, 2 insertions(+)