| Message ID | 163043485613.1415.4979286233971984855.stgit@klimt.1015granger.net (mailing list archive) |
|---|---|
| Headers | show |
| Series | NFSD: Pull Read chunks in XDR decoders | expand |
On Tue, Aug 31, 2021 at 03:05:09PM -0400, Chuck Lever wrote: > Hi Bruce- > > Here is part of what we discussed recently about trying to align > pages in NFS WRITE requests so that splice can be used. This > series updates server-side RPC and svcrdma to ensure that an > aligned xdr_buf::pages array is presented to NFSD, which is then > converted into an aligned rq_vec for the VFS layer. Seems sensible to me. Do you have a git tree? It didn't apply cleanly to 5.14 when I tried, but I didn't stop to figure out why. > The next step would be to look at how to make the best use of the > aligned rq_vec. Have you done any performance comparison just with this? Doesn't seem like it should make a significant difference, but it might be interesting to check anyway. --b. > My naive thought is that where there is a PAGE_SIZE > entry in rq_vec and there is no page in the file's page cache at > that offset, the transport-provided page can be flipped into place. > Might work for replacing whole pages as well, but baby steps first. > > This series has been exercised a bit with both TCP and RDMA, but no > guarantees that it is completely bug-free. NFSv4 compounds with > multiple WRITE payloads on RDMA are treated like TCP: the RPC > message is contained in an unstructured stream of unaligned pages. > > Comments encouraged. > > --- > > Chuck Lever (6): > SUNRPC: Capture value of xdr_buf::page_base > SUNRPC: xdr_stream_subsegment() must handle non-zero page_bases > NFSD: Have legacy NFSD WRITE decoders use xdr_stream_subsegment() > SUNRPC: svc_fill_write_vector() must handle non-zero page_bases > NFSD: Add a transport hook for pulling argument payloads > svcrdma: Pull Read chunks in ->xpo_argument_payload > > > fs/nfsd/nfs3proc.c | 3 +- > fs/nfsd/nfs3xdr.c | 16 +-- > fs/nfsd/nfs4proc.c | 3 +- > fs/nfsd/nfs4xdr.c | 6 + > fs/nfsd/nfsproc.c | 3 +- > fs/nfsd/nfsxdr.c | 13 +-- > fs/nfsd/xdr.h | 2 +- > fs/nfsd/xdr3.h | 2 +- > include/linux/sunrpc/svc.h | 6 +- > include/linux/sunrpc/svc_rdma.h | 8 ++ > include/linux/sunrpc/svc_xprt.h | 3 + > include/trace/events/rpcrdma.h | 26 +++++ > include/trace/events/sunrpc.h | 20 +++- > net/sunrpc/svc.c | 38 +++++-- > net/sunrpc/svcsock.c | 8 ++ > net/sunrpc/xdr.c | 32 +++--- > net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 37 +++++- > net/sunrpc/xprtrdma/svc_rdma_rw.c | 139 ++++++++++++++++++++--- > net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + > 19 files changed, 292 insertions(+), 74 deletions(-) > > -- > Chuck Lever
> On Aug 31, 2021, at 4:42 PM, J. Bruce Fields <bfields@fieldses.org> wrote: > > On Tue, Aug 31, 2021 at 03:05:09PM -0400, Chuck Lever wrote: >> Hi Bruce- >> >> Here is part of what we discussed recently about trying to align >> pages in NFS WRITE requests so that splice can be used. This >> series updates server-side RPC and svcrdma to ensure that an >> aligned xdr_buf::pages array is presented to NFSD, which is then >> converted into an aligned rq_vec for the VFS layer. > > Seems sensible to me. > > Do you have a git tree? I don't yet, but can set up a topic branch somewhere where we're a little further along. > It didn't apply cleanly to 5.14 when I tried, > but I didn't stop to figure out why. It might apply to 5.15-pre now that nfsd-5.15 has been merged. >> The next step would be to look at how to make the best use of the >> aligned rq_vec. > > Have you done any performance comparison just with this? Not yet. I just got it behaving correctly with the usual tests. Baby steps, sir. > Doesn't seem like it should make a significant difference, but it might > be interesting to check anyway. I expect it to add a small amount of latency to NFS WRITEs, since RDMA Reads are done just a little later than before. > --b. > >> My naive thought is that where there is a PAGE_SIZE >> entry in rq_vec and there is no page in the file's page cache at >> that offset, the transport-provided page can be flipped into place. >> Might work for replacing whole pages as well, but baby steps first. >> >> This series has been exercised a bit with both TCP and RDMA, but no >> guarantees that it is completely bug-free. NFSv4 compounds with >> multiple WRITE payloads on RDMA are treated like TCP: the RPC >> message is contained in an unstructured stream of unaligned pages. >> >> Comments encouraged. >> >> --- >> >> Chuck Lever (6): >> SUNRPC: Capture value of xdr_buf::page_base >> SUNRPC: xdr_stream_subsegment() must handle non-zero page_bases >> NFSD: Have legacy NFSD WRITE decoders use xdr_stream_subsegment() >> SUNRPC: svc_fill_write_vector() must handle non-zero page_bases >> NFSD: Add a transport hook for pulling argument payloads >> svcrdma: Pull Read chunks in ->xpo_argument_payload >> >> >> fs/nfsd/nfs3proc.c | 3 +- >> fs/nfsd/nfs3xdr.c | 16 +-- >> fs/nfsd/nfs4proc.c | 3 +- >> fs/nfsd/nfs4xdr.c | 6 + >> fs/nfsd/nfsproc.c | 3 +- >> fs/nfsd/nfsxdr.c | 13 +-- >> fs/nfsd/xdr.h | 2 +- >> fs/nfsd/xdr3.h | 2 +- >> include/linux/sunrpc/svc.h | 6 +- >> include/linux/sunrpc/svc_rdma.h | 8 ++ >> include/linux/sunrpc/svc_xprt.h | 3 + >> include/trace/events/rpcrdma.h | 26 +++++ >> include/trace/events/sunrpc.h | 20 +++- >> net/sunrpc/svc.c | 38 +++++-- >> net/sunrpc/svcsock.c | 8 ++ >> net/sunrpc/xdr.c | 32 +++--- >> net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 37 +++++- >> net/sunrpc/xprtrdma/svc_rdma_rw.c | 139 ++++++++++++++++++++--- >> net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + >> 19 files changed, 292 insertions(+), 74 deletions(-) >> >> -- >> Chuck Lever -- Chuck Lever
Hi Bruce- Here is part of what we discussed recently about trying to align pages in NFS WRITE requests so that splice can be used. This series updates server-side RPC and svcrdma to ensure that an aligned xdr_buf::pages array is presented to NFSD, which is then converted into an aligned rq_vec for the VFS layer. The next step would be to look at how to make the best use of the aligned rq_vec. My naive thought is that where there is a PAGE_SIZE entry in rq_vec and there is no page in the file's page cache at that offset, the transport-provided page can be flipped into place. Might work for replacing whole pages as well, but baby steps first. This series has been exercised a bit with both TCP and RDMA, but no guarantees that it is completely bug-free. NFSv4 compounds with multiple WRITE payloads on RDMA are treated like TCP: the RPC message is contained in an unstructured stream of unaligned pages. Comments encouraged. --- Chuck Lever (6): SUNRPC: Capture value of xdr_buf::page_base SUNRPC: xdr_stream_subsegment() must handle non-zero page_bases NFSD: Have legacy NFSD WRITE decoders use xdr_stream_subsegment() SUNRPC: svc_fill_write_vector() must handle non-zero page_bases NFSD: Add a transport hook for pulling argument payloads svcrdma: Pull Read chunks in ->xpo_argument_payload fs/nfsd/nfs3proc.c | 3 +- fs/nfsd/nfs3xdr.c | 16 +-- fs/nfsd/nfs4proc.c | 3 +- fs/nfsd/nfs4xdr.c | 6 + fs/nfsd/nfsproc.c | 3 +- fs/nfsd/nfsxdr.c | 13 +-- fs/nfsd/xdr.h | 2 +- fs/nfsd/xdr3.h | 2 +- include/linux/sunrpc/svc.h | 6 +- include/linux/sunrpc/svc_rdma.h | 8 ++ include/linux/sunrpc/svc_xprt.h | 3 + include/trace/events/rpcrdma.h | 26 +++++ include/trace/events/sunrpc.h | 20 +++- net/sunrpc/svc.c | 38 +++++-- net/sunrpc/svcsock.c | 8 ++ net/sunrpc/xdr.c | 32 +++--- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 37 +++++- net/sunrpc/xprtrdma/svc_rdma_rw.c | 139 ++++++++++++++++++++--- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 19 files changed, 292 insertions(+), 74 deletions(-) -- Chuck Lever