| Message ID | 20190708192352.12614-1-olga.kornievskaia@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | server-side support for "inter" SSC copy | expand |
Thanks for resending. What's the status of Amir's series? I guess I've been using that as an excuse to put off reviewing these, but I really should anyway.... --b. On Mon, Jul 08, 2019 at 03:23:44PM -0400, Olga Kornievskaia wrote: > This patch series adds support for NFSv4.2 copy offload feature > allowing copy between two different NFS servers. > > This functionality depends on the VFS ability to support generic > copy_file_range() where a copy is done between an NFS file and > a local file system. This is on top of Amir's VFS generic copy > offload series. > > This feature is enabled by the kernel module parameter -- > inter_copy_offload_enable -- and by default is disabled. There is > also a kernel compile configuration of NFSD_V4_2_INTER_SSC that > adds dependency on the NFS client side functions called from the > server. > > These patches work on top of existing async intra copy offload > patches. For the "inter" SSC, the implementation only supports > asynchronous inter copy. > > On the source server, upon receiving a COPY_NOTIFY, it generate a > unique stateid that's kept in the global list. Upon receiving a READ > with a stateid, the code checks the normal list of open stateid and > now additionally, it'll check the copy state list as well before > deciding to either fail with BAD_STATEID or find one that matches. > The stored stateid is only valid to be used for the first time > with a choosen lease period (90s currently). When the source server > received an OFFLOAD_CANCEL, it will remove the stateid from the > global list. Otherwise, the copy stateid is removed upon the removal > of its "parent" stateid (open/lock/delegation stateid). > > On the destination server, upon receiving a COPY request, the server > establishes the necessary clientid/session with the source server. > It calls into the NFS client code to establish the necessary > open stateid, filehandle, file description (without doing an NFS open). > Then the server calls into the copy_file_range() to preform the copy > where the source file will issue NFS READs and then do local file > system writes (this depends on the VFS ability to do cross device > copy_file_range(). > > v4: > --- allowing for synchronous inter server-to-server copy > --- added missing offload_cancel on the source server > > Already presented numbers for performance improvement for large > file transfer but here are times for copying linux kernel tree > (which is mostly small files): > -- regular cp 6m1s (intra) > -- copy offload cp 4m11s (intra) > -- benefit of using copy offload with small copies using sync copy > -- regular cp 6m9s (inter) > -- copy offload cp 6m3s (inter) > -- same performance as traditional as for most it fallback to traditional > copy offload > > Olga Kornievskaia (8): > NFSD fill-in netloc4 structure > NFSD add ca_source_server<> to COPY > NFSD return nfs4_stid in nfs4_preprocess_stateid_op > NFSD add COPY_NOTIFY operation > NFSD check stateids against copy stateids > NFSD generalize nfsd4_compound_state flag names > NFSD: allow inter server COPY to have a STALE source server fh > NFSD add nfs4 inter ssc to nfsd4_copy > > fs/nfsd/Kconfig | 10 ++ > fs/nfsd/nfs4proc.c | 434 +++++++++++++++++++++++++++++++++++++++++++++++----- > fs/nfsd/nfs4state.c | 135 ++++++++++++++-- > fs/nfsd/nfs4xdr.c | 172 ++++++++++++++++++++- > fs/nfsd/nfsd.h | 32 ++++ > fs/nfsd/nfsfh.h | 5 +- > fs/nfsd/nfssvc.c | 6 + > fs/nfsd/state.h | 25 ++- > fs/nfsd/xdr4.h | 37 ++++- > 9 files changed, 790 insertions(+), 66 deletions(-) > > -- > 1.8.3.1
Amir's patches have been in the linux-next xfs tree and will go into 5.3. I would like for NFS patches to go into 5.4 (I'm assuming hoping for 5.3 is unrealistic). On Mon, Jul 8, 2019 at 11:53 PM J. Bruce Fields <bfields@fieldses.org> wrote: > > Thanks for resending. What's the status of Amir's series? I guess I've > been using that as an excuse to put off reviewing these, but I really > should anyway.... > > --b. > > On Mon, Jul 08, 2019 at 03:23:44PM -0400, Olga Kornievskaia wrote: > > This patch series adds support for NFSv4.2 copy offload feature > > allowing copy between two different NFS servers. > > > > This functionality depends on the VFS ability to support generic > > copy_file_range() where a copy is done between an NFS file and > > a local file system. This is on top of Amir's VFS generic copy > > offload series. > > > > This feature is enabled by the kernel module parameter -- > > inter_copy_offload_enable -- and by default is disabled. There is > > also a kernel compile configuration of NFSD_V4_2_INTER_SSC that > > adds dependency on the NFS client side functions called from the > > server. > > > > These patches work on top of existing async intra copy offload > > patches. For the "inter" SSC, the implementation only supports > > asynchronous inter copy. > > > > On the source server, upon receiving a COPY_NOTIFY, it generate a > > unique stateid that's kept in the global list. Upon receiving a READ > > with a stateid, the code checks the normal list of open stateid and > > now additionally, it'll check the copy state list as well before > > deciding to either fail with BAD_STATEID or find one that matches. > > The stored stateid is only valid to be used for the first time > > with a choosen lease period (90s currently). When the source server > > received an OFFLOAD_CANCEL, it will remove the stateid from the > > global list. Otherwise, the copy stateid is removed upon the removal > > of its "parent" stateid (open/lock/delegation stateid). > > > > On the destination server, upon receiving a COPY request, the server > > establishes the necessary clientid/session with the source server. > > It calls into the NFS client code to establish the necessary > > open stateid, filehandle, file description (without doing an NFS open). > > Then the server calls into the copy_file_range() to preform the copy > > where the source file will issue NFS READs and then do local file > > system writes (this depends on the VFS ability to do cross device > > copy_file_range(). > > > > v4: > > --- allowing for synchronous inter server-to-server copy > > --- added missing offload_cancel on the source server > > > > Already presented numbers for performance improvement for large > > file transfer but here are times for copying linux kernel tree > > (which is mostly small files): > > -- regular cp 6m1s (intra) > > -- copy offload cp 4m11s (intra) > > -- benefit of using copy offload with small copies using sync copy > > -- regular cp 6m9s (inter) > > -- copy offload cp 6m3s (inter) > > -- same performance as traditional as for most it fallback to traditional > > copy offload > > > > Olga Kornievskaia (8): > > NFSD fill-in netloc4 structure > > NFSD add ca_source_server<> to COPY > > NFSD return nfs4_stid in nfs4_preprocess_stateid_op > > NFSD add COPY_NOTIFY operation > > NFSD check stateids against copy stateids > > NFSD generalize nfsd4_compound_state flag names > > NFSD: allow inter server COPY to have a STALE source server fh > > NFSD add nfs4 inter ssc to nfsd4_copy > > > > fs/nfsd/Kconfig | 10 ++ > > fs/nfsd/nfs4proc.c | 434 +++++++++++++++++++++++++++++++++++++++++++++++----- > > fs/nfsd/nfs4state.c | 135 ++++++++++++++-- > > fs/nfsd/nfs4xdr.c | 172 ++++++++++++++++++++- > > fs/nfsd/nfsd.h | 32 ++++ > > fs/nfsd/nfsfh.h | 5 +- > > fs/nfsd/nfssvc.c | 6 + > > fs/nfsd/state.h | 25 ++- > > fs/nfsd/xdr4.h | 37 ++++- > > 9 files changed, 790 insertions(+), 66 deletions(-) > > > > -- > > 1.8.3.1
Hi Bruce, Do you have any more comments than what Anna already provided? I have been waiting for more comments before putting up those 2 small changes. On Tue, Jul 9, 2019 at 11:47 AM Olga Kornievskaia <olga.kornievskaia@gmail.com> wrote: > > Amir's patches have been in the linux-next xfs tree and will go into > 5.3. I would like for NFS patches to go into 5.4 (I'm assuming hoping > for 5.3 is unrealistic). > > On Mon, Jul 8, 2019 at 11:53 PM J. Bruce Fields <bfields@fieldses.org> wrote: > > > > Thanks for resending. What's the status of Amir's series? I guess I've > > been using that as an excuse to put off reviewing these, but I really > > should anyway.... > > > > --b. > > > > On Mon, Jul 08, 2019 at 03:23:44PM -0400, Olga Kornievskaia wrote: > > > This patch series adds support for NFSv4.2 copy offload feature > > > allowing copy between two different NFS servers. > > > > > > This functionality depends on the VFS ability to support generic > > > copy_file_range() where a copy is done between an NFS file and > > > a local file system. This is on top of Amir's VFS generic copy > > > offload series. > > > > > > This feature is enabled by the kernel module parameter -- > > > inter_copy_offload_enable -- and by default is disabled. There is > > > also a kernel compile configuration of NFSD_V4_2_INTER_SSC that > > > adds dependency on the NFS client side functions called from the > > > server. > > > > > > These patches work on top of existing async intra copy offload > > > patches. For the "inter" SSC, the implementation only supports > > > asynchronous inter copy. > > > > > > On the source server, upon receiving a COPY_NOTIFY, it generate a > > > unique stateid that's kept in the global list. Upon receiving a READ > > > with a stateid, the code checks the normal list of open stateid and > > > now additionally, it'll check the copy state list as well before > > > deciding to either fail with BAD_STATEID or find one that matches. > > > The stored stateid is only valid to be used for the first time > > > with a choosen lease period (90s currently). When the source server > > > received an OFFLOAD_CANCEL, it will remove the stateid from the > > > global list. Otherwise, the copy stateid is removed upon the removal > > > of its "parent" stateid (open/lock/delegation stateid). > > > > > > On the destination server, upon receiving a COPY request, the server > > > establishes the necessary clientid/session with the source server. > > > It calls into the NFS client code to establish the necessary > > > open stateid, filehandle, file description (without doing an NFS open). > > > Then the server calls into the copy_file_range() to preform the copy > > > where the source file will issue NFS READs and then do local file > > > system writes (this depends on the VFS ability to do cross device > > > copy_file_range(). > > > > > > v4: > > > --- allowing for synchronous inter server-to-server copy > > > --- added missing offload_cancel on the source server > > > > > > Already presented numbers for performance improvement for large > > > file transfer but here are times for copying linux kernel tree > > > (which is mostly small files): > > > -- regular cp 6m1s (intra) > > > -- copy offload cp 4m11s (intra) > > > -- benefit of using copy offload with small copies using sync copy > > > -- regular cp 6m9s (inter) > > > -- copy offload cp 6m3s (inter) > > > -- same performance as traditional as for most it fallback to traditional > > > copy offload > > > > > > Olga Kornievskaia (8): > > > NFSD fill-in netloc4 structure > > > NFSD add ca_source_server<> to COPY > > > NFSD return nfs4_stid in nfs4_preprocess_stateid_op > > > NFSD add COPY_NOTIFY operation > > > NFSD check stateids against copy stateids > > > NFSD generalize nfsd4_compound_state flag names > > > NFSD: allow inter server COPY to have a STALE source server fh > > > NFSD add nfs4 inter ssc to nfsd4_copy > > > > > > fs/nfsd/Kconfig | 10 ++ > > > fs/nfsd/nfs4proc.c | 434 +++++++++++++++++++++++++++++++++++++++++++++++----- > > > fs/nfsd/nfs4state.c | 135 ++++++++++++++-- > > > fs/nfsd/nfs4xdr.c | 172 ++++++++++++++++++++- > > > fs/nfsd/nfsd.h | 32 ++++ > > > fs/nfsd/nfsfh.h | 5 +- > > > fs/nfsd/nfssvc.c | 6 + > > > fs/nfsd/state.h | 25 ++- > > > fs/nfsd/xdr4.h | 37 ++++- > > > 9 files changed, 790 insertions(+), 66 deletions(-) > > > > > > -- > > > 1.8.3.1