| Message ID | 1403189450-18729-105-git-send-email-jlayton@primarydata.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
> +KNFSD4 State Object Model: > +========================== > +Written 2014 by Jeff Layton <jlayton@primarydata.com> > + > +Introduction: > +------------- > +Until recently, knfsd relied heavily on a global mutex to ensure that objects > +didn't disappear while they were being operated on. That has proven to be a > +scalability bottleneck however, so the code has been overhauled to make heavy > +use of reference counting and spinlocks for tracking the different objects. It's hard enough to keep documents uptodate with the current version of the code, so I don't think this historical blurb buys us anything. > +struct nfs4_client: > +------------------- > +The initial object created by an NFS client using SETCLIENTID (for NFSv4.0) or > +EXCHANGE_ID (for NFSv4.1+). These objects are refcounted and timestamped. Each > +nfsd_net_ns object contains a set of these and they are tracked via short and > +long form clientid. They are hashed and searched for under the per-nfsd-net > +client_lock spinlock. > + > +The lifecycle of these is a little strange. References to it are only held > +during the processing of compounds, and in certain other operations. In their > +"resting state" they have a refcount of 0. If they are not renewed within a > +lease period, they become eligible for destruction by the laundromat. That's the normal lifecycle of objects on a lru, so I'd strike the "strange" part. > +strict nfs4_stid: > +----------------- > +A core object that represents a "generic" stateid. These are generally embedded > +within the different (more specific) stateid objects and contain fields that > +are of general use to any stateid. Maybe replace "generic" stateid with common stateid or similar? The generic_stateid term is unfortunately is used in the code in a weird way for open/lock stateids. Or better yet fix up those names in the code.. Given how documentation outside the source code gets out of data easily maybe you should move these texts to comments above each of the structures? -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 24 Jun 2014 05:11:58 -0700 Christoph Hellwig <hch@infradead.org> wrote: > > +KNFSD4 State Object Model: > > +========================== > > +Written 2014 by Jeff Layton <jlayton@primarydata.com> > > + > > +Introduction: > > +------------- > > +Until recently, knfsd relied heavily on a global mutex to ensure that objects > > +didn't disappear while they were being operated on. That has proven to be a > > +scalability bottleneck however, so the code has been overhauled to make heavy > > +use of reference counting and spinlocks for tracking the different objects. > > It's hard enough to keep documents uptodate with the current version of > the code, so I don't think this historical blurb buys us anything. > OK > > +struct nfs4_client: > > +------------------- > > +The initial object created by an NFS client using SETCLIENTID (for NFSv4.0) or > > +EXCHANGE_ID (for NFSv4.1+). These objects are refcounted and timestamped. Each > > +nfsd_net_ns object contains a set of these and they are tracked via short and > > +long form clientid. They are hashed and searched for under the per-nfsd-net > > +client_lock spinlock. > > + > > +The lifecycle of these is a little strange. References to it are only held > > +during the processing of compounds, and in certain other operations. In their > > +"resting state" they have a refcount of 0. If they are not renewed within a > > +lease period, they become eligible for destruction by the laundromat. > > That's the normal lifecycle of objects on a lru, so I'd strike the > "strange" part. > Fair enough. > > +strict nfs4_stid: > > +----------------- > > +A core object that represents a "generic" stateid. These are generally embedded > > +within the different (more specific) stateid objects and contain fields that > > +are of general use to any stateid. > > Maybe replace "generic" stateid with common stateid or similar? The > generic_stateid term is unfortunately is used in the code in a weird way > for open/lock stateids. Or better yet fix up those names in the code.. > Yeah, I didn't make that change here as we have enough "churn" going on, but maybe it's a good idea to just do it. "generic" really ought to refer to nfs4_stid, and we should rename the nfs4_ol_stateid to something besides "generic". > > Given how documentation outside the source code gets out of data > easily maybe you should move these texts to comments above each of the > structures? Yeah, probably so. This document started for my own benefit as I was going through the code and documenting things. We can drop it if it's not helpful but I figured I'd toss it on top of the pile in case it was.
diff --git a/Documentation/filesystems/nfs/nfsd4-state-objects.txt b/Documentation/filesystems/nfs/nfsd4-state-objects.txt new file mode 100644 index 000000000000..fa3f70608422 --- /dev/null +++ b/Documentation/filesystems/nfs/nfsd4-state-objects.txt @@ -0,0 +1,110 @@ +KNFSD4 State Object Model: +========================== +Written 2014 by Jeff Layton <jlayton@primarydata.com> + +Introduction: +------------- +Until recently, knfsd relied heavily on a global mutex to ensure that objects +didn't disappear while they were being operated on. That has proven to be a +scalability bottleneck however, so the code has been overhauled to make heavy +use of reference counting and spinlocks for tracking the different objects. + +The state model in NFSv4 is quite complex. Thus, the relationship between the +objects to track this state in knfsd is also complex. This document attempts to +lay out how all of the different objects relate to one another, and which ones +hold references to others. + +In addition to the "persistent" references documented here, references to these +objects can also be taken during the processing of compounds in order to ensure +that the objects don't disappear suddenly. + +struct nfsd_net: +---------------- +Represents a nfsd "container". With respect to nfsv4 state tracking, the fields +of interest are the *_id_hashtbls and the *_name_tree. These track the +nfs4_client objects by either short or long form clientid. + +Each nfsd_net runs a nfs4_laundromat workqueue job every lease period to clean +up expired clients and delegations within the container. + +struct nfs4_file: +----------------- +These objects are global. nfsd only keeps one instance of a nfs4_file per inode +(though it may keep multiple file descriptors open per inode). These are +tracked in the file_hashtbl which is protected by the state_lock spinlock. + +struct nfs4_client: +------------------- +The initial object created by an NFS client using SETCLIENTID (for NFSv4.0) or +EXCHANGE_ID (for NFSv4.1+). These objects are refcounted and timestamped. Each +nfsd_net_ns object contains a set of these and they are tracked via short and +long form clientid. They are hashed and searched for under the per-nfsd-net +client_lock spinlock. + +The lifecycle of these is a little strange. References to it are only held +during the processing of compounds, and in certain other operations. In their +"resting state" they have a refcount of 0. If they are not renewed within a +lease period, they become eligible for destruction by the laundromat. + +These objects can also be destroyed prematurely by the fault injection code, +or if the client sends certain forms of SETCLIENTID or EXCHANGE_ID updates. +Care is taken *not* to do this however when the objects have an elevated +refcount. + +struct nfsd4_session: +--------------------- +Represents a v4.1+ session. These are refcounted in a similar fashion to +the nfs4_client. References are only taken when the server is actively working +on the object (primarily during the processing of compounds). + +struct nfs4_stateowner: +----------------------- +A core object that represents either an open or lock owner. The object and lock +owner objects have one of these embedded within them. Refcounts and other +fields common to both owner types are contained within these structures. + +struct nfs4_openowner: +---------------------- +When a file is opened, the client provides an open state owner opaque string +that indicates the "owner" of that open. These objects are refcounted. +References to it are held by each open state associated with it. This object is +a superset of the nfs4_stateowner struct. + +struct nfs4_lockowner: +---------------------- +Represents a generic "lockowner". Similar to an openowner. References to it are +held by the lock stateids that are created on its behalf. This object is a +superset of the nfs4_stateowner struct. + +strict nfs4_stid: +----------------- +A core object that represents a "generic" stateid. These are generally embedded +within the different (more specific) stateid objects and contain fields that +are of general use to any stateid. + +struct nfs4_ol_stateid: +----------------------- +A generic struct representing either a open or lock stateid. The nfs4_client +holds a reference to each of these objects, and they in turn hold a reference +to their respective stateowners. The client's reference is released in response +to a close or unlock (depending on whether it's an open or lock stateid) or +when the client is being destroyed. + +In the case of v4.0, these objects are preserved for a little while after close +in order to handle CLOSE replays. They are eventually reclaimed via a LRU +scheme by the laundromat. + +This object is a superset of the nfs4_stid. + +struct nfs4_delegation: +----------------------- +Represents a delegation stateid. The nfs4_client holds references to these and +they are put when it is being destroyed or when the delegation is returned by +the client. + +If the server attempts to recall a delegation and the client doesn't do so +before a timeout, the server may also revoke the delegation. In that case, the +object will either be destroyed (v4.0) or moved to a per-client list of revoked +delegations (v4.1+). + +This object is a superset of the nfs4_stid.
Signed-off-by: Jeff Layton <jlayton@primarydata.com> --- .../filesystems/nfs/nfsd4-state-objects.txt | 110 +++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 Documentation/filesystems/nfs/nfsd4-state-objects.txt