| Message ID | 20241206004829.3497925-5-neilb@suse.de (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | nfsd: allocate/free session-based DRC slots on demand | expand |
On Fri, 2024-12-06 at 11:43 +1100, NeilBrown wrote: > If a client ever uses the highest available slot for a given session, > attempt to allocate more slots so there is room for the client to use > them if wanted. GFP_NOWAIT is used so if there is not plenty of > free memory, failure is expected - which is what we want. It also > allows the allocation while holding a spinlock. > > Each time we increase the number of slots by 20% (rounded up). This > allows fairly quick growth while avoiding excessive over-shoot. > > We would expect to stablise with around 10% more slots available than > the client actually uses. > > Signed-off-by: NeilBrown <neilb@suse.de> > --- > fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++----- > 1 file changed, 35 insertions(+), 5 deletions(-) > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c > index 67dfc699e411..ec4468ebbd40 100644 > --- a/fs/nfsd/nfs4state.c > +++ b/fs/nfsd/nfs4state.c > @@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > slot = xa_load(&session->se_slots, seq->slotid); > dprintk("%s: slotid %d\n", __func__, seq->slotid); > > - /* We do not negotiate the number of slots yet, so set the > - * maxslots to the session maxreqs which is used to encode > - * sr_highest_slotid and the sr_target_slot id to maxslots */ > - seq->maxslots = session->se_fchannel.maxreqs; > - > trace_nfsd_slot_seqid_sequence(clp, seq, slot); > status = check_slot_seqid(seq->seqid, slot->sl_seqid, > slot->sl_flags & NFSD4_SLOT_INUSE); > @@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > cstate->session = session; > cstate->clp = clp; > > + /* > + * If the client ever uses the highest available slot, > + * gently try to allocate another 20%. This allows > + * fairly quick growth without grossly over-shooting what > + * the client might use. > + */ 20% seems like a reasonable place to start, but I do wonder if this might need to be tunable under some workloads. Oh well, we can cross that bridge if/when someone complains. > + if (seq->slotid == session->se_fchannel.maxreqs - 1 && > + session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) { > + int s = session->se_fchannel.maxreqs; > + int cnt = DIV_ROUND_UP(s, 5); > + > + do { > + /* > + * GFP_NOWAIT is a low-priority non-blocking > + * allocation which can be used under > + * client_lock and only succeeds if there is > + * plenty of memory. > + * Use GFP_ATOMIC which is higher priority for > + * xa_store() so we are less likely to waste the > + * effort of the first allocation. > + */ I don't know here. Why not just use GFP_NOWAIT for the xa_store too? If we're so memory constrained that that fails, we're probably better off releasing the slot. > + slot = kzalloc(slot_bytes(&session->se_fchannel), > + GFP_NOWAIT); > + if (slot && > + !xa_is_err(xa_store(&session->se_slots, s, slot, > + GFP_ATOMIC | __GFP_NOWARN))) { > + s += 1; > + session->se_fchannel.maxreqs = s; > + } else { > + kfree(slot); > + } > + } while (slot && --cnt > 0); > + } > + seq->maxslots = session->se_fchannel.maxreqs; > + > out: > switch (clp->cl_cb_state) { > case NFSD4_CB_DOWN:
On Fri, 06 Dec 2024, Jeff Layton wrote: > On Fri, 2024-12-06 at 11:43 +1100, NeilBrown wrote: > > If a client ever uses the highest available slot for a given session, > > attempt to allocate more slots so there is room for the client to use > > them if wanted. GFP_NOWAIT is used so if there is not plenty of > > free memory, failure is expected - which is what we want. It also > > allows the allocation while holding a spinlock. > > > > Each time we increase the number of slots by 20% (rounded up). This > > allows fairly quick growth while avoiding excessive over-shoot. > > > > We would expect to stablise with around 10% more slots available than > > the client actually uses. > > > > Signed-off-by: NeilBrown <neilb@suse.de> > > --- > > fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++----- > > 1 file changed, 35 insertions(+), 5 deletions(-) > > > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c > > index 67dfc699e411..ec4468ebbd40 100644 > > --- a/fs/nfsd/nfs4state.c > > +++ b/fs/nfsd/nfs4state.c > > @@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > > slot = xa_load(&session->se_slots, seq->slotid); > > dprintk("%s: slotid %d\n", __func__, seq->slotid); > > > > - /* We do not negotiate the number of slots yet, so set the > > - * maxslots to the session maxreqs which is used to encode > > - * sr_highest_slotid and the sr_target_slot id to maxslots */ > > - seq->maxslots = session->se_fchannel.maxreqs; > > - > > trace_nfsd_slot_seqid_sequence(clp, seq, slot); > > status = check_slot_seqid(seq->seqid, slot->sl_seqid, > > slot->sl_flags & NFSD4_SLOT_INUSE); > > @@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > > cstate->session = session; > > cstate->clp = clp; > > > > + /* > > + * If the client ever uses the highest available slot, > > + * gently try to allocate another 20%. This allows > > + * fairly quick growth without grossly over-shooting what > > + * the client might use. > > + */ > > 20% seems like a reasonable place to start, but I do wonder if this > might need to be tunable under some workloads. Oh well, we can cross > that bridge if/when someone complains. I think that if we need a tunable, then it is a failure of design. If?when someone complains we may well need to redesign. I hope we could avoid a tunable in that design! > > > + if (seq->slotid == session->se_fchannel.maxreqs - 1 && > > + session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) { > > + int s = session->se_fchannel.maxreqs; > > + int cnt = DIV_ROUND_UP(s, 5); > > + > > + do { > > + /* > > + * GFP_NOWAIT is a low-priority non-blocking > > + * allocation which can be used under > > + * client_lock and only succeeds if there is > > + * plenty of memory. > > + * Use GFP_ATOMIC which is higher priority for > > + * xa_store() so we are less likely to waste the > > + * effort of the first allocation. > > + */ > > I don't know here. Why not just use GFP_NOWAIT for the xa_store too? If > we're so memory constrained that that fails, we're probably better off > releasing the slot. Maybe. I'm open simple using GFP_NOWAIT both places. Most often xa_store won't need to allocate anything - it adds slots to the array in batches (at least I assume it does - anything else would be inefficient). So it mostly won't matter. So if seems at all inelegant - let's drop it. Thanks, NeilBrown > > > + slot = kzalloc(slot_bytes(&session->se_fchannel), > > + GFP_NOWAIT); > > + if (slot && > > + !xa_is_err(xa_store(&session->se_slots, s, slot, > > + GFP_ATOMIC | __GFP_NOWARN))) { > > + s += 1; > > + session->se_fchannel.maxreqs = s; > > + } else { > > + kfree(slot); > > + } > > + } while (slot && --cnt > 0); > > + } > > + seq->maxslots = session->se_fchannel.maxreqs; > > + > > out: > > switch (clp->cl_cb_state) { > > case NFSD4_CB_DOWN: > > -- > Jeff Layton <jlayton@kernel.org> > >
On Fri, 2024-12-06 at 12:43 +1100, NeilBrown wrote: > On Fri, 06 Dec 2024, Jeff Layton wrote: > > On Fri, 2024-12-06 at 11:43 +1100, NeilBrown wrote: > > > If a client ever uses the highest available slot for a given session, > > > attempt to allocate more slots so there is room for the client to use > > > them if wanted. GFP_NOWAIT is used so if there is not plenty of > > > free memory, failure is expected - which is what we want. It also > > > allows the allocation while holding a spinlock. > > > > > > Each time we increase the number of slots by 20% (rounded up). This > > > allows fairly quick growth while avoiding excessive over-shoot. > > > > > > We would expect to stablise with around 10% more slots available than > > > the client actually uses. > > > > > > Signed-off-by: NeilBrown <neilb@suse.de> > > > --- > > > fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++----- > > > 1 file changed, 35 insertions(+), 5 deletions(-) > > > > > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c > > > index 67dfc699e411..ec4468ebbd40 100644 > > > --- a/fs/nfsd/nfs4state.c > > > +++ b/fs/nfsd/nfs4state.c > > > @@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > > > slot = xa_load(&session->se_slots, seq->slotid); > > > dprintk("%s: slotid %d\n", __func__, seq->slotid); > > > > > > - /* We do not negotiate the number of slots yet, so set the > > > - * maxslots to the session maxreqs which is used to encode > > > - * sr_highest_slotid and the sr_target_slot id to maxslots */ > > > - seq->maxslots = session->se_fchannel.maxreqs; > > > - > > > trace_nfsd_slot_seqid_sequence(clp, seq, slot); > > > status = check_slot_seqid(seq->seqid, slot->sl_seqid, > > > slot->sl_flags & NFSD4_SLOT_INUSE); > > > @@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > > > cstate->session = session; > > > cstate->clp = clp; > > > > > > + /* > > > + * If the client ever uses the highest available slot, > > > + * gently try to allocate another 20%. This allows > > > + * fairly quick growth without grossly over-shooting what > > > + * the client might use. > > > + */ > > > > 20% seems like a reasonable place to start, but I do wonder if this > > might need to be tunable under some workloads. Oh well, we can cross > > that bridge if/when someone complains. > > I think that if we need a tunable, then it is a failure of design. > If?when someone complains we may well need to redesign. I hope we could > avoid a tunable in that design! > I hope so too. > > > > > + if (seq->slotid == session->se_fchannel.maxreqs - 1 && > > > + session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) { > > > + int s = session->se_fchannel.maxreqs; > > > + int cnt = DIV_ROUND_UP(s, 5); > > > + > > > + do { > > > + /* > > > + * GFP_NOWAIT is a low-priority non-blocking > > > + * allocation which can be used under > > > + * client_lock and only succeeds if there is > > > + * plenty of memory. > > > + * Use GFP_ATOMIC which is higher priority for > > > + * xa_store() so we are less likely to waste the > > > + * effort of the first allocation. > > > + */ > > > > I don't know here. Why not just use GFP_NOWAIT for the xa_store too? If > > we're so memory constrained that that fails, we're probably better off > > releasing the slot. > > Maybe. I'm open simple using GFP_NOWAIT both places. > Most often xa_store won't need to allocate anything - it adds slots to > the array in batches (at least I assume it does - anything else would be > inefficient). So it mostly won't matter. > So if seems at all inelegant - let's drop it. > > I'd prefer we drop that part. It probably won't matter much in the long run anyway. > > > > > > + slot = kzalloc(slot_bytes(&session->se_fchannel), > > > + GFP_NOWAIT); > > > + if (slot && > > > + !xa_is_err(xa_store(&session->se_slots, s, slot, > > > + GFP_ATOMIC | __GFP_NOWARN))) { > > > + s += 1; > > > + session->se_fchannel.maxreqs = s; > > > + } else { > > > + kfree(slot); > > > + } > > > + } while (slot && --cnt > 0); > > > + } > > > + seq->maxslots = session->se_fchannel.maxreqs; > > > + > > > out: > > > switch (clp->cl_cb_state) { > > > case NFSD4_CB_DOWN: > > > > -- > > Jeff Layton <jlayton@kernel.org> > > > > >
On 12/5/24 7:43 PM, NeilBrown wrote: > If a client ever uses the highest available slot for a given session, > attempt to allocate more slots so there is room for the client to use > them if wanted. GFP_NOWAIT is used so if there is not plenty of > free memory, failure is expected - which is what we want. It also > allows the allocation while holding a spinlock. > > Each time we increase the number of slots by 20% (rounded up). This > allows fairly quick growth while avoiding excessive over-shoot. > > We would expect to stablise with around 10% more slots available than > the client actually uses. > > Signed-off-by: NeilBrown <neilb@suse.de> > --- > fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++----- > 1 file changed, 35 insertions(+), 5 deletions(-) > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c > index 67dfc699e411..ec4468ebbd40 100644 > --- a/fs/nfsd/nfs4state.c > +++ b/fs/nfsd/nfs4state.c > @@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > slot = xa_load(&session->se_slots, seq->slotid); > dprintk("%s: slotid %d\n", __func__, seq->slotid); > > - /* We do not negotiate the number of slots yet, so set the > - * maxslots to the session maxreqs which is used to encode > - * sr_highest_slotid and the sr_target_slot id to maxslots */ > - seq->maxslots = session->se_fchannel.maxreqs; > - > trace_nfsd_slot_seqid_sequence(clp, seq, slot); > status = check_slot_seqid(seq->seqid, slot->sl_seqid, > slot->sl_flags & NFSD4_SLOT_INUSE); > @@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > cstate->session = session; > cstate->clp = clp; > > + /* > + * If the client ever uses the highest available slot, > + * gently try to allocate another 20%. This allows > + * fairly quick growth without grossly over-shooting what > + * the client might use. > + */ > + if (seq->slotid == session->se_fchannel.maxreqs - 1 && > + session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) { > + int s = session->se_fchannel.maxreqs; > + int cnt = DIV_ROUND_UP(s, 5); > + > + do { > + /* > + * GFP_NOWAIT is a low-priority non-blocking > + * allocation which can be used under > + * client_lock and only succeeds if there is > + * plenty of memory. > + * Use GFP_ATOMIC which is higher priority for > + * xa_store() so we are less likely to waste the > + * effort of the first allocation. > + */ > + slot = kzalloc(slot_bytes(&session->se_fchannel), > + GFP_NOWAIT); > + if (slot && > + !xa_is_err(xa_store(&session->se_slots, s, slot, > + GFP_ATOMIC | __GFP_NOWARN))) { > + s += 1; > + session->se_fchannel.maxreqs = s; > + } else { > + kfree(slot); Don't you want to break out of this loop if slot allocation or the xa_store() fails? Does the session logic work if there is a gap of unallocated slots in the slot table? Seems like we want to wait a bit anyway after an allocation failure before asking again. Otherwise, LGTM. I assume a v4 is forthcoming to address review comments. > + } > + } while (slot && --cnt > 0); > + } > + seq->maxslots = session->se_fchannel.maxreqs; > + > out: > switch (clp->cl_cb_state) { > case NFSD4_CB_DOWN:
On Sat, 07 Dec 2024, Chuck Lever wrote: > On 12/5/24 7:43 PM, NeilBrown wrote: > > If a client ever uses the highest available slot for a given session, > > attempt to allocate more slots so there is room for the client to use > > them if wanted. GFP_NOWAIT is used so if there is not plenty of > > free memory, failure is expected - which is what we want. It also > > allows the allocation while holding a spinlock. > > > > Each time we increase the number of slots by 20% (rounded up). This > > allows fairly quick growth while avoiding excessive over-shoot. > > > > We would expect to stablise with around 10% more slots available than > > the client actually uses. > > > > Signed-off-by: NeilBrown <neilb@suse.de> > > --- > > fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++----- > > 1 file changed, 35 insertions(+), 5 deletions(-) > > > > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c > > index 67dfc699e411..ec4468ebbd40 100644 > > --- a/fs/nfsd/nfs4state.c > > +++ b/fs/nfsd/nfs4state.c > > @@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > > slot = xa_load(&session->se_slots, seq->slotid); > > dprintk("%s: slotid %d\n", __func__, seq->slotid); > > > > - /* We do not negotiate the number of slots yet, so set the > > - * maxslots to the session maxreqs which is used to encode > > - * sr_highest_slotid and the sr_target_slot id to maxslots */ > > - seq->maxslots = session->se_fchannel.maxreqs; > > - > > trace_nfsd_slot_seqid_sequence(clp, seq, slot); > > status = check_slot_seqid(seq->seqid, slot->sl_seqid, > > slot->sl_flags & NFSD4_SLOT_INUSE); > > @@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, > > cstate->session = session; > > cstate->clp = clp; > > > > + /* > > + * If the client ever uses the highest available slot, > > + * gently try to allocate another 20%. This allows > > + * fairly quick growth without grossly over-shooting what > > + * the client might use. > > + */ > > + if (seq->slotid == session->se_fchannel.maxreqs - 1 && > > + session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) { > > + int s = session->se_fchannel.maxreqs; > > + int cnt = DIV_ROUND_UP(s, 5); > > + > > + do { > > + /* > > + * GFP_NOWAIT is a low-priority non-blocking > > + * allocation which can be used under > > + * client_lock and only succeeds if there is > > + * plenty of memory. > > + * Use GFP_ATOMIC which is higher priority for > > + * xa_store() so we are less likely to waste the > > + * effort of the first allocation. > > + */ > > + slot = kzalloc(slot_bytes(&session->se_fchannel), > > + GFP_NOWAIT); > > + if (slot && > > + !xa_is_err(xa_store(&session->se_slots, s, slot, > > + GFP_ATOMIC | __GFP_NOWARN))) { > > + s += 1; > > + session->se_fchannel.maxreqs = s; > > + } else { > > + kfree(slot); > > Don't you want to break out of this loop if slot allocation or the > xa_store() fails? Does the session logic work if there is a gap > of unallocated slots in the slot table? Seems like we want to wait > a bit anyway after an allocation failure before asking again. Indeed! The "slot = NULL" which the next patch adds should be in this patch. That makes the loop abort. > > Otherwise, LGTM. I assume a v4 is forthcoming to address review > comments. I'll send that out Monday morning. Thanks, NeilBrown > > > > + } > > + } while (slot && --cnt > 0); > > + } > > + seq->maxslots = session->se_fchannel.maxreqs; > > + > > out: > > switch (clp->cl_cb_state) { > > case NFSD4_CB_DOWN: > > > -- > Chuck Lever >
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 67dfc699e411..ec4468ebbd40 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -4235,11 +4235,6 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, slot = xa_load(&session->se_slots, seq->slotid); dprintk("%s: slotid %d\n", __func__, seq->slotid); - /* We do not negotiate the number of slots yet, so set the - * maxslots to the session maxreqs which is used to encode - * sr_highest_slotid and the sr_target_slot id to maxslots */ - seq->maxslots = session->se_fchannel.maxreqs; - trace_nfsd_slot_seqid_sequence(clp, seq, slot); status = check_slot_seqid(seq->seqid, slot->sl_seqid, slot->sl_flags & NFSD4_SLOT_INUSE); @@ -4289,6 +4284,41 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, cstate->session = session; cstate->clp = clp; + /* + * If the client ever uses the highest available slot, + * gently try to allocate another 20%. This allows + * fairly quick growth without grossly over-shooting what + * the client might use. + */ + if (seq->slotid == session->se_fchannel.maxreqs - 1 && + session->se_fchannel.maxreqs < NFSD_MAX_SLOTS_PER_SESSION) { + int s = session->se_fchannel.maxreqs; + int cnt = DIV_ROUND_UP(s, 5); + + do { + /* + * GFP_NOWAIT is a low-priority non-blocking + * allocation which can be used under + * client_lock and only succeeds if there is + * plenty of memory. + * Use GFP_ATOMIC which is higher priority for + * xa_store() so we are less likely to waste the + * effort of the first allocation. + */ + slot = kzalloc(slot_bytes(&session->se_fchannel), + GFP_NOWAIT); + if (slot && + !xa_is_err(xa_store(&session->se_slots, s, slot, + GFP_ATOMIC | __GFP_NOWARN))) { + s += 1; + session->se_fchannel.maxreqs = s; + } else { + kfree(slot); + } + } while (slot && --cnt > 0); + } + seq->maxslots = session->se_fchannel.maxreqs; + out: switch (clp->cl_cb_state) { case NFSD4_CB_DOWN:
If a client ever uses the highest available slot for a given session, attempt to allocate more slots so there is room for the client to use them if wanted. GFP_NOWAIT is used so if there is not plenty of free memory, failure is expected - which is what we want. It also allows the allocation while holding a spinlock. Each time we increase the number of slots by 20% (rounded up). This allows fairly quick growth while avoiding excessive over-shoot. We would expect to stablise with around 10% more slots available than the client actually uses. Signed-off-by: NeilBrown <neilb@suse.de> --- fs/nfsd/nfs4state.c | 40 +++++++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-)