mbox series

[6.6.y,00/12] Backport "make svc_stat per-net instead of global"

Message ID 20240812223604.32592-1-cel@kernel.org (mailing list archive)
Headers show
Series Backport "make svc_stat per-net instead of global" | expand

Message

Chuck Lever Aug. 12, 2024, 10:35 p.m. UTC
From: Chuck Lever <chuck.lever@oracle.com>

Following up on:

https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/

Here is a backport series targeting origin/linux-6.6.y that closes
the information leak described in the above thread. It passes basic
NFSD regression testing.

Review comments welcome.   

Chuck Lever (2):
  NFSD: Rewrite synopsis of nfsd_percpu_counters_init()
  NFSD: Fix frame size warning in svc_export_parse()

Josef Bacik (10):
  sunrpc: don't change ->sv_stats if it doesn't exist
  nfsd: stop setting ->pg_stats for unused stats
  sunrpc: pass in the sv_stats struct through svc_create_pooled
  sunrpc: remove ->pg_stats from svc_program
  sunrpc: use the struct net as the svc proc private
  nfsd: rename NFSD_NET_* to NFSD_STATS_*
  nfsd: expose /proc/net/sunrpc/nfsd in net namespaces
  nfsd: make all of the nfsd stats per-network namespace
  nfsd: remove nfsd_stats, make th_cnt a global counter
  nfsd: make svc_stat per-network namespace instead of global

 fs/lockd/svc.c             |  3 --
 fs/nfs/callback.c          |  3 --
 fs/nfsd/cache.h            |  2 -
 fs/nfsd/export.c           | 32 ++++++++++----
 fs/nfsd/export.h           |  4 +-
 fs/nfsd/netns.h            | 25 +++++++++--
 fs/nfsd/nfs4proc.c         |  6 +--
 fs/nfsd/nfs4state.c        |  3 +-
 fs/nfsd/nfscache.c         | 40 ++++-------------
 fs/nfsd/nfsctl.c           | 16 +++----
 fs/nfsd/nfsd.h             |  1 +
 fs/nfsd/nfsfh.c            |  3 +-
 fs/nfsd/nfssvc.c           | 14 +++---
 fs/nfsd/stats.c            | 54 ++++++++++-------------
 fs/nfsd/stats.h            | 88 ++++++++++++++------------------------
 fs/nfsd/vfs.c              |  6 ++-
 include/linux/sunrpc/svc.h |  5 ++-
 net/sunrpc/stats.c         |  2 +-
 net/sunrpc/svc.c           | 39 +++++++++++------
 19 files changed, 163 insertions(+), 183 deletions(-)

Comments

Petr Vorel Aug. 14, 2024, 7:45 a.m. UTC | #1
Hi Chuck,

> Following up on:

> https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/

> Here is a backport series targeting origin/linux-6.6.y that closes
> the information leak described in the above thread. It passes basic
> NFSD regression testing.


Thank you for handling this! The link above mentions that it was already
backported to 5.4 and indeed I see at least d47151b79e322 ("nfs: expose
/proc/net/sunrpc/nfs in net namespaces") is backported in 5.4, 5.10, 5.15, 6.1.
And you're now preparing 6.6. Thus we can expect the behavior changed from
5.4 kernels.

I wonder if we consider this as a fix, thus expect any kernel newer than 5.4
should backport all these 12 patches.

Or, whether we should relax and just check if version is higher than the one
which got it in stable/LTS (e.g. >= 5.4.276 || >= 5.10.217 ...). The question is
also if enterprise distros will take this patchset.

BTW We have in LTP functionality which points as a hint to kernel fixes. But
it's usually a single commit. I might need to list all.

Kind regards,
Petr

> Review comments welcome.   

> Chuck Lever (2):
>   NFSD: Rewrite synopsis of nfsd_percpu_counters_init()
>   NFSD: Fix frame size warning in svc_export_parse()

> Josef Bacik (10):
>   sunrpc: don't change ->sv_stats if it doesn't exist
>   nfsd: stop setting ->pg_stats for unused stats
>   sunrpc: pass in the sv_stats struct through svc_create_pooled
>   sunrpc: remove ->pg_stats from svc_program
>   sunrpc: use the struct net as the svc proc private
>   nfsd: rename NFSD_NET_* to NFSD_STATS_*
>   nfsd: expose /proc/net/sunrpc/nfsd in net namespaces
>   nfsd: make all of the nfsd stats per-network namespace
>   nfsd: remove nfsd_stats, make th_cnt a global counter
>   nfsd: make svc_stat per-network namespace instead of global

>  fs/lockd/svc.c             |  3 --
>  fs/nfs/callback.c          |  3 --
>  fs/nfsd/cache.h            |  2 -
>  fs/nfsd/export.c           | 32 ++++++++++----
>  fs/nfsd/export.h           |  4 +-
>  fs/nfsd/netns.h            | 25 +++++++++--
>  fs/nfsd/nfs4proc.c         |  6 +--
>  fs/nfsd/nfs4state.c        |  3 +-
>  fs/nfsd/nfscache.c         | 40 ++++-------------
>  fs/nfsd/nfsctl.c           | 16 +++----
>  fs/nfsd/nfsd.h             |  1 +
>  fs/nfsd/nfsfh.c            |  3 +-
>  fs/nfsd/nfssvc.c           | 14 +++---
>  fs/nfsd/stats.c            | 54 ++++++++++-------------
>  fs/nfsd/stats.h            | 88 ++++++++++++++------------------------
>  fs/nfsd/vfs.c              |  6 ++-
>  include/linux/sunrpc/svc.h |  5 ++-
>  net/sunrpc/stats.c         |  2 +-
>  net/sunrpc/svc.c           | 39 +++++++++++------
>  19 files changed, 163 insertions(+), 183 deletions(-)
Petr Vorel Aug. 14, 2024, 10:09 a.m. UTC | #2
Hi Calum,

> Hi Petr,

> There are two sets of changes here, for NFS client, and NFS server.

> The NFS client changes have already been backported from v6.9 all the way to v5.4.

> Here, Chuck is discussing the NFS server changes (and others), which were not backported from v6.9 (actually, a few were, but only to v6.8).

Thanks for info! Now I'll see the patchset "Make nfsd stats visible in network
ns" [1]. kernelnewbies [2] starts with d98416cc2154 ("nfsd: rename
NFSD_NET_* to NFSD_STATS_*"), the others are probably some preparation commits.

Anyway, I'll update the patch with NFS server patchset.

Kind regards,
Petr

[1] https://lore.kernel.org/linux-nfs/cover.1706283433.git.josef@toxicpanda.com/
[2] https://kernelnewbies.org/Linux_6.9#File_systems


> Thanks,
> Calum.

> Sent from Outlook for Android<https://aka.ms/AAb9ysg>
> ________________________________
> From: Petr Vorel <pvorel@suse.cz>
> Sent: Wednesday, August 14, 2024 8:45:59 AM
> To: cel@kernel.org <cel@kernel.org>
> Cc: stable@vger.kernel.org <stable@vger.kernel.org>; linux-nfs@vger.kernel.org <linux-nfs@vger.kernel.org>; Sherry Yang <sherry.yang@oracle.com>; Calum Mackay <calum.mackay@oracle.com>; kernel-team@fb.com <kernel-team@fb.com>; Chuck Lever III <chuck.lever@oracle.com>; Cyril Hrubis <chrubis@suse.cz>; ltp@lists.linux.it <ltp@lists.linux.it>
> Subject: Re: [PATCH 6.6.y 00/12] Backport "make svc_stat per-net instead of global"

> Hi Chuck,

> > Following up on:

> > https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/

> > Here is a backport series targeting origin/linux-6.6.y that closes
> > the information leak described in the above thread. It passes basic
> > NFSD regression testing.


> Thank you for handling this! The link above mentions that it was already
> backported to 5.4 and indeed I see at least d47151b79e322 ("nfs: expose
> /proc/net/sunrpc/nfs in net namespaces") is backported in 5.4, 5.10, 5.15, 6.1.
> And you're now preparing 6.6. Thus we can expect the behavior changed from
> 5.4 kernels.

> I wonder if we consider this as a fix, thus expect any kernel newer than 5.4
> should backport all these 12 patches.

> Or, whether we should relax and just check if version is higher than the one
> which got it in stable/LTS (e.g. >= 5.4.276 || >= 5.10.217 ...). The question is
> also if enterprise distros will take this patchset.

> BTW We have in LTP functionality which points as a hint to kernel fixes. But
> it's usually a single commit. I might need to list all.

> Kind regards,
> Petr

> > Review comments welcome.

> > Chuck Lever (2):
> >   NFSD: Rewrite synopsis of nfsd_percpu_counters_init()
> >   NFSD: Fix frame size warning in svc_export_parse()

> > Josef Bacik (10):
> >   sunrpc: don't change ->sv_stats if it doesn't exist
> >   nfsd: stop setting ->pg_stats for unused stats
> >   sunrpc: pass in the sv_stats struct through svc_create_pooled
> >   sunrpc: remove ->pg_stats from svc_program
> >   sunrpc: use the struct net as the svc proc private
> >   nfsd: rename NFSD_NET_* to NFSD_STATS_*
> >   nfsd: expose /proc/net/sunrpc/nfsd in net namespaces
> >   nfsd: make all of the nfsd stats per-network namespace
> >   nfsd: remove nfsd_stats, make th_cnt a global counter
> >   nfsd: make svc_stat per-network namespace instead of global

> >  fs/lockd/svc.c             |  3 --
> >  fs/nfs/callback.c          |  3 --
> >  fs/nfsd/cache.h            |  2 -
> >  fs/nfsd/export.c           | 32 ++++++++++----
> >  fs/nfsd/export.h           |  4 +-
> >  fs/nfsd/netns.h            | 25 +++++++++--
> >  fs/nfsd/nfs4proc.c         |  6 +--
> >  fs/nfsd/nfs4state.c        |  3 +-
> >  fs/nfsd/nfscache.c         | 40 ++++-------------
> >  fs/nfsd/nfsctl.c           | 16 +++----
> >  fs/nfsd/nfsd.h             |  1 +
> >  fs/nfsd/nfsfh.c            |  3 +-
> >  fs/nfsd/nfssvc.c           | 14 +++---
> >  fs/nfsd/stats.c            | 54 ++++++++++-------------
> >  fs/nfsd/stats.h            | 88 ++++++++++++++------------------------
> >  fs/nfsd/vfs.c              |  6 ++-
> >  include/linux/sunrpc/svc.h |  5 ++-
> >  net/sunrpc/stats.c         |  2 +-
> >  net/sunrpc/svc.c           | 39 +++++++++++------
> >  19 files changed, 163 insertions(+), 183 deletions(-)
Calum Mackay Aug. 14, 2024, 12:10 p.m. UTC | #3
Hi Petr,

[resending for the lists, owing to blockage, sorry]

There are two sets of changes here, for NFS client, and NFS server.

The NFS client changes have already been backported from v6.9 all the 
way to v5.4.

Here, Chuck is discussing the NFS server changes (and others), which 
were not backported from v6.9 (actually, a few were, but only to v6.8).

Thanks,
Calum.

On 14/08/2024 8:45 am, Petr Vorel wrote:
> Hi Chuck,
> 
>> Following up on:
> 
>> https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/
> 
>> Here is a backport series targeting origin/linux-6.6.y that closes
>> the information leak described in the above thread. It passes basic
>> NFSD regression testing.
> 
> 
> Thank you for handling this! The link above mentions that it was already
> backported to 5.4 and indeed I see at least d47151b79e322 ("nfs: expose
> /proc/net/sunrpc/nfs in net namespaces") is backported in 5.4, 5.10, 5.15, 6.1.
> And you're now preparing 6.6. Thus we can expect the behavior changed from
> 5.4 kernels.
> 
> I wonder if we consider this as a fix, thus expect any kernel newer than 5.4
> should backport all these 12 patches.
> 
> Or, whether we should relax and just check if version is higher than the one
> which got it in stable/LTS (e.g. >= 5.4.276 || >= 5.10.217 ...). The question is
> also if enterprise distros will take this patchset.
> 
> BTW We have in LTP functionality which points as a hint to kernel fixes. But
> it's usually a single commit. I might need to list all.
> 
> Kind regards,
> Petr
> 
>> Review comments welcome.
> 
>> Chuck Lever (2):
>>    NFSD: Rewrite synopsis of nfsd_percpu_counters_init()
>>    NFSD: Fix frame size warning in svc_export_parse()
> 
>> Josef Bacik (10):
>>    sunrpc: don't change ->sv_stats if it doesn't exist
>>    nfsd: stop setting ->pg_stats for unused stats
>>    sunrpc: pass in the sv_stats struct through svc_create_pooled
>>    sunrpc: remove ->pg_stats from svc_program
>>    sunrpc: use the struct net as the svc proc private
>>    nfsd: rename NFSD_NET_* to NFSD_STATS_*
>>    nfsd: expose /proc/net/sunrpc/nfsd in net namespaces
>>    nfsd: make all of the nfsd stats per-network namespace
>>    nfsd: remove nfsd_stats, make th_cnt a global counter
>>    nfsd: make svc_stat per-network namespace instead of global
> 
>>   fs/lockd/svc.c             |  3 --
>>   fs/nfs/callback.c          |  3 --
>>   fs/nfsd/cache.h            |  2 -
>>   fs/nfsd/export.c           | 32 ++++++++++----
>>   fs/nfsd/export.h           |  4 +-
>>   fs/nfsd/netns.h            | 25 +++++++++--
>>   fs/nfsd/nfs4proc.c         |  6 +--
>>   fs/nfsd/nfs4state.c        |  3 +-
>>   fs/nfsd/nfscache.c         | 40 ++++-------------
>>   fs/nfsd/nfsctl.c           | 16 +++----
>>   fs/nfsd/nfsd.h             |  1 +
>>   fs/nfsd/nfsfh.c            |  3 +-
>>   fs/nfsd/nfssvc.c           | 14 +++---
>>   fs/nfsd/stats.c            | 54 ++++++++++-------------
>>   fs/nfsd/stats.h            | 88 ++++++++++++++------------------------
>>   fs/nfsd/vfs.c              |  6 ++-
>>   include/linux/sunrpc/svc.h |  5 ++-
>>   net/sunrpc/stats.c         |  2 +-
>>   net/sunrpc/svc.c           | 39 +++++++++++------
>>   19 files changed, 163 insertions(+), 183 deletions(-)
>
Calum Mackay Aug. 14, 2024, 12:13 p.m. UTC | #4
hi again Petr

On 14/08/2024 11:09 am, Petr Vorel wrote:
> Hi Calum,
> 
>> Hi Petr,
> 
>> There are two sets of changes here, for NFS client, and NFS server.
> 
>> The NFS client changes have already been backported from v6.9 all the way to v5.4.
> 
>> Here, Chuck is discussing the NFS server changes (and others), which were not backported from v6.9 (actually, a few were, but only to v6.8).
> 
> Thanks for info! Now I'll see the patchset "Make nfsd stats visible in network
> ns" [1]. kernelnewbies [2] starts with d98416cc2154 ("nfsd: rename
> NFSD_NET_* to NFSD_STATS_*"), the others are probably some preparation commits.

See also my note:

	https://lore.kernel.org/linux-nfs/2fc3a3fd-7433-45ba-b281-578355dca64c@oracle.com/

> 
> Anyway, I'll update the patch with NFS server patchset.
> 
> Kind regards,
> Petr
> 
> [1] https://lore.kernel.org/linux-nfs/cover.1706283433.git.josef@toxicpanda.com/
> [2] https://kernelnewbies.org/Linux_6.9#File_systems
> 
> 
>> Thanks,
>> Calum.
> 
>> Sent from Outlook for Android<https://aka.ms/AAb9ysg>
>> ________________________________
>> From: Petr Vorel <pvorel@suse.cz>
>> Sent: Wednesday, August 14, 2024 8:45:59 AM
>> To: cel@kernel.org <cel@kernel.org>
>> Cc: stable@vger.kernel.org <stable@vger.kernel.org>; linux-nfs@vger.kernel.org <linux-nfs@vger.kernel.org>; Sherry Yang <sherry.yang@oracle.com>; Calum Mackay <calum.mackay@oracle.com>; kernel-team@fb.com <kernel-team@fb.com>; Chuck Lever III <chuck.lever@oracle.com>; Cyril Hrubis <chrubis@suse.cz>; ltp@lists.linux.it <ltp@lists.linux.it>
>> Subject: Re: [PATCH 6.6.y 00/12] Backport "make svc_stat per-net instead of global"
> 
>> Hi Chuck,
> 
>>> Following up on:
> 
>>> https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/
> 
>>> Here is a backport series targeting origin/linux-6.6.y that closes
>>> the information leak described in the above thread. It passes basic
>>> NFSD regression testing.
> 
> 
>> Thank you for handling this! The link above mentions that it was already
>> backported to 5.4 and indeed I see at least d47151b79e322 ("nfs: expose
>> /proc/net/sunrpc/nfs in net namespaces") is backported in 5.4, 5.10, 5.15, 6.1.
>> And you're now preparing 6.6. Thus we can expect the behavior changed from
>> 5.4 kernels.
> 
>> I wonder if we consider this as a fix, thus expect any kernel newer than 5.4
>> should backport all these 12 patches.
> 
>> Or, whether we should relax and just check if version is higher than the one
>> which got it in stable/LTS (e.g. >= 5.4.276 || >= 5.10.217 ...). The question is
>> also if enterprise distros will take this patchset.
> 
>> BTW We have in LTP functionality which points as a hint to kernel fixes. But
>> it's usually a single commit. I might need to list all.
> 
>> Kind regards,
>> Petr
> 
>>> Review comments welcome.
> 
>>> Chuck Lever (2):
>>>    NFSD: Rewrite synopsis of nfsd_percpu_counters_init()
>>>    NFSD: Fix frame size warning in svc_export_parse()
> 
>>> Josef Bacik (10):
>>>    sunrpc: don't change ->sv_stats if it doesn't exist
>>>    nfsd: stop setting ->pg_stats for unused stats
>>>    sunrpc: pass in the sv_stats struct through svc_create_pooled
>>>    sunrpc: remove ->pg_stats from svc_program
>>>    sunrpc: use the struct net as the svc proc private
>>>    nfsd: rename NFSD_NET_* to NFSD_STATS_*
>>>    nfsd: expose /proc/net/sunrpc/nfsd in net namespaces
>>>    nfsd: make all of the nfsd stats per-network namespace
>>>    nfsd: remove nfsd_stats, make th_cnt a global counter
>>>    nfsd: make svc_stat per-network namespace instead of global
> 
>>>   fs/lockd/svc.c             |  3 --
>>>   fs/nfs/callback.c          |  3 --
>>>   fs/nfsd/cache.h            |  2 -
>>>   fs/nfsd/export.c           | 32 ++++++++++----
>>>   fs/nfsd/export.h           |  4 +-
>>>   fs/nfsd/netns.h            | 25 +++++++++--
>>>   fs/nfsd/nfs4proc.c         |  6 +--
>>>   fs/nfsd/nfs4state.c        |  3 +-
>>>   fs/nfsd/nfscache.c         | 40 ++++-------------
>>>   fs/nfsd/nfsctl.c           | 16 +++----
>>>   fs/nfsd/nfsd.h             |  1 +
>>>   fs/nfsd/nfsfh.c            |  3 +-
>>>   fs/nfsd/nfssvc.c           | 14 +++---
>>>   fs/nfsd/stats.c            | 54 ++++++++++-------------
>>>   fs/nfsd/stats.h            | 88 ++++++++++++++------------------------
>>>   fs/nfsd/vfs.c              |  6 ++-
>>>   include/linux/sunrpc/svc.h |  5 ++-
>>>   net/sunrpc/stats.c         |  2 +-
>>>   net/sunrpc/svc.c           | 39 +++++++++++------
>>>   19 files changed, 163 insertions(+), 183 deletions(-)
>
Calum Mackay Aug. 14, 2024, 12:14 p.m. UTC | #5
Hi Petr,

[resending for the lists, owing to blockage, sorry]

There are two sets of changes here, for NFS client, and NFS server.

The NFS client changes have already been backported from v6.9 all the 
way to v5.4.

Here, Chuck is discussing the NFS server changes (and others), which 
were not backported from v6.9 (actually, a few were, but only to v6.8).

Thanks,
Calum.

On 14/08/2024 8:45 am, Petr Vorel wrote:
> Hi Chuck,
> 
>> Following up on:
> 
>> https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/
> 
>> Here is a backport series targeting origin/linux-6.6.y that closes
>> the information leak described in the above thread. It passes basic
>> NFSD regression testing.
> 
> 
> Thank you for handling this! The link above mentions that it was already
> backported to 5.4 and indeed I see at least d47151b79e322 ("nfs: expose
> /proc/net/sunrpc/nfs in net namespaces") is backported in 5.4, 5.10, 5.15, 6.1.
> And you're now preparing 6.6. Thus we can expect the behavior changed from
> 5.4 kernels.
> 
> I wonder if we consider this as a fix, thus expect any kernel newer than 5.4
> should backport all these 12 patches.
> 
> Or, whether we should relax and just check if version is higher than the one
> which got it in stable/LTS (e.g. >= 5.4.276 || >= 5.10.217 ...). The question is
> also if enterprise distros will take this patchset.
> 
> BTW We have in LTP functionality which points as a hint to kernel fixes. But
> it's usually a single commit. I might need to list all.
> 
> Kind regards,
> Petr
> 
>> Review comments welcome.
> 
>> Chuck Lever (2):
>>    NFSD: Rewrite synopsis of nfsd_percpu_counters_init()
>>    NFSD: Fix frame size warning in svc_export_parse()
> 
>> Josef Bacik (10):
>>    sunrpc: don't change ->sv_stats if it doesn't exist
>>    nfsd: stop setting ->pg_stats for unused stats
>>    sunrpc: pass in the sv_stats struct through svc_create_pooled
>>    sunrpc: remove ->pg_stats from svc_program
>>    sunrpc: use the struct net as the svc proc private
>>    nfsd: rename NFSD_NET_* to NFSD_STATS_*
>>    nfsd: expose /proc/net/sunrpc/nfsd in net namespaces
>>    nfsd: make all of the nfsd stats per-network namespace
>>    nfsd: remove nfsd_stats, make th_cnt a global counter
>>    nfsd: make svc_stat per-network namespace instead of global
> 
>>   fs/lockd/svc.c             |  3 --
>>   fs/nfs/callback.c          |  3 --
>>   fs/nfsd/cache.h            |  2 -
>>   fs/nfsd/export.c           | 32 ++++++++++----
>>   fs/nfsd/export.h           |  4 +-
>>   fs/nfsd/netns.h            | 25 +++++++++--
>>   fs/nfsd/nfs4proc.c         |  6 +--
>>   fs/nfsd/nfs4state.c        |  3 +-
>>   fs/nfsd/nfscache.c         | 40 ++++-------------
>>   fs/nfsd/nfsctl.c           | 16 +++----
>>   fs/nfsd/nfsd.h             |  1 +
>>   fs/nfsd/nfsfh.c            |  3 +-
>>   fs/nfsd/nfssvc.c           | 14 +++---
>>   fs/nfsd/stats.c            | 54 ++++++++++-------------
>>   fs/nfsd/stats.h            | 88 ++++++++++++++------------------------
>>   fs/nfsd/vfs.c              |  6 ++-
>>   include/linux/sunrpc/svc.h |  5 ++-
>>   net/sunrpc/stats.c         |  2 +-
>>   net/sunrpc/svc.c           | 39 +++++++++++------
>>   19 files changed, 163 insertions(+), 183 deletions(-)
>
Greg KH Aug. 15, 2024, 8:25 a.m. UTC | #6
On Mon, Aug 12, 2024 at 06:35:52PM -0400, cel@kernel.org wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> Following up on:
> 
> https://lore.kernel.org/linux-nfs/d4b235df-4ee5-4824-9d48-e3b3c1f1f4d1@oracle.com/
> 
> Here is a backport series targeting origin/linux-6.6.y that closes
> the information leak described in the above thread. It passes basic
> NFSD regression testing.
> 
> Review comments welcome.   

All now queued up, thanks.

greg k-h