mbox series

[v4,0/5] NFSD: Add support for the v4.2 READ_PLUS operation

Message ID 20200817165310.354092-1-Anna.Schumaker@Netapp.com (mailing list archive)
Headers show
Series NFSD: Add support for the v4.2 READ_PLUS operation | expand

Message

Anna Schumaker Aug. 17, 2020, 4:53 p.m. UTC
From: Anna Schumaker <Anna.Schumaker@Netapp.com>

These patches add server support for the READ_PLUS operation, which
breaks read requests into several "data" and "hole" segments when
replying to the client.

- Changes since v3:
  - Combine first two patches related to xdr_reserve_space_vec()
  - Remove unnecessary call to svc_encode_read_payload()

Here are the results of some performance tests I ran on some lab
machines. I tested by reading various 2G files from a few different underlying
filesystems and across several NFS versions. I used the `vmtouch` utility
to make sure files were only cached when we wanted them to be. In addition
to 100% data and 100% hole cases, I also tested with files that alternate
between data and hole segments. These files have either 4K, 8K, 16K, or 32K
segment sizes and start with either data or hole segments. So the file
mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h
has 32K segments beginning with a hole. The units are in seconds, with the
first number for each NFS version being the uncached read time and the second
number is for when the file is cached on the server.

I added some extra data collection (client cpu percentage and sys time),
but the extra data means I couldn't figure out a way to break this down
into a concise table. I cut out v3 and v4.0 performance numbers to get
the size down, but I kept v4.1 for comparison because it uses the same
code that v4.2 without read plus uses.


Read Plus Results (ext4):
  data
   :... v4.1 ... Uncached ... 20.540 s, 105 MB/s, 0.65 s kern, 3% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 20.605 s, 104 MB/s, 0.65 s kern, 3% cpu
        :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu
  hole
   :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.72 s kern,  3% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern,  3% cpu
   :... v4.2 ... Uncached ...  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
        :....... Cached .....  0.845 s, 2.5 GB/s, 0.72 s kern, 85% cpu
  mixed-4d
   :... v4.1 ... Uncached ... 54.691 s,  39 MB/s, 0.75 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
   :... v4.2 ... Uncached ... 51.587 s,  42 MB/s, 0.75 s kern, 1% cpu
        :....... Cached .....  9.215 s, 233 MB/s, 0.67 s kern, 7% cpu
  mixed-8d
   :... v4.1 ... Uncached ... 37.072 s,  58 MB/s, 0.67 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
   :... v4.2 ... Uncached ... 33.259 s,  65 MB/s, 0.68 s kern, 2% cpu
        :....... Cached .....  9.172 s, 234 MB/s, 0.67 s kern, 7% cpu
  mixed-16d
   :... v4.1 ... Uncached ... 27.138 s,  79 MB/s, 0.73 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
   :... v4.2 ... Uncached ... 23.042 s,  93 MB/s, 0.73 s kern, 3% cpu
        :....... Cached .....  9.150 s, 235 MB/s, 0.66 s kern, 7% cpu
  mixed-32d
   :... v4.1 ... Uncached ... 25.326 s,  85 MB/s, 0.68 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 21.125 s, 102 MB/s, 0.69 s kern, 3% cpu
        :....... Cached .....  9.140 s, 235 MB/s, 0.67 s kern, 7% cpu
  mixed-4h
   :... v4.1 ... Uncached ... 58.317 s,  37 MB/s, 0.75 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 51.878 s,  41 MB/s, 0.74 s kern, 1% cpu
        :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
  mixed-8h
   :... v4.1 ... Uncached ... 36.855 s,  58 MB/s, 0.68 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
   :... v4.2 ... Uncached ... 29.457 s,  73 MB/s, 0.68 s kern, 2% cpu
        :....... Cached .....  9.172 s, 234 MB/s, 0.67 s kern, 7% cpu
  mixed-16h
   :... v4.1 ... Uncached ... 26.460 s,  81 MB/s, 0.74 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
   :... v4.2 ... Uncached ... 19.587 s, 110 MB/s, 0.74 s kern, 3% cpu
        :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
  mixed-32h
   :... v4.1 ... Uncached ... 25.495 s,  84 MB/s, 0.69 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
   :... v4.2 ... Uncached ... 17.634 s, 122 MB/s, 0.69 s kern, 3% cpu
        :....... Cached .....  9.140 s, 235 MB/s, 0.68 s kern, 7% cpu



Read Plus Results (xfs):
  data
   :... v4.1 ... Uncached ... 20.230 s, 106 MB/s, 0.65 s kern, 3% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
   :... v4.2 ... Uncached ... 20.724 s, 104 MB/s, 0.65 s kern, 3% cpu
        :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu
  hole
   :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.68 s kern,  3% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern,  3% cpu
   :... v4.2 ... Uncached ...  0.904 s, 2.4 GB/s, 0.72 s kern, 79% cpu
        :....... Cached .....  0.908 s, 2.4 GB/s, 0.73 s kern, 80% cpu
  mixed-4d
   :... v4.1 ... Uncached ... 57.553 s,  37 MB/s, 0.77 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 37.162 s,  58 MB/s, 0.73 s kern, 1% cpu
        :....... Cached .....  9.215 s, 233 MB/s, 0.67 s kern, 7% cpu
  mixed-8d
   :... v4.1 ... Uncached ... 36.754 s,  58 MB/s, 0.69 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
   :... v4.2 ... Uncached ... 24.454 s,  88 MB/s, 0.69 s kern, 2% cpu
        :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
  mixed-16d
   :... v4.1 ... Uncached ... 27.156 s,  79 MB/s, 0.73 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
   :... v4.2 ... Uncached ... 22.934 s,  94 MB/s, 0.72 s kern, 3% cpu
        :....... Cached .....  9.150 s, 235 MB/s, 0.68 s kern, 7% cpu
  mixed-32d
   :... v4.1 ... Uncached ... 27.849 s,  77 MB/s, 0.68 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
   :... v4.2 ... Uncached ... 23.670 s,  91 MB/s, 0.67 s kern, 2% cpu
        :....... Cached .....  9.139 s, 235 MB/s, 0.64 s kern, 7% cpu
  mixed-4h
   :... v4.1 ... Uncached ... 57.639 s,  37 MB/s, 0.72 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern, 3% cpu
   :... v4.2 ... Uncached ... 35.503 s,  61 MB/s, 0.72 s kern, 2% cpu
        :....... Cached .....  9.215 s, 233 MB/s, 0.66 s kern, 7% cpu
  mixed-8h
   :... v4.1 ... Uncached ... 37.044 s,  58 MB/s, 0.71 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
   :... v4.2 ... Uncached ... 23.779 s,  90 MB/s, 0.69 s kern, 2% cpu
        :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
  mixed-16h
   :... v4.1 ... Uncached ... 27.167 s,  79 MB/s, 0.73 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
   :... v4.2 ... Uncached ... 19.088 s, 113 MB/s, 0.75 s kern, 3% cpu
        :....... Cached .....  9.159 s, 234 MB/s, 0.66 s kern, 7% cpu
  mixed-32h
   :... v4.1 ... Uncached ... 27.592 s,  78 MB/s, 0.71 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
   :... v4.2 ... Uncached ... 19.682 s, 109 MB/s, 0.67 s kern, 3% cpu
        :....... Cached .....  9.140 s, 235 MB/s, 0.67 s kern, 7% cpu



Read Plus Results (btrfs):
  data
   :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
   :... v4.2 ... Uncached ... 28.665 s,  75 MB/s, 0.65 s kern, 2% cpu
        :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu
  hole
   :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern,  3% cpu
   :    :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern,  4% cpu
   :... v4.2 ... Uncached ...  0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu
        :....... Cached .....  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
  mixed-4d
   :... v4.1 ... Uncached ... 56.857 s,  38 MB/s, 0.76 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
   :... v4.2 ... Uncached ... 54.455 s,  39 MB/s, 0.73 s kern, 1% cpu
        :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
  mixed-8d
   :... v4.1 ... Uncached ... 36.641 s,  59 MB/s, 0.68 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 33.205 s,  65 MB/s, 0.67 s kern, 2% cpu
        :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
  mixed-16d
   :... v4.1 ... Uncached ... 28.653 s,  75 MB/s, 0.72 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 25.748 s,  83 MB/s, 0.71 s kern, 2% cpu
        :....... Cached .....  9.150 s, 235 MB/s, 0.64 s kern, 7% cpu
  mixed-32d
   :... v4.1 ... Uncached ... 28.886 s,  74 MB/s, 0.67 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
   :... v4.2 ... Uncached ... 24.724 s,  87 MB/s, 0.74 s kern, 2% cpu
        :....... Cached .....  9.140 s, 235 MB/s, 0.63 s kern, 6% cpu
  mixed-4h
   :... v4.1 ... Uncached ...  52.181 s,  41 MB/s, 0.73 s kern, 1% cpu
   :    :....... Cached .....  18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
   :... v4.2 ... Uncached ... 150.341 s,  14 MB/s, 0.72 s kern, 0% cpu
        :....... Cached .....   9.216 s, 233 MB/s, 0.63 s kern, 6% cpu
  mixed-8h
   :... v4.1 ... Uncached ... 36.945 s,  58 MB/s, 0.68 s kern, 1% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
   :... v4.2 ... Uncached ... 79.781 s,  27 MB/s, 0.68 s kern, 0% cpu
        :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
  mixed-16h
   :... v4.1 ... Uncached ... 28.651 s,  75 MB/s, 0.73 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
   :... v4.2 ... Uncached ... 47.428 s,  45 MB/s, 0.71 s kern, 1% cpu
        :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
  mixed-32h
   :... v4.1 ... Uncached ... 28.618 s,  75 MB/s, 0.69 s kern, 2% cpu
   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
   :... v4.2 ... Uncached ... 38.813 s,  55 MB/s, 0.67 s kern, 1% cpu
        :....... Cached .....  9.140 s, 235 MB/s, 0.61 s kern, 6% cpu



Thoughts?
Anna


Anna Schumaker (5):
  SUNRPC/NFSD: Implement xdr_reserve_space_vec()
  NFSD: Add READ_PLUS data support
  NFSD: Add READ_PLUS hole segment encoding
  NFSD: Return both a hole and a data segment
  NFSD: Encode a full READ_PLUS reply

 fs/nfsd/nfs4proc.c         |  17 ++++
 fs/nfsd/nfs4xdr.c          | 167 +++++++++++++++++++++++++++++++------
 include/linux/sunrpc/xdr.h |   2 +
 net/sunrpc/xdr.c           |  45 ++++++++++
 4 files changed, 204 insertions(+), 27 deletions(-)

Comments

Chuck Lever Aug. 19, 2020, 5:07 p.m. UTC | #1
> On Aug 17, 2020, at 12:53 PM, schumaker.anna@gmail.com wrote:
> 
> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> 
> These patches add server support for the READ_PLUS operation, which
> breaks read requests into several "data" and "hole" segments when
> replying to the client.
> 
> - Changes since v3:
>  - Combine first two patches related to xdr_reserve_space_vec()
>  - Remove unnecessary call to svc_encode_read_payload()

My vote is let's merge v3 and continue refining.


> Here are the results of some performance tests I ran on some lab
> machines. I tested by reading various 2G files from a few different underlying
> filesystems and across several NFS versions. I used the `vmtouch` utility
> to make sure files were only cached when we wanted them to be. In addition
> to 100% data and 100% hole cases, I also tested with files that alternate
> between data and hole segments. These files have either 4K, 8K, 16K, or 32K
> segment sizes and start with either data or hole segments. So the file
> mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h
> has 32K segments beginning with a hole. The units are in seconds, with the
> first number for each NFS version being the uncached read time and the second
> number is for when the file is cached on the server.
> 
> I added some extra data collection (client cpu percentage and sys time),
> but the extra data means I couldn't figure out a way to break this down
> into a concise table. I cut out v3 and v4.0 performance numbers to get
> the size down, but I kept v4.1 for comparison because it uses the same
> code that v4.2 without read plus uses.
> 
> 
> Read Plus Results (ext4):
>  data
>   :... v4.1 ... Uncached ... 20.540 s, 105 MB/s, 0.65 s kern, 3% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 20.605 s, 104 MB/s, 0.65 s kern, 3% cpu
>        :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu
>  hole
>   :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.72 s kern,  3% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern,  3% cpu
>   :... v4.2 ... Uncached ...  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
>        :....... Cached .....  0.845 s, 2.5 GB/s, 0.72 s kern, 85% cpu
>  mixed-4d
>   :... v4.1 ... Uncached ... 54.691 s,  39 MB/s, 0.75 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 51.587 s,  42 MB/s, 0.75 s kern, 1% cpu
>        :....... Cached .....  9.215 s, 233 MB/s, 0.67 s kern, 7% cpu
>  mixed-8d
>   :... v4.1 ... Uncached ... 37.072 s,  58 MB/s, 0.67 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 33.259 s,  65 MB/s, 0.68 s kern, 2% cpu
>        :....... Cached .....  9.172 s, 234 MB/s, 0.67 s kern, 7% cpu
>  mixed-16d
>   :... v4.1 ... Uncached ... 27.138 s,  79 MB/s, 0.73 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 23.042 s,  93 MB/s, 0.73 s kern, 3% cpu
>        :....... Cached .....  9.150 s, 235 MB/s, 0.66 s kern, 7% cpu
>  mixed-32d
>   :... v4.1 ... Uncached ... 25.326 s,  85 MB/s, 0.68 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 21.125 s, 102 MB/s, 0.69 s kern, 3% cpu
>        :....... Cached .....  9.140 s, 235 MB/s, 0.67 s kern, 7% cpu
>  mixed-4h
>   :... v4.1 ... Uncached ... 58.317 s,  37 MB/s, 0.75 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 51.878 s,  41 MB/s, 0.74 s kern, 1% cpu
>        :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
>  mixed-8h
>   :... v4.1 ... Uncached ... 36.855 s,  58 MB/s, 0.68 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 29.457 s,  73 MB/s, 0.68 s kern, 2% cpu
>        :....... Cached .....  9.172 s, 234 MB/s, 0.67 s kern, 7% cpu
>  mixed-16h
>   :... v4.1 ... Uncached ... 26.460 s,  81 MB/s, 0.74 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 19.587 s, 110 MB/s, 0.74 s kern, 3% cpu
>        :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
>  mixed-32h
>   :... v4.1 ... Uncached ... 25.495 s,  84 MB/s, 0.69 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 17.634 s, 122 MB/s, 0.69 s kern, 3% cpu
>        :....... Cached .....  9.140 s, 235 MB/s, 0.68 s kern, 7% cpu
> 
> 
> 
> Read Plus Results (xfs):
>  data
>   :... v4.1 ... Uncached ... 20.230 s, 106 MB/s, 0.65 s kern, 3% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 20.724 s, 104 MB/s, 0.65 s kern, 3% cpu
>        :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu
>  hole
>   :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.68 s kern,  3% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern,  3% cpu
>   :... v4.2 ... Uncached ...  0.904 s, 2.4 GB/s, 0.72 s kern, 79% cpu
>        :....... Cached .....  0.908 s, 2.4 GB/s, 0.73 s kern, 80% cpu
>  mixed-4d
>   :... v4.1 ... Uncached ... 57.553 s,  37 MB/s, 0.77 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 37.162 s,  58 MB/s, 0.73 s kern, 1% cpu
>        :....... Cached .....  9.215 s, 233 MB/s, 0.67 s kern, 7% cpu
>  mixed-8d
>   :... v4.1 ... Uncached ... 36.754 s,  58 MB/s, 0.69 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 24.454 s,  88 MB/s, 0.69 s kern, 2% cpu
>        :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
>  mixed-16d
>   :... v4.1 ... Uncached ... 27.156 s,  79 MB/s, 0.73 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 22.934 s,  94 MB/s, 0.72 s kern, 3% cpu
>        :....... Cached .....  9.150 s, 235 MB/s, 0.68 s kern, 7% cpu
>  mixed-32d
>   :... v4.1 ... Uncached ... 27.849 s,  77 MB/s, 0.68 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 23.670 s,  91 MB/s, 0.67 s kern, 2% cpu
>        :....... Cached .....  9.139 s, 235 MB/s, 0.64 s kern, 7% cpu
>  mixed-4h
>   :... v4.1 ... Uncached ... 57.639 s,  37 MB/s, 0.72 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 35.503 s,  61 MB/s, 0.72 s kern, 2% cpu
>        :....... Cached .....  9.215 s, 233 MB/s, 0.66 s kern, 7% cpu
>  mixed-8h
>   :... v4.1 ... Uncached ... 37.044 s,  58 MB/s, 0.71 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 23.779 s,  90 MB/s, 0.69 s kern, 2% cpu
>        :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
>  mixed-16h
>   :... v4.1 ... Uncached ... 27.167 s,  79 MB/s, 0.73 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 19.088 s, 113 MB/s, 0.75 s kern, 3% cpu
>        :....... Cached .....  9.159 s, 234 MB/s, 0.66 s kern, 7% cpu
>  mixed-32h
>   :... v4.1 ... Uncached ... 27.592 s,  78 MB/s, 0.71 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 19.682 s, 109 MB/s, 0.67 s kern, 3% cpu
>        :....... Cached .....  9.140 s, 235 MB/s, 0.67 s kern, 7% cpu
> 
> 
> 
> Read Plus Results (btrfs):
>  data
>   :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 28.665 s,  75 MB/s, 0.65 s kern, 2% cpu
>        :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu
>  hole
>   :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern,  3% cpu
>   :    :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern,  4% cpu
>   :... v4.2 ... Uncached ...  0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu
>        :....... Cached .....  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
>  mixed-4d
>   :... v4.1 ... Uncached ... 56.857 s,  38 MB/s, 0.76 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 54.455 s,  39 MB/s, 0.73 s kern, 1% cpu
>        :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
>  mixed-8d
>   :... v4.1 ... Uncached ... 36.641 s,  59 MB/s, 0.68 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 33.205 s,  65 MB/s, 0.67 s kern, 2% cpu
>        :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
>  mixed-16d
>   :... v4.1 ... Uncached ... 28.653 s,  75 MB/s, 0.72 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 25.748 s,  83 MB/s, 0.71 s kern, 2% cpu
>        :....... Cached .....  9.150 s, 235 MB/s, 0.64 s kern, 7% cpu
>  mixed-32d
>   :... v4.1 ... Uncached ... 28.886 s,  74 MB/s, 0.67 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 24.724 s,  87 MB/s, 0.74 s kern, 2% cpu
>        :....... Cached .....  9.140 s, 235 MB/s, 0.63 s kern, 6% cpu
>  mixed-4h
>   :... v4.1 ... Uncached ...  52.181 s,  41 MB/s, 0.73 s kern, 1% cpu
>   :    :....... Cached .....  18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 150.341 s,  14 MB/s, 0.72 s kern, 0% cpu
>        :....... Cached .....   9.216 s, 233 MB/s, 0.63 s kern, 6% cpu
>  mixed-8h
>   :... v4.1 ... Uncached ... 36.945 s,  58 MB/s, 0.68 s kern, 1% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 79.781 s,  27 MB/s, 0.68 s kern, 0% cpu
>        :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
>  mixed-16h
>   :... v4.1 ... Uncached ... 28.651 s,  75 MB/s, 0.73 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 47.428 s,  45 MB/s, 0.71 s kern, 1% cpu
>        :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
>  mixed-32h
>   :... v4.1 ... Uncached ... 28.618 s,  75 MB/s, 0.69 s kern, 2% cpu
>   :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>   :... v4.2 ... Uncached ... 38.813 s,  55 MB/s, 0.67 s kern, 1% cpu
>        :....... Cached .....  9.140 s, 235 MB/s, 0.61 s kern, 6% cpu
> 
> 
> 
> Thoughts?
> Anna
> 
> 
> Anna Schumaker (5):
>  SUNRPC/NFSD: Implement xdr_reserve_space_vec()
>  NFSD: Add READ_PLUS data support
>  NFSD: Add READ_PLUS hole segment encoding
>  NFSD: Return both a hole and a data segment
>  NFSD: Encode a full READ_PLUS reply
> 
> fs/nfsd/nfs4proc.c         |  17 ++++
> fs/nfsd/nfs4xdr.c          | 167 +++++++++++++++++++++++++++++++------
> include/linux/sunrpc/xdr.h |   2 +
> net/sunrpc/xdr.c           |  45 ++++++++++
> 4 files changed, 204 insertions(+), 27 deletions(-)
> 
> -- 
> 2.28.0
> 

--
Chuck Lever
J. Bruce Fields Aug. 26, 2020, 9:54 p.m. UTC | #2
On Mon, Aug 17, 2020 at 12:53:05PM -0400, schumaker.anna@gmail.com wrote:
> From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> 
> These patches add server support for the READ_PLUS operation, which
> breaks read requests into several "data" and "hole" segments when
> replying to the client.
> 
> - Changes since v3:
>   - Combine first two patches related to xdr_reserve_space_vec()
>   - Remove unnecessary call to svc_encode_read_payload()
> 
> Here are the results of some performance tests I ran on some lab
> machines.

What's the hardware setup (do you know network and disk bandwidth?).

> I tested by reading various 2G files from a few different underlying
> filesystems and across several NFS versions. I used the `vmtouch` utility
> to make sure files were only cached when we wanted them to be. In addition
> to 100% data and 100% hole cases, I also tested with files that alternate
> between data and hole segments. These files have either 4K, 8K, 16K, or 32K
> segment sizes and start with either data or hole segments. So the file
> mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h
> has 32K segments beginning with a hole. The units are in seconds, with the
> first number for each NFS version being the uncached read time and the second
> number is for when the file is cached on the server.

The only numbers that look really strange are in the btrfs uncached
case, in the data-only case and the mixed case that start with a hole.
Do we have any idea what's up there?

--b.

> Read Plus Results (btrfs):
>   data
>    :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 28.665 s,  75 MB/s, 0.65 s kern, 2% cpu
>         :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu
>   hole
>    :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern,  3% cpu
>    :    :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern,  4% cpu
>    :... v4.2 ... Uncached ...  0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu
>         :....... Cached .....  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
>   mixed-4d
>    :... v4.1 ... Uncached ... 56.857 s,  38 MB/s, 0.76 s kern, 1% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 54.455 s,  39 MB/s, 0.73 s kern, 1% cpu
>         :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
>   mixed-8d
>    :... v4.1 ... Uncached ... 36.641 s,  59 MB/s, 0.68 s kern, 1% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 33.205 s,  65 MB/s, 0.67 s kern, 2% cpu
>         :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
>   mixed-16d
>    :... v4.1 ... Uncached ... 28.653 s,  75 MB/s, 0.72 s kern, 2% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 25.748 s,  83 MB/s, 0.71 s kern, 2% cpu
>         :....... Cached .....  9.150 s, 235 MB/s, 0.64 s kern, 7% cpu
>   mixed-32d
>    :... v4.1 ... Uncached ... 28.886 s,  74 MB/s, 0.67 s kern, 2% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 24.724 s,  87 MB/s, 0.74 s kern, 2% cpu
>         :....... Cached .....  9.140 s, 235 MB/s, 0.63 s kern, 6% cpu
>   mixed-4h
>    :... v4.1 ... Uncached ...  52.181 s,  41 MB/s, 0.73 s kern, 1% cpu
>    :    :....... Cached .....  18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 150.341 s,  14 MB/s, 0.72 s kern, 0% cpu
>         :....... Cached .....   9.216 s, 233 MB/s, 0.63 s kern, 6% cpu
>   mixed-8h
>    :... v4.1 ... Uncached ... 36.945 s,  58 MB/s, 0.68 s kern, 1% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 79.781 s,  27 MB/s, 0.68 s kern, 0% cpu
>         :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
>   mixed-16h
>    :... v4.1 ... Uncached ... 28.651 s,  75 MB/s, 0.73 s kern, 2% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 47.428 s,  45 MB/s, 0.71 s kern, 1% cpu
>         :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
>   mixed-32h
>    :... v4.1 ... Uncached ... 28.618 s,  75 MB/s, 0.69 s kern, 2% cpu
>    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
>    :... v4.2 ... Uncached ... 38.813 s,  55 MB/s, 0.67 s kern, 1% cpu
>         :....... Cached .....  9.140 s, 235 MB/s, 0.61 s kern, 6% cpu
Anna Schumaker Aug. 31, 2020, 6:33 p.m. UTC | #3
On Wed, Aug 26, 2020 at 5:54 PM J. Bruce Fields <bfields@redhat.com> wrote:
>
> On Mon, Aug 17, 2020 at 12:53:05PM -0400, schumaker.anna@gmail.com wrote:
> > From: Anna Schumaker <Anna.Schumaker@Netapp.com>
> >
> > These patches add server support for the READ_PLUS operation, which
> > breaks read requests into several "data" and "hole" segments when
> > replying to the client.
> >
> > - Changes since v3:
> >   - Combine first two patches related to xdr_reserve_space_vec()
> >   - Remove unnecessary call to svc_encode_read_payload()
> >
> > Here are the results of some performance tests I ran on some lab
> > machines.
>
> What's the hardware setup (do you know network and disk bandwidth?).

I used iperf to benchmark the network, and it said it transferred 1.10
GBytes with a bandwidth of 941 Mbits/sec

I ran hdparm -tT to benchmark reads on the disk and it said this:
Timing cached reads:   13394 MB in  2.00 seconds = 6713.72 MB/sec
Timing buffered disk reads: 362 MB in 3.00 seconds = 120.60 MB/sec

>
> > I tested by reading various 2G files from a few different underlying
> > filesystems and across several NFS versions. I used the `vmtouch` utility
> > to make sure files were only cached when we wanted them to be. In addition
> > to 100% data and 100% hole cases, I also tested with files that alternate
> > between data and hole segments. These files have either 4K, 8K, 16K, or 32K
> > segment sizes and start with either data or hole segments. So the file
> > mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h
> > has 32K segments beginning with a hole. The units are in seconds, with the
> > first number for each NFS version being the uncached read time and the second
> > number is for when the file is cached on the server.
>
> The only numbers that look really strange are in the btrfs uncached
> case, in the data-only case and the mixed case that start with a hole.
> Do we have any idea what's up there?

I'm not really sure. BTRFS does some work to make sure the page cache
is synced up with their internal extent representation as part of
llseek, so my guess is something related to that (But it's been a
while since I looked into that code, so I'm not sure if that's still
how it works)

Anna

>
> --b.
>
> > Read Plus Results (btrfs):
> >   data
> >    :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 28.665 s,  75 MB/s, 0.65 s kern, 2% cpu
> >         :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu
> >   hole
> >    :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern,  3% cpu
> >    :    :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern,  4% cpu
> >    :... v4.2 ... Uncached ...  0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu
> >         :....... Cached .....  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
> >   mixed-4d
> >    :... v4.1 ... Uncached ... 56.857 s,  38 MB/s, 0.76 s kern, 1% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 54.455 s,  39 MB/s, 0.73 s kern, 1% cpu
> >         :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
> >   mixed-8d
> >    :... v4.1 ... Uncached ... 36.641 s,  59 MB/s, 0.68 s kern, 1% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 33.205 s,  65 MB/s, 0.67 s kern, 2% cpu
> >         :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
> >   mixed-16d
> >    :... v4.1 ... Uncached ... 28.653 s,  75 MB/s, 0.72 s kern, 2% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 25.748 s,  83 MB/s, 0.71 s kern, 2% cpu
> >         :....... Cached .....  9.150 s, 235 MB/s, 0.64 s kern, 7% cpu
> >   mixed-32d
> >    :... v4.1 ... Uncached ... 28.886 s,  74 MB/s, 0.67 s kern, 2% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 24.724 s,  87 MB/s, 0.74 s kern, 2% cpu
> >         :....... Cached .....  9.140 s, 235 MB/s, 0.63 s kern, 6% cpu
> >   mixed-4h
> >    :... v4.1 ... Uncached ...  52.181 s,  41 MB/s, 0.73 s kern, 1% cpu
> >    :    :....... Cached .....  18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 150.341 s,  14 MB/s, 0.72 s kern, 0% cpu
> >         :....... Cached .....   9.216 s, 233 MB/s, 0.63 s kern, 6% cpu
> >   mixed-8h
> >    :... v4.1 ... Uncached ... 36.945 s,  58 MB/s, 0.68 s kern, 1% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 79.781 s,  27 MB/s, 0.68 s kern, 0% cpu
> >         :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
> >   mixed-16h
> >    :... v4.1 ... Uncached ... 28.651 s,  75 MB/s, 0.73 s kern, 2% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 47.428 s,  45 MB/s, 0.71 s kern, 1% cpu
> >         :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
> >   mixed-32h
> >    :... v4.1 ... Uncached ... 28.618 s,  75 MB/s, 0.69 s kern, 2% cpu
> >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
> >    :... v4.2 ... Uncached ... 38.813 s,  55 MB/s, 0.67 s kern, 1% cpu
> >         :....... Cached .....  9.140 s, 235 MB/s, 0.61 s kern, 6% cpu
>
J. Bruce Fields Sept. 4, 2020, 3:56 p.m. UTC | #4
On Mon, Aug 31, 2020 at 02:33:30PM -0400, Anna Schumaker wrote:
> On Wed, Aug 26, 2020 at 5:54 PM J. Bruce Fields <bfields@redhat.com> wrote:
> > On Mon, Aug 17, 2020 at 12:53:05PM -0400, schumaker.anna@gmail.com wrote:
> > > I tested by reading various 2G files from a few different underlying
> > > filesystems and across several NFS versions. I used the `vmtouch` utility
> > > to make sure files were only cached when we wanted them to be. In addition
> > > to 100% data and 100% hole cases, I also tested with files that alternate
> > > between data and hole segments. These files have either 4K, 8K, 16K, or 32K
> > > segment sizes and start with either data or hole segments. So the file
> > > mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h
> > > has 32K segments beginning with a hole. The units are in seconds, with the
> > > first number for each NFS version being the uncached read time and the second
> > > number is for when the file is cached on the server.
> >
> > The only numbers that look really strange are in the btrfs uncached
> > case, in the data-only case and the mixed case that start with a hole.
> > Do we have any idea what's up there?
> 
> I'm not really sure. BTRFS does some work to make sure the page cache
> is synced up with their internal extent representation as part of
> llseek, so my guess is something related to that (But it's been a
> while since I looked into that code, so I'm not sure if that's still
> how it works)

Adding linux-btrfs in case they have any updates--are btrfs developers
aware of known performances issues with SEEK_HOLE/SEEK_DATA, and is it
something anyone's working on?

Anna's implementing a read optimization where the server uses seek to
identify holes to save transmitting all those zeroes back to the client,
and it's working as expected for ext4 and xfs but performing weirdly for
btrfs.

Original message:
	https://lore.kernel.org/linux-nfs/20200817165310.354092-1-Anna.Schumaker@Netapp.com/

--b.


> > > Read Plus Results (btrfs):
> > >   data
> > >    :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 28.665 s,  75 MB/s, 0.65 s kern, 2% cpu
> > >         :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu
> > >   hole
> > >    :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern,  3% cpu
> > >    :    :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern,  4% cpu
> > >    :... v4.2 ... Uncached ...  0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu
> > >         :....... Cached .....  0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu
> > >   mixed-4d
> > >    :... v4.1 ... Uncached ... 56.857 s,  38 MB/s, 0.76 s kern, 1% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 54.455 s,  39 MB/s, 0.73 s kern, 1% cpu
> > >         :....... Cached .....  9.215 s, 233 MB/s, 0.68 s kern, 7% cpu
> > >   mixed-8d
> > >    :... v4.1 ... Uncached ... 36.641 s,  59 MB/s, 0.68 s kern, 1% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 33.205 s,  65 MB/s, 0.67 s kern, 2% cpu
> > >         :....... Cached .....  9.172 s, 234 MB/s, 0.65 s kern, 7% cpu
> > >   mixed-16d
> > >    :... v4.1 ... Uncached ... 28.653 s,  75 MB/s, 0.72 s kern, 2% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 25.748 s,  83 MB/s, 0.71 s kern, 2% cpu
> > >         :....... Cached .....  9.150 s, 235 MB/s, 0.64 s kern, 7% cpu
> > >   mixed-32d
> > >    :... v4.1 ... Uncached ... 28.886 s,  74 MB/s, 0.67 s kern, 2% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 24.724 s,  87 MB/s, 0.74 s kern, 2% cpu
> > >         :....... Cached .....  9.140 s, 235 MB/s, 0.63 s kern, 6% cpu
> > >   mixed-4h
> > >    :... v4.1 ... Uncached ...  52.181 s,  41 MB/s, 0.73 s kern, 1% cpu
> > >    :    :....... Cached .....  18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 150.341 s,  14 MB/s, 0.72 s kern, 0% cpu
> > >         :....... Cached .....   9.216 s, 233 MB/s, 0.63 s kern, 6% cpu
> > >   mixed-8h
> > >    :... v4.1 ... Uncached ... 36.945 s,  58 MB/s, 0.68 s kern, 1% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 79.781 s,  27 MB/s, 0.68 s kern, 0% cpu
> > >         :....... Cached .....  9.172 s, 234 MB/s, 0.66 s kern, 7% cpu
> > >   mixed-16h
> > >    :... v4.1 ... Uncached ... 28.651 s,  75 MB/s, 0.73 s kern, 2% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 47.428 s,  45 MB/s, 0.71 s kern, 1% cpu
> > >         :....... Cached .....  9.150 s, 235 MB/s, 0.67 s kern, 7% cpu
> > >   mixed-32h
> > >    :... v4.1 ... Uncached ... 28.618 s,  75 MB/s, 0.69 s kern, 2% cpu
> > >    :    :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu
> > >    :... v4.2 ... Uncached ... 38.813 s,  55 MB/s, 0.67 s kern, 1% cpu
> > >         :....... Cached .....  9.140 s, 235 MB/s, 0.61 s kern, 6% cpu
> >