mbox series

[V2,0/2] null_blk: usr memcpy_[to|from]_page()

Message ID 20230330184926.64209-1-kch@nvidia.com (mailing list archive)
Headers show
Series null_blk: usr memcpy_[to|from]_page() | expand

Message

Chaitanya Kulkarni March 30, 2023, 6:49 p.m. UTC 
Hi,

From :include/linux/highmem.h:
"kmap_atomic - Atomically map a page for temporary usage - Deprecated!"

Use memcpy_from_page() since does the same job of mapping, copying, and
unmaping except it uses non deprecated kmap_local_page() and
kunmap_local(). Following are the differences between kmal_local_page()
and kmap_atomic() :-

* creates local mapping per thread, local to CPU & not globally visible
* allows to be called from any context
* allows task preemption 

Below is the test for the fio verification job and perf numbers on nullb
along with the test log with debug patch to cover all the calls that are
modified in this series:-
copy_to_nullb()
	memcpy_to_page()
copy_from_nullb()
	memcpy_to_page()
	zero_user()
null_fill_pattern()
	memset_page()
null_flush_cache_page()
	kmap_local_page()
	kunmap_local_page()

* Avg IOPs delta (higher is better) :- ~5k higher with this patch seires
default-nullb.1.fio:           read: IOPS=1050k, BW=4101MiB/s (4300MB/s)(240GiB/60001msec)
default-nullb.2.fio:           read: IOPS=1049k, BW=4096MiB/s (4295MB/s)(240GiB/60001msec)
default-nullb.3.fio:           read: IOPS=1051k, BW=4105MiB/s (4304MB/s)(241GiB/60001msec)
(1050+1049+1051)/3 = 1050

with-memcpy-page-nullb.1.fio:  read: IOPS=1057k, BW=4129MiB/s (4330MB/s)(242GiB/60001msec)
with-memcpy-page-nullb.2.fio:  read: IOPS=1057k, BW=4127MiB/s (4328MB/s)(242GiB/60001msec)
with-memcpy-page-nullb.3.fio:  read: IOPS=1053k, BW=4114MiB/s (4314MB/s)(241GiB/60002msec)
(1057+1057+1053)/3 = 1055

In case someone shows up with performance regression on the arch that
I've don't have access to we can decide then if we want to drop it this
series or keep using deprecated kernel API, but I think removing
deprecated API is useful in long term in anyway.

-ck

v1->v2:

* Consolidate patch 1-3 in a single patch.
* Rebase/retest.

Chaitanya Kulkarni (2):
  null_blk: use non-deprecated lib functions
  null_blk: use kmap_local_page() and kunmap_local()

 drivers/block/null_blk/main.c | 37 +++++++++++------------------------
 1 file changed, 11 insertions(+), 26 deletions(-)

linux-block (null-memcpy-page) # ./compile_nullb.sh 
+ umount /mnt/nullb0
umount: /mnt/nullb0: not mounted.
+ rmdir config/nullb/nullb0
+ dmesg -c
+ modprobe -r null_blk
+ lsmod
+ grep null_blk
++ nproc
+ make -j 48 M=drivers/block modules
+ HOST=drivers/block/null_blk/
++ uname -r
+ HOST_DEST=/lib/modules/6.3.0-rc1lblk+/kernel/drivers/block/null_blk/
+ cp drivers/block/null_blk//null_blk.ko /lib/modules/6.3.0-rc1lblk+/kernel/drivers/block/null_blk//
+ ls -lrth /lib/modules/6.3.0-rc1lblk+/kernel/drivers/block/null_blk//null_blk.ko
-rw-r--r--. 1 root root 1.1M Mar 29 23:04 /lib/modules/6.3.0-rc1lblk+/kernel/drivers/block/null_blk//null_blk.ko
+ sleep 1
+ dmesg -c
linux-block (null-memcpy-page) # modprobe  -r null_blk 
linux-block (null-memcpy-page) # modprobe null_blk gb=1 memory_backed=1 zoned=1
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null bs=4k count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000217633 s, 18.8 MB/s
linux-block (null-memcpy-page) # blkzone report /dev/nullb0
  start: 0x000000000, len 0x080000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
  start: 0x000080000, len 0x080000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
  start: 0x000100000, len 0x080000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
  start: 0x000180000, len 0x080000, wptr 0x000000 reset:0 non-seq:0, zcond: 1(em) [type: 2(SEQ_WRITE_REQUIRED)]
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null count=10 bs=4k
10+0 records in
10+0 records out
40960 bytes (41 kB, 40 KiB) copied, 0.000409999 s, 99.9 MB/s
linux-block (null-memcpy-page) # dd if=/dev/zero of=/dev/nullb0 count=1 bs=4k oflag=direct
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000240897 s, 17.0 MB/s
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null count=1 bs=4k
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000291694 s, 14.0 MB/s
linux-block (null-memcpy-page) # dmesg  -c  
[30856.801950] blk_queue_max_hw_sectors: set to minimum 8
[30856.803437] null_blk: nullb_fill_pattern 1175 memset_page()
[30856.803484] null_blk: nullb_fill_pattern 1175 memset_page()
[30856.803514] null_blk: disk nullb0 created
[30856.803516] null_blk: module loaded
[30860.129178] null_blk: nullb_fill_pattern 1175 memset_page()
[30860.129194] null_blk: nullb_fill_pattern 1175 memset_page()
[30860.129197] null_blk: nullb_fill_pattern 1175 memset_page()
[30860.129199] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063243] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063258] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063261] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063264] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063324] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063330] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063333] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063338] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063340] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063342] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063346] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063349] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063394] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063399] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063401] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063405] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063408] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063413] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063415] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063419] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063421] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063426] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063428] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063444] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063447] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063448] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063453] null_blk: nullb_fill_pattern 1175 memset_page()
[30869.063455] null_blk: nullb_fill_pattern 1175 memset_page()
[30874.238091] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238099] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238101] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238102] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238103] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238104] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238105] null_blk: copy_to_nullb 1129 memcpy_page()
[30874.238106] null_blk: copy_to_nullb 1129 memcpy_page()
[30877.398604] null_blk: nullb_fill_pattern 1175 memset_page()
[30877.398619] null_blk: nullb_fill_pattern 1175 memset_page()
[30877.398622] null_blk: nullb_fill_pattern 1175 memset_page()
[30877.398626] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398627] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398628] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398629] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398630] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398631] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398632] null_blk: copy_from_nullb 1160 memcpy_page()
[30877.398633] null_blk: copy_from_nullb 1160 memcpy_page()
linux-block (null-memcpy-page) # modprobe  -r null_blk
linux-block (null-memcpy-page) # modprobe null_blk gb=1 memory_backed=1 cache_size=1
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null bs=4k count=1 
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000258311 s, 15.9 MB/s
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null bs=4k count=1 
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000250157 s, 16.4 MB/s
linux-block (null-memcpy-page) # modprobe  -r null_blk
linux-block (null-memcpy-page) # dmesg  -c  
[30891.505589] blk_queue_max_hw_sectors: set to minimum 8
[30891.506898] null_blk: copy_from_nullb 1163 zero_user()
[30891.506901] null_blk: copy_from_nullb 1163 zero_user()
[30891.506901] null_blk: copy_from_nullb 1163 zero_user()
[30891.506902] null_blk: copy_from_nullb 1163 zero_user()
[30891.506903] null_blk: copy_from_nullb 1163 zero_user()
[30891.506903] null_blk: copy_from_nullb 1163 zero_user()
[30891.506904] null_blk: copy_from_nullb 1163 zero_user()
[30891.506904] null_blk: copy_from_nullb 1163 zero_user()
[30891.506914] null_blk: copy_from_nullb 1163 zero_user()
[30891.506914] null_blk: copy_from_nullb 1163 zero_user()
[30891.506915] null_blk: copy_from_nullb 1163 zero_user()
[30891.506915] null_blk: copy_from_nullb 1163 zero_user()
[30891.506916] null_blk: copy_from_nullb 1163 zero_user()
[30891.506916] null_blk: copy_from_nullb 1163 zero_user()
[30891.506917] null_blk: copy_from_nullb 1163 zero_user()
[30891.506917] null_blk: copy_from_nullb 1163 zero_user()
[30891.506936] null_blk: disk nullb0 created
[30891.506937] null_blk: module loaded
[30894.734835] null_blk: copy_from_nullb 1163 zero_user()
[30894.734843] null_blk: copy_from_nullb 1163 zero_user()
[30894.734845] null_blk: copy_from_nullb 1163 zero_user()
[30894.734854] null_blk: copy_from_nullb 1163 zero_user()
[30897.718799] null_blk: copy_from_nullb 1163 zero_user()
[30897.718800] null_blk: copy_from_nullb 1163 zero_user()
[30897.718801] null_blk: copy_from_nullb 1163 zero_user()
[30897.718802] null_blk: copy_from_nullb 1163 zero_user()
[30897.718803] null_blk: copy_from_nullb 1163 zero_user()
[30897.718804] null_blk: copy_from_nullb 1163 zero_user()
[30897.718806] null_blk: copy_from_nullb 1163 zero_user()
linux-block (null-memcpy-page) # 
linux-block (null-memcpy-page) # 
linux-block (null-memcpy-page) # 
linux-block (null-memcpy-page) # modprobe null_blk gb=1 cache_size=1 memory_backed=1
linux-block (null-memcpy-page) # dd if=/dev/zero of=/dev/nullb0 count=$((2*1024*1024/4096)) bs=4k
512+0 records in
512+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00729801 s, 287 MB/s
linux-block (null-memcpy-page) # dmesg  -c
[30915.665977] blk_queue_max_hw_sectors: set to minimum 8
[30915.667332] null_blk: copy_from_nullb 1163 zero_user()
[30915.667334] null_blk: copy_from_nullb 1163 zero_user()
[30915.667335] null_blk: copy_from_nullb 1163 zero_user()
[30915.667336] null_blk: copy_from_nullb 1163 zero_user()
[30915.667336] null_blk: copy_from_nullb 1163 zero_user()
[30915.667337] null_blk: copy_from_nullb 1163 zero_user()
[30915.667337] null_blk: copy_from_nullb 1163 zero_user()
[30915.667338] null_blk: copy_from_nullb 1163 zero_user()
[30915.667347] null_blk: copy_from_nullb 1163 zero_user()
[30915.667347] null_blk: copy_from_nullb 1163 zero_user()
[30915.667348] null_blk: copy_from_nullb 1163 zero_user()
[30915.667348] null_blk: copy_from_nullb 1163 zero_user()
[30915.667349] null_blk: copy_from_nullb 1163 zero_user()
[30915.667349] null_blk: copy_from_nullb 1163 zero_user()
[30915.667350] null_blk: copy_from_nullb 1163 zero_user()
[30915.667350] null_blk: copy_from_nullb 1163 zero_user()
[30915.667372] null_blk: disk nullb0 created
[30915.667373] null_blk: module loaded
[30919.905781] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.905786] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.905788] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.905788] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.905793] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.905794] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911028] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911030] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911032] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911033] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911035] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911039] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911040] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911042] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911043] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911044] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911046] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911047] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911050] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911051] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911052] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911054] null_blk: null_flush_cache_page 1033 kmap_local_page()
[30919.911055] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911056] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911057] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911058] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911058] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911059] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911060] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911062] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911063] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911063] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911064] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911065] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911066] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911066] null_blk: copy_to_nullb 1129 memcpy_page()
[30919.911067] null_blk: copy_to_nullb 1129 memcpy_page()
linux-block (null-memcpy-page) # modprobe -r null_blk
linux-block (null-memcpy-page) # odprobe null_blk gb=1 memory_backed=1 zoned=1
bash: odprobe: command not found...
linux-block (null-memcpy-page) # modprobe null_blk gb=1 memory_backed=1 zoned=1
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null bs=4k count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000259513 s, 15.8 MB/s
linux-block (null-memcpy-page) # dmesg  -c
[30955.086508] blk_queue_max_hw_sectors: set to minimum 8
[30955.087986] null_blk: nullb_fill_pattern 1175 memset_page()
[30955.088030] null_blk: nullb_fill_pattern 1175 memset_page()
[30955.088061] null_blk: disk nullb0 created
[30955.088063] null_blk: module loaded
[30961.489556] null_blk: nullb_fill_pattern 1175 memset_page()
[30961.489572] null_blk: nullb_fill_pattern 1175 memset_page()
[30961.489575] null_blk: nullb_fill_pattern 1175 memset_page()
[30961.489577] null_blk: nullb_fill_pattern 1175 memset_page()
linux-block (null-memcpy-page) # dd if=/dev/zero of=/dev/nullb0 count=1 bs=4k oflag=direct
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000243984 s, 16.8 MB/s
linux-block (null-memcpy-page) # dd if=/dev/nullb0 of=/dev/null count=1 bs=4k
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000253021 s, 16.2 MB/s
linux-block (null-memcpy-page) # dmesg  -c
[30973.806184] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806190] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806191] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806192] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806193] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806194] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806196] null_blk: copy_to_nullb 1129 memcpy_page()
[30973.806197] null_blk: copy_to_nullb 1129 memcpy_page()
[30978.670129] null_blk: nullb_fill_pattern 1175 memset_page()
[30978.670145] null_blk: nullb_fill_pattern 1175 memset_page()
[30978.670148] null_blk: nullb_fill_pattern 1175 memset_page()
[30978.670168] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670170] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670171] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670172] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670173] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670174] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670176] null_blk: copy_from_nullb 1160 memcpy_page()
[30978.670177] null_blk: copy_from_nullb 1160 memcpy_page()
linux-block (null-memcpy-page) #  modprobe  -r null_blk
linux-block (null-memcpy-page) # modprobe null_blk gb=1 memory_backed=1 cache_size=1
linux-block (null-memcpy-page) #  dd if=/dev/nullb0 of=/dev/null bs=4k count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000227442 s, 18.0 MB/s
linux-block (null-memcpy-page) # dmesg  -c
[30996.959156] blk_queue_max_hw_sectors: set to minimum 8
[30996.960786] null_blk: copy_from_nullb 1163 zero_user()
[30996.960789] null_blk: copy_from_nullb 1163 zero_user()
[30996.960803] null_blk: copy_from_nullb 1163 zero_user()
[30996.960804] null_blk: copy_from_nullb 1163 zero_user()
[30996.960804] null_blk: copy_from_nullb 1163 zero_user()
[30996.960805] null_blk: copy_from_nullb 1163 zero_user()
[30996.960805] null_blk: copy_from_nullb 1163 zero_user()
[30996.960806] null_blk: copy_from_nullb 1163 zero_user()
[30996.960833] null_blk: disk nullb0 created
[30996.960834] null_blk: module loaded
[31003.136892] null_blk: copy_from_nullb 1163 zero_user()
[31003.136898] null_blk: copy_from_nullb 1163 zero_user()
[31003.136900] null_blk: copy_from_nullb 1163 zero_user()
[31003.136901] null_blk: copy_from_nullb 1163 zero_user()
[31003.136902] null_blk: copy_from_nullb 1163 zero_user()
[31003.136903] null_blk: copy_from_nullb 1163 zero_user()
[31003.136903] null_blk: copy_from_nullb 1163 zero_user()
[31003.136904] null_blk: copy_from_nullb 1163 zero_user()
linux-block (null-memcpy-page) #

Comments

Jens Axboe April 2, 2023, 2:28 a.m. UTC | #1 
On Thu, 30 Mar 2023 11:49:24 -0700, Chaitanya Kulkarni wrote:
> From :include/linux/highmem.h:
> "kmap_atomic - Atomically map a page for temporary usage - Deprecated!"
> 
> Use memcpy_from_page() since does the same job of mapping, copying, and
> unmaping except it uses non deprecated kmap_local_page() and
> kunmap_local(). Following are the differences between kmal_local_page()
> and kmap_atomic() :-
> 
> [...]

Applied, thanks!

[1/2] null_blk: use non-deprecated lib functions
      commit: acc3c8799b9723d0b6a8cd67f8036dfdaa280825
[2/2] null_blk: use kmap_local_page() and kunmap_local()
      commit: acc3c8799b9723d0b6a8cd67f8036dfdaa280825

Best regards,