| Message ID | 20210719184001.1750630-3-willy@infradead.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Folio support in block + iomap layers | expand |
On Mon, Jul 19, 2021 at 07:39:46PM +0100, Matthew Wilcox (Oracle) wrote: > Allow callers to iterate over each folio instead of each page. The > bio need not have been constructed using folios originally. > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> Whoops, I never did remember to circle back and ack this patch. Reviewed-by: Darrick J. Wong <djwong@kernel.org> --D > --- > Documentation/core-api/kernel-api.rst | 1 + > include/linux/bio.h | 53 ++++++++++++++++++++++++++- > 2 files changed, 53 insertions(+), 1 deletion(-) > > diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst > index 2a7444e3a4c2..b804605f705e 100644 > --- a/Documentation/core-api/kernel-api.rst > +++ b/Documentation/core-api/kernel-api.rst > @@ -279,6 +279,7 @@ Accounting Framework > Block Devices > ============= > > +.. kernel-doc:: include/linux/bio.h > .. kernel-doc:: block/blk-core.c > :export: > > diff --git a/include/linux/bio.h b/include/linux/bio.h > index ade93e2de6a1..a90a79ad7bd1 100644 > --- a/include/linux/bio.h > +++ b/include/linux/bio.h > @@ -189,7 +189,7 @@ static inline void bio_advance_iter_single(const struct bio *bio, > */ > #define bio_for_each_bvec_all(bvl, bio, i) \ > for (i = 0, bvl = bio_first_bvec_all(bio); \ > - i < (bio)->bi_vcnt; i++, bvl++) \ > + i < (bio)->bi_vcnt; i++, bvl++) > > #define bio_iter_last(bvec, iter) ((iter).bi_size == (bvec).bv_len) > > @@ -314,6 +314,57 @@ static inline struct bio_vec *bio_last_bvec_all(struct bio *bio) > return &bio->bi_io_vec[bio->bi_vcnt - 1]; > } > > +/** > + * struct folio_iter - State for iterating all folios in a bio. > + * @folio: The current folio we're iterating. NULL after the last folio. > + * @offset: The byte offset within the current folio. > + * @length: The number of bytes in this iteration (will not cross folio > + * boundary). > + */ > +struct folio_iter { > + struct folio *folio; > + size_t offset; > + size_t length; > + /* private: for use by the iterator */ > + size_t _seg_count; > + int _i; > +}; > + > +static inline > +void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i) > +{ > + struct bio_vec *bvec = bio_first_bvec_all(bio) + i; > + > + fi->folio = page_folio(bvec->bv_page); > + fi->offset = bvec->bv_offset + > + PAGE_SIZE * (bvec->bv_page - &fi->folio->page); > + fi->_seg_count = bvec->bv_len; > + fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count); > + fi->_i = i; > +} > + > +static inline void bio_next_folio(struct folio_iter *fi, struct bio *bio) > +{ > + fi->_seg_count -= fi->length; > + if (fi->_seg_count) { > + fi->folio = folio_next(fi->folio); > + fi->offset = 0; > + fi->length = min(folio_size(fi->folio), fi->_seg_count); > + } else if (fi->_i + 1 < bio->bi_vcnt) { > + bio_first_folio(fi, bio, fi->_i + 1); > + } else { > + fi->folio = NULL; > + } > +} > + > +/** > + * bio_for_each_folio_all - Iterate over each folio in a bio. > + * @fi: struct folio_iter which is updated for each folio. > + * @bio: struct bio to iterate over. > + */ > +#define bio_for_each_folio_all(fi, bio) \ > + for (bio_first_folio(&fi, bio, 0); fi.folio; bio_next_folio(&fi, bio)) > + > enum bip_flags { > BIP_BLOCK_INTEGRITY = 1 << 0, /* block layer owns integrity data */ > BIP_MAPPED_INTEGRITY = 1 << 1, /* ref tag has been remapped */ > -- > 2.30.2 >
On Mon, Jul 19, 2021 at 06:29:45PM -0700, Darrick J. Wong wrote: > On Mon, Jul 19, 2021 at 07:39:46PM +0100, Matthew Wilcox (Oracle) wrote: > > Allow callers to iterate over each folio instead of each page. The > > bio need not have been constructed using folios originally. > > > > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> > > Whoops, I never did remember to circle back and ack this patch. > Reviewed-by: Darrick J. Wong <djwong@kernel.org> That's all good; it made me ask "What more can I do?" and "Add documentation" as always a good answer ...
On Mon, Jul 19, 2021 at 07:39:46PM +0100, Matthew Wilcox (Oracle) wrote: > #define bio_for_each_bvec_all(bvl, bio, i) \ > for (i = 0, bvl = bio_first_bvec_all(bio); \ > - i < (bio)->bi_vcnt; i++, bvl++) \ > + i < (bio)->bi_vcnt; i++, bvl++) Pleae split out this unrelated fixup. > +static inline > +void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i) Please fix the strange formatting. > +{ > + struct bio_vec *bvec = bio_first_bvec_all(bio) + i; > + > + fi->folio = page_folio(bvec->bv_page); > + fi->offset = bvec->bv_offset + > + PAGE_SIZE * (bvec->bv_page - &fi->folio->page); Can we have a little helper for the offset in folio calculation, like: static inline size_t offset_of_page_in_folio(struct page *page) { return (bvec->bv_page - &page_folio(page)->page) * PAGE; } as that makes the callers a lot easier to read.
Hi "Matthew,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on linus/master]
[also build test WARNING on v5.14-rc2 next-20210720]
[cannot apply to xfs-linux/for-next block/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Matthew-Wilcox-Oracle/Folio-support-in-block-iomap-layers/20210720-152323
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2734d6c1b1a089fb593ef6a23d4b70903526fe0c
config: x86_64-randconfig-r016-20210720 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project 87039c048c0cbc3d8cbba86187269b006bf2f373)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/d611eec3f37483e1acbcc299bf47e929766afc8d
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Matthew-Wilcox-Oracle/Folio-support-in-block-iomap-layers/20210720-152323
git checkout d611eec3f37483e1acbcc299bf47e929766afc8d
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All warnings (new ones prefixed by >>):
In file included from arch/x86/kernel/asm-offsets.c:13:
In file included from include/linux/suspend.h:5:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:18:
include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror,-Wimplicit-function-declaration]
fi->folio = page_folio(bvec->bv_page);
^
>> include/linux/bio.h:338:12: warning: incompatible integer to pointer conversion assigning to 'struct folio *' from 'int' [-Wint-conversion]
fi->folio = page_folio(bvec->bv_page);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/bio.h:340:43: error: incomplete definition of type 'struct folio'
PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
~~~~~~~~~^
include/linux/bio.h:325:9: note: forward declaration of 'struct folio'
struct folio *folio;
^
include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror,-Wimplicit-function-declaration]
fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
^
include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror,-Wimplicit-function-declaration]
fi->folio = folio_next(fi->folio);
^
include/linux/bio.h:350:13: warning: incompatible integer to pointer conversion assigning to 'struct folio *' from 'int' [-Wint-conversion]
fi->folio = folio_next(fi->folio);
^ ~~~~~~~~~~~~~~~~~~~~~
include/linux/bio.h:352:20: error: implicit declaration of function 'folio_size' [-Werror,-Wimplicit-function-declaration]
fi->length = min(folio_size(fi->folio), fi->_seg_count);
^
>> include/linux/bio.h:352:16: warning: comparison of distinct pointer types ('typeof (folio_size(fi->folio)) *' (aka 'int *') and 'typeof (fi->_seg_count) *' (aka 'unsigned long *')) [-Wcompare-distinct-pointer-types]
fi->length = min(folio_size(fi->folio), fi->_seg_count);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/minmax.h:45:19: note: expanded from macro 'min'
#define min(x, y) __careful_cmp(x, y, <)
^~~~~~~~~~~~~~~~~~~~~~
include/linux/minmax.h:36:24: note: expanded from macro '__careful_cmp'
__builtin_choose_expr(__safe_cmp(x, y), \
^~~~~~~~~~~~~~~~
include/linux/minmax.h:26:4: note: expanded from macro '__safe_cmp'
(__typecheck(x, y) && __no_side_effects(x, y))
^~~~~~~~~~~~~~~~~
include/linux/minmax.h:20:28: note: expanded from macro '__typecheck'
(!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~
3 warnings and 5 errors generated.
--
In file included from arch/x86/kernel/asm-offsets.c:13:
In file included from include/linux/suspend.h:5:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:18:
include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror,-Wimplicit-function-declaration]
fi->folio = page_folio(bvec->bv_page);
^
>> include/linux/bio.h:338:12: warning: incompatible integer to pointer conversion assigning to 'struct folio *' from 'int' [-Wint-conversion]
fi->folio = page_folio(bvec->bv_page);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/bio.h:340:43: error: incomplete definition of type 'struct folio'
PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
~~~~~~~~~^
include/linux/bio.h:325:9: note: forward declaration of 'struct folio'
struct folio *folio;
^
include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror,-Wimplicit-function-declaration]
fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
^
include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror,-Wimplicit-function-declaration]
fi->folio = folio_next(fi->folio);
^
include/linux/bio.h:350:13: warning: incompatible integer to pointer conversion assigning to 'struct folio *' from 'int' [-Wint-conversion]
fi->folio = folio_next(fi->folio);
^ ~~~~~~~~~~~~~~~~~~~~~
include/linux/bio.h:352:20: error: implicit declaration of function 'folio_size' [-Werror,-Wimplicit-function-declaration]
fi->length = min(folio_size(fi->folio), fi->_seg_count);
^
>> include/linux/bio.h:352:16: warning: comparison of distinct pointer types ('typeof (folio_size(fi->folio)) *' (aka 'int *') and 'typeof (fi->_seg_count) *' (aka 'unsigned long *')) [-Wcompare-distinct-pointer-types]
fi->length = min(folio_size(fi->folio), fi->_seg_count);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/minmax.h:45:19: note: expanded from macro 'min'
#define min(x, y) __careful_cmp(x, y, <)
^~~~~~~~~~~~~~~~~~~~~~
include/linux/minmax.h:36:24: note: expanded from macro '__careful_cmp'
__builtin_choose_expr(__safe_cmp(x, y), \
^~~~~~~~~~~~~~~~
include/linux/minmax.h:26:4: note: expanded from macro '__safe_cmp'
(__typecheck(x, y) && __no_side_effects(x, y))
^~~~~~~~~~~~~~~~~
include/linux/minmax.h:20:28: note: expanded from macro '__typecheck'
(!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~
3 warnings and 5 errors generated.
make[2]: *** [scripts/Makefile.build:117: arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1213: prepare0] Error 2
make[1]: Target 'modules_prepare' not remade because of errors.
make: *** [Makefile:220: __sub-make] Error 2
make: Target 'modules_prepare' not remade because of errors.
--
In file included from arch/x86/kernel/asm-offsets.c:13:
In file included from include/linux/suspend.h:5:
In file included from include/linux/swap.h:9:
In file included from include/linux/memcontrol.h:22:
In file included from include/linux/writeback.h:14:
In file included from include/linux/blk-cgroup.h:23:
In file included from include/linux/blkdev.h:18:
include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror,-Wimplicit-function-declaration]
fi->folio = page_folio(bvec->bv_page);
^
>> include/linux/bio.h:338:12: warning: incompatible integer to pointer conversion assigning to 'struct folio *' from 'int' [-Wint-conversion]
fi->folio = page_folio(bvec->bv_page);
^ ~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/bio.h:340:43: error: incomplete definition of type 'struct folio'
PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
~~~~~~~~~^
include/linux/bio.h:325:9: note: forward declaration of 'struct folio'
struct folio *folio;
^
include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror,-Wimplicit-function-declaration]
fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
^
include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror,-Wimplicit-function-declaration]
fi->folio = folio_next(fi->folio);
^
include/linux/bio.h:350:13: warning: incompatible integer to pointer conversion assigning to 'struct folio *' from 'int' [-Wint-conversion]
fi->folio = folio_next(fi->folio);
^ ~~~~~~~~~~~~~~~~~~~~~
include/linux/bio.h:352:20: error: implicit declaration of function 'folio_size' [-Werror,-Wimplicit-function-declaration]
fi->length = min(folio_size(fi->folio), fi->_seg_count);
^
>> include/linux/bio.h:352:16: warning: comparison of distinct pointer types ('typeof (folio_size(fi->folio)) *' (aka 'int *') and 'typeof (fi->_seg_count) *' (aka 'unsigned long *')) [-Wcompare-distinct-pointer-types]
fi->length = min(folio_size(fi->folio), fi->_seg_count);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/minmax.h:45:19: note: expanded from macro 'min'
#define min(x, y) __careful_cmp(x, y, <)
^~~~~~~~~~~~~~~~~~~~~~
include/linux/minmax.h:36:24: note: expanded from macro '__careful_cmp'
__builtin_choose_expr(__safe_cmp(x, y), \
^~~~~~~~~~~~~~~~
include/linux/minmax.h:26:4: note: expanded from macro '__safe_cmp'
(__typecheck(x, y) && __no_side_effects(x, y))
^~~~~~~~~~~~~~~~~
include/linux/minmax.h:20:28: note: expanded from macro '__typecheck'
(!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~
3 warnings and 5 errors generated.
make[2]: *** [scripts/Makefile.build:117: arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1213: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:220: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +338 include/linux/bio.h
332
333 static inline
334 void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i)
335 {
336 struct bio_vec *bvec = bio_first_bvec_all(bio) + i;
337
> 338 fi->folio = page_folio(bvec->bv_page);
339 fi->offset = bvec->bv_offset +
340 PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
341 fi->_seg_count = bvec->bv_len;
342 fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
343 fi->_i = i;
344 }
345
346 static inline void bio_next_folio(struct folio_iter *fi, struct bio *bio)
347 {
348 fi->_seg_count -= fi->length;
349 if (fi->_seg_count) {
350 fi->folio = folio_next(fi->folio);
351 fi->offset = 0;
> 352 fi->length = min(folio_size(fi->folio), fi->_seg_count);
353 } else if (fi->_i + 1 < bio->bi_vcnt) {
354 bio_first_folio(fi, bio, fi->_i + 1);
355 } else {
356 fi->folio = NULL;
357 }
358 }
359
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
Hi "Matthew,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.14-rc2 next-20210720]
[cannot apply to xfs-linux/for-next block/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Matthew-Wilcox-Oracle/Folio-support-in-block-iomap-layers/20210720-152323
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2734d6c1b1a089fb593ef6a23d4b70903526fe0c
config: sparc-randconfig-r015-20210720 (attached as .config)
compiler: sparc64-linux-gcc (GCC) 10.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/d611eec3f37483e1acbcc299bf47e929766afc8d
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Matthew-Wilcox-Oracle/Folio-support-in-block-iomap-layers/20210720-152323
git checkout d611eec3f37483e1acbcc299bf47e929766afc8d
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-10.3.0 make.cross ARCH=sparc
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/tracehook.h:50,
from arch/sparc/kernel/signal_64.c:18:
include/linux/bio.h: In function 'bio_first_folio':
include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror=implicit-function-declaration]
338 | fi->folio = page_folio(bvec->bv_page);
| ^~~~~~~~~~
>> include/linux/bio.h:338:12: error: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
338 | fi->folio = page_folio(bvec->bv_page);
| ^
include/linux/bio.h:340:43: error: invalid use of undefined type 'struct folio'
340 | PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
| ^~
In file included from include/linux/kernel.h:16,
from include/linux/list.h:9,
from include/linux/rculist.h:10,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from arch/sparc/kernel/signal_64.c:12:
include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror=implicit-function-declaration]
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~~~~~~~~
include/linux/minmax.h:20:21: note: in definition of macro '__typecheck'
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:342:15: note: in expansion of macro 'min'
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/tracehook.h:50,
from arch/sparc/kernel/signal_64.c:18:
include/linux/bio.h: In function 'bio_next_folio':
include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror=implicit-function-declaration]
350 | fi->folio = folio_next(fi->folio);
| ^~~~~~~~~~
include/linux/bio.h:350:13: error: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
350 | fi->folio = folio_next(fi->folio);
| ^
In file included from include/linux/kernel.h:16,
from include/linux/list.h:9,
from include/linux/rculist.h:10,
from include/linux/pid.h:5,
from include/linux/sched.h:14,
from arch/sparc/kernel/signal_64.c:12:
include/linux/minmax.h:20:28: error: comparison of distinct pointer types lacks a cast [-Werror]
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^~
include/linux/minmax.h:26:4: note: in expansion of macro '__typecheck'
26 | (__typecheck(x, y) && __no_side_effects(x, y))
| ^~~~~~~~~~~
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:352:16: note: in expansion of macro 'min'
352 | fi->length = min(folio_size(fi->folio), fi->_seg_count);
| ^~~
cc1: all warnings being treated as errors
--
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/swap.h:9,
from arch/sparc/include/asm/tlb_64.h:5,
from arch/sparc/include/asm/tlb.h:5,
from arch/sparc/kernel/smp_64.c:49:
include/linux/bio.h: In function 'bio_first_folio':
include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio'; did you mean 'page_endio'? [-Werror=implicit-function-declaration]
338 | fi->folio = page_folio(bvec->bv_page);
| ^~~~~~~~~~
| page_endio
>> include/linux/bio.h:338:12: error: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
338 | fi->folio = page_folio(bvec->bv_page);
| ^
include/linux/bio.h:340:43: error: invalid use of undefined type 'struct folio'
340 | PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
| ^~
In file included from include/linux/kernel.h:16,
from arch/sparc/kernel/smp_64.c:8:
include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror=implicit-function-declaration]
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~~~~~~~~
include/linux/minmax.h:20:21: note: in definition of macro '__typecheck'
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:342:15: note: in expansion of macro 'min'
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/swap.h:9,
from arch/sparc/include/asm/tlb_64.h:5,
from arch/sparc/include/asm/tlb.h:5,
from arch/sparc/kernel/smp_64.c:49:
include/linux/bio.h: In function 'bio_next_folio':
include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror=implicit-function-declaration]
350 | fi->folio = folio_next(fi->folio);
| ^~~~~~~~~~
include/linux/bio.h:350:13: error: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
350 | fi->folio = folio_next(fi->folio);
| ^
In file included from include/linux/kernel.h:16,
from arch/sparc/kernel/smp_64.c:8:
include/linux/minmax.h:20:28: error: comparison of distinct pointer types lacks a cast [-Werror]
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^~
include/linux/minmax.h:26:4: note: in expansion of macro '__typecheck'
26 | (__typecheck(x, y) && __no_side_effects(x, y))
| ^~~~~~~~~~~
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:352:16: note: in expansion of macro 'min'
352 | fi->length = min(folio_size(fi->folio), fi->_seg_count);
| ^~~
cc1: all warnings being treated as errors
--
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/net/sock.h:53,
from include/linux/tcp.h:19,
from include/linux/ipv6.h:88,
from arch/sparc/kernel/sys_sparc32.c:33:
include/linux/bio.h: In function 'bio_first_folio':
include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror=implicit-function-declaration]
338 | fi->folio = page_folio(bvec->bv_page);
| ^~~~~~~~~~
>> include/linux/bio.h:338:12: error: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
338 | fi->folio = page_folio(bvec->bv_page);
| ^
include/linux/bio.h:340:43: error: invalid use of undefined type 'struct folio'
340 | PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
| ^~
In file included from include/linux/kernel.h:16,
from arch/sparc/kernel/sys_sparc32.c:11:
include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror=implicit-function-declaration]
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~~~~~~~~
include/linux/minmax.h:20:21: note: in definition of macro '__typecheck'
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:342:15: note: in expansion of macro 'min'
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/net/sock.h:53,
from include/linux/tcp.h:19,
from include/linux/ipv6.h:88,
from arch/sparc/kernel/sys_sparc32.c:33:
include/linux/bio.h: In function 'bio_next_folio':
include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next'; did you mean 'nla_next'? [-Werror=implicit-function-declaration]
350 | fi->folio = folio_next(fi->folio);
| ^~~~~~~~~~
| nla_next
include/linux/bio.h:350:13: error: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
350 | fi->folio = folio_next(fi->folio);
| ^
In file included from include/linux/kernel.h:16,
from arch/sparc/kernel/sys_sparc32.c:11:
include/linux/minmax.h:20:28: error: comparison of distinct pointer types lacks a cast [-Werror]
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^~
include/linux/minmax.h:26:4: note: in expansion of macro '__typecheck'
26 | (__typecheck(x, y) && __no_side_effects(x, y))
| ^~~~~~~~~~~
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:352:16: note: in expansion of macro 'min'
352 | fi->length = min(folio_size(fi->folio), fi->_seg_count);
| ^~~
cc1: all warnings being treated as errors
vim +338 include/linux/bio.h
332
333 static inline
334 void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i)
335 {
336 struct bio_vec *bvec = bio_first_bvec_all(bio) + i;
337
> 338 fi->folio = page_folio(bvec->bv_page);
339 fi->offset = bvec->bv_offset +
340 PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
341 fi->_seg_count = bvec->bv_len;
342 fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
343 fi->_i = i;
344 }
345
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
Hi "Matthew,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on linus/master]
[also build test ERROR on v5.14-rc2 next-20210720]
[cannot apply to xfs-linux/for-next block/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Matthew-Wilcox-Oracle/Folio-support-in-block-iomap-layers/20210720-152323
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2734d6c1b1a089fb593ef6a23d4b70903526fe0c
config: i386-tinyconfig (attached as .config)
compiler: gcc-10 (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0
reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/d611eec3f37483e1acbcc299bf47e929766afc8d
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Matthew-Wilcox-Oracle/Folio-support-in-block-iomap-layers/20210720-152323
git checkout d611eec3f37483e1acbcc299bf47e929766afc8d
# save the attached .config to linux build tree
make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/swap.h:9,
from include/linux/suspend.h:5,
from arch/x86/kernel/asm-offsets.c:13:
include/linux/bio.h: In function 'bio_first_folio':
>> include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror=implicit-function-declaration]
338 | fi->folio = page_folio(bvec->bv_page);
| ^~~~~~~~~~
>> include/linux/bio.h:338:12: warning: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
338 | fi->folio = page_folio(bvec->bv_page);
| ^
>> include/linux/bio.h:340:43: error: invalid use of undefined type 'struct folio'
340 | PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
| ^~
In file included from include/linux/kernel.h:16,
from include/linux/crypto.h:16,
from arch/x86/kernel/asm-offsets.c:9:
>> include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror=implicit-function-declaration]
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~~~~~~~~
include/linux/minmax.h:20:21: note: in definition of macro '__typecheck'
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:342:15: note: in expansion of macro 'min'
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/swap.h:9,
from include/linux/suspend.h:5,
from arch/x86/kernel/asm-offsets.c:13:
include/linux/bio.h: In function 'bio_next_folio':
>> include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror=implicit-function-declaration]
350 | fi->folio = folio_next(fi->folio);
| ^~~~~~~~~~
include/linux/bio.h:350:13: warning: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
350 | fi->folio = folio_next(fi->folio);
| ^
In file included from include/linux/kernel.h:16,
from include/linux/crypto.h:16,
from arch/x86/kernel/asm-offsets.c:9:
include/linux/minmax.h:20:28: warning: comparison of distinct pointer types lacks a cast
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^~
include/linux/minmax.h:26:4: note: in expansion of macro '__typecheck'
26 | (__typecheck(x, y) && __no_side_effects(x, y))
| ^~~~~~~~~~~
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:352:16: note: in expansion of macro 'min'
352 | fi->length = min(folio_size(fi->folio), fi->_seg_count);
| ^~~
cc1: some warnings being treated as errors
--
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/swap.h:9,
from include/linux/suspend.h:5,
from arch/x86/kernel/asm-offsets.c:13:
include/linux/bio.h: In function 'bio_first_folio':
>> include/linux/bio.h:338:14: error: implicit declaration of function 'page_folio' [-Werror=implicit-function-declaration]
338 | fi->folio = page_folio(bvec->bv_page);
| ^~~~~~~~~~
>> include/linux/bio.h:338:12: warning: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
338 | fi->folio = page_folio(bvec->bv_page);
| ^
>> include/linux/bio.h:340:43: error: invalid use of undefined type 'struct folio'
340 | PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
| ^~
In file included from include/linux/kernel.h:16,
from include/linux/crypto.h:16,
from arch/x86/kernel/asm-offsets.c:9:
>> include/linux/bio.h:342:19: error: implicit declaration of function 'folio_size' [-Werror=implicit-function-declaration]
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~~~~~~~~
include/linux/minmax.h:20:21: note: in definition of macro '__typecheck'
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:342:15: note: in expansion of macro 'min'
342 | fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
| ^~~
In file included from include/linux/blkdev.h:18,
from include/linux/blk-cgroup.h:23,
from include/linux/writeback.h:14,
from include/linux/memcontrol.h:22,
from include/linux/swap.h:9,
from include/linux/suspend.h:5,
from arch/x86/kernel/asm-offsets.c:13:
include/linux/bio.h: In function 'bio_next_folio':
>> include/linux/bio.h:350:15: error: implicit declaration of function 'folio_next' [-Werror=implicit-function-declaration]
350 | fi->folio = folio_next(fi->folio);
| ^~~~~~~~~~
include/linux/bio.h:350:13: warning: assignment to 'struct folio *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
350 | fi->folio = folio_next(fi->folio);
| ^
In file included from include/linux/kernel.h:16,
from include/linux/crypto.h:16,
from arch/x86/kernel/asm-offsets.c:9:
include/linux/minmax.h:20:28: warning: comparison of distinct pointer types lacks a cast
20 | (!!(sizeof((typeof(x) *)1 == (typeof(y) *)1)))
| ^~
include/linux/minmax.h:26:4: note: in expansion of macro '__typecheck'
26 | (__typecheck(x, y) && __no_side_effects(x, y))
| ^~~~~~~~~~~
include/linux/minmax.h:36:24: note: in expansion of macro '__safe_cmp'
36 | __builtin_choose_expr(__safe_cmp(x, y), \
| ^~~~~~~~~~
include/linux/minmax.h:45:19: note: in expansion of macro '__careful_cmp'
45 | #define min(x, y) __careful_cmp(x, y, <)
| ^~~~~~~~~~~~~
include/linux/bio.h:352:16: note: in expansion of macro 'min'
352 | fi->length = min(folio_size(fi->folio), fi->_seg_count);
| ^~~
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:117: arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1213: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:220: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +/page_folio +338 include/linux/bio.h
332
333 static inline
334 void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i)
335 {
336 struct bio_vec *bvec = bio_first_bvec_all(bio) + i;
337
> 338 fi->folio = page_folio(bvec->bv_page);
339 fi->offset = bvec->bv_offset +
> 340 PAGE_SIZE * (bvec->bv_page - &fi->folio->page);
341 fi->_seg_count = bvec->bv_len;
> 342 fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count);
343 fi->_i = i;
344 }
345
346 static inline void bio_next_folio(struct folio_iter *fi, struct bio *bio)
347 {
348 fi->_seg_count -= fi->length;
349 if (fi->_seg_count) {
> 350 fi->folio = folio_next(fi->folio);
351 fi->offset = 0;
352 fi->length = min(folio_size(fi->folio), fi->_seg_count);
353 } else if (fi->_i + 1 < bio->bi_vcnt) {
354 bio_first_folio(fi, bio, fi->_i + 1);
355 } else {
356 fi->folio = NULL;
357 }
358 }
359
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
On Tue, Jul 20, 2021 at 08:48:07AM +0200, Christoph Hellwig wrote: > On Mon, Jul 19, 2021 at 07:39:46PM +0100, Matthew Wilcox (Oracle) wrote: > > #define bio_for_each_bvec_all(bvl, bio, i) \ > > for (i = 0, bvl = bio_first_bvec_all(bio); \ > > - i < (bio)->bi_vcnt; i++, bvl++) \ > > + i < (bio)->bi_vcnt; i++, bvl++) > > Pleae split out this unrelated fixup. > > > +static inline > > +void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i) > > Please fix the strange formatting. static inline void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i) > > +{ > > + struct bio_vec *bvec = bio_first_bvec_all(bio) + i; > > + > > + fi->folio = page_folio(bvec->bv_page); > > + fi->offset = bvec->bv_offset + > > + PAGE_SIZE * (bvec->bv_page - &fi->folio->page); > > Can we have a little helper for the offset in folio calculation, like: > > static inline size_t offset_of_page_in_folio(struct page *page) > { > return (bvec->bv_page - &page_folio(page)->page) * PAGE; > } > > as that makes the callers a lot easier to read. I've spent most of today thinking about this one. I actually don't want to make this easy to read. This is code that, in an ideal world, would not exist. A bio_vec should not contain a struct page; it should probably be: struct bio_vec { phys_addr_t bv_start; unsigned int bv_len; }; and then the helper to get from a bio_vec to a folio_iter looks like: fi->folio = pfn_folio(bvec->bv_start >> PAGE_SHIFT); fi->offset = offset_in_folio(fi->folio, bvec->bv_start); If instead we decide to keep bvecs the way they are, we can at least turn the bv_page into bv_folio, and then we won't need this code either.
diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index 2a7444e3a4c2..b804605f705e 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -279,6 +279,7 @@ Accounting Framework Block Devices ============= +.. kernel-doc:: include/linux/bio.h .. kernel-doc:: block/blk-core.c :export: diff --git a/include/linux/bio.h b/include/linux/bio.h index ade93e2de6a1..a90a79ad7bd1 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -189,7 +189,7 @@ static inline void bio_advance_iter_single(const struct bio *bio, */ #define bio_for_each_bvec_all(bvl, bio, i) \ for (i = 0, bvl = bio_first_bvec_all(bio); \ - i < (bio)->bi_vcnt; i++, bvl++) \ + i < (bio)->bi_vcnt; i++, bvl++) #define bio_iter_last(bvec, iter) ((iter).bi_size == (bvec).bv_len) @@ -314,6 +314,57 @@ static inline struct bio_vec *bio_last_bvec_all(struct bio *bio) return &bio->bi_io_vec[bio->bi_vcnt - 1]; } +/** + * struct folio_iter - State for iterating all folios in a bio. + * @folio: The current folio we're iterating. NULL after the last folio. + * @offset: The byte offset within the current folio. + * @length: The number of bytes in this iteration (will not cross folio + * boundary). + */ +struct folio_iter { + struct folio *folio; + size_t offset; + size_t length; + /* private: for use by the iterator */ + size_t _seg_count; + int _i; +}; + +static inline +void bio_first_folio(struct folio_iter *fi, struct bio *bio, int i) +{ + struct bio_vec *bvec = bio_first_bvec_all(bio) + i; + + fi->folio = page_folio(bvec->bv_page); + fi->offset = bvec->bv_offset + + PAGE_SIZE * (bvec->bv_page - &fi->folio->page); + fi->_seg_count = bvec->bv_len; + fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count); + fi->_i = i; +} + +static inline void bio_next_folio(struct folio_iter *fi, struct bio *bio) +{ + fi->_seg_count -= fi->length; + if (fi->_seg_count) { + fi->folio = folio_next(fi->folio); + fi->offset = 0; + fi->length = min(folio_size(fi->folio), fi->_seg_count); + } else if (fi->_i + 1 < bio->bi_vcnt) { + bio_first_folio(fi, bio, fi->_i + 1); + } else { + fi->folio = NULL; + } +} + +/** + * bio_for_each_folio_all - Iterate over each folio in a bio. + * @fi: struct folio_iter which is updated for each folio. + * @bio: struct bio to iterate over. + */ +#define bio_for_each_folio_all(fi, bio) \ + for (bio_first_folio(&fi, bio, 0); fi.folio; bio_next_folio(&fi, bio)) + enum bip_flags { BIP_BLOCK_INTEGRITY = 1 << 0, /* block layer owns integrity data */ BIP_MAPPED_INTEGRITY = 1 << 1, /* ref tag has been remapped */
Allow callers to iterate over each folio instead of each page. The bio need not have been constructed using folios originally. Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org> --- Documentation/core-api/kernel-api.rst | 1 + include/linux/bio.h | 53 ++++++++++++++++++++++++++- 2 files changed, 53 insertions(+), 1 deletion(-)