| Message ID | 20231026174153.1133-7-mwilck@suse.com (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Delegated to: | christophe varoqui |
| Headers | show |
| Series | multipath: aio, systemd, and documentation improvements | expand |
On Thu, Oct 26, 2023 at 07:41:45PM +0200, mwilck@suse.com wrote: > From: Martin Wilck <mwilck@suse.com> > > Currently the number of iocbs per path to test is the same as the > total number of iocbs in the ioctx. This can easily cause iocb > starvation, in particular if some IOs are hanging. In that case > io_submit() will fail, and some paths under test will use much > less IOs as intended, or in the worst case, none at all. > > The total number of iocbs reserved in the kernel should be higher. > With this patch, we will be able to run the marginal path test for > at least NR_IOSTAT_PATHS=32 paths at the same time. This is not an upper > limit, because kernel IOCBs can be reused between paths. > > Increase the log levels of io_setup and io_submit to make it sure > we catch problems with this approach. > > Signed-off-by: Martin Wilck <mwilck@suse.com> Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com> > Cc: Li Xiao Keng <lixiaokeng@huawei.com> > Cc: Miao Guanqin <miaoguanqin@huawei.com> > Cc: Guan Junxiong <guanjunxiong@huawei.com> > --- > libmultipath/io_err_stat.c | 7 ++++--- > 1 file changed, 4 insertions(+), 3 deletions(-) > > diff --git a/libmultipath/io_err_stat.c b/libmultipath/io_err_stat.c > index 3f32e32..5749003 100644 > --- a/libmultipath/io_err_stat.c > +++ b/libmultipath/io_err_stat.c > @@ -38,6 +38,7 @@ > #define TIMEOUT_NO_IO_NSEC 10000000 /*10ms = 10000000ns*/ > #define FLAKY_PATHFAIL_THRESHOLD 2 > #define CONCUR_NR_EVENT 32 > +#define NR_IOSTAT_PATHS 32 > > #define PATH_IO_ERR_IN_CHECKING -1 > #define PATH_IO_ERR_WAITING_TO_CHECK -2 > @@ -477,7 +478,7 @@ static int send_each_async_io(struct dio_ctx *ct, int fd, char *dev) > get_monotonic_time(&ct->io_starttime); > io_prep_pread(&ct->io, fd, ct->buf, ct->blksize, 0); > if (io_submit(ioctx, 1, ios) != 1) { > - io_err_stat_log(5, "%s: io_submit error %i", > + io_err_stat_log(2, "%s: io_submit error %i", > dev, errno); > return rc; > } > @@ -703,8 +704,8 @@ int start_io_err_stat_thread(void *data) > if (uatomic_read(&io_err_thread_running) == 1) > return 0; > > - if (io_setup(CONCUR_NR_EVENT, &ioctx) != 0) { > - io_err_stat_log(4, "io_setup failed"); > + if (io_setup(CONCUR_NR_EVENT * NR_IOSTAT_PATHS, &ioctx) != 0) { > + io_err_stat_log(1, "io_setup failed - increase /proc/sys/fs/aio-nr ?"); > return 1; > } > > -- > 2.42.0
diff --git a/libmultipath/io_err_stat.c b/libmultipath/io_err_stat.c index 3f32e32..5749003 100644 --- a/libmultipath/io_err_stat.c +++ b/libmultipath/io_err_stat.c @@ -38,6 +38,7 @@ #define TIMEOUT_NO_IO_NSEC 10000000 /*10ms = 10000000ns*/ #define FLAKY_PATHFAIL_THRESHOLD 2 #define CONCUR_NR_EVENT 32 +#define NR_IOSTAT_PATHS 32 #define PATH_IO_ERR_IN_CHECKING -1 #define PATH_IO_ERR_WAITING_TO_CHECK -2 @@ -477,7 +478,7 @@ static int send_each_async_io(struct dio_ctx *ct, int fd, char *dev) get_monotonic_time(&ct->io_starttime); io_prep_pread(&ct->io, fd, ct->buf, ct->blksize, 0); if (io_submit(ioctx, 1, ios) != 1) { - io_err_stat_log(5, "%s: io_submit error %i", + io_err_stat_log(2, "%s: io_submit error %i", dev, errno); return rc; } @@ -703,8 +704,8 @@ int start_io_err_stat_thread(void *data) if (uatomic_read(&io_err_thread_running) == 1) return 0; - if (io_setup(CONCUR_NR_EVENT, &ioctx) != 0) { - io_err_stat_log(4, "io_setup failed"); + if (io_setup(CONCUR_NR_EVENT * NR_IOSTAT_PATHS, &ioctx) != 0) { + io_err_stat_log(1, "io_setup failed - increase /proc/sys/fs/aio-nr ?"); return 1; }