| Message ID | 20210902204622.54354-1-mgurtovoy@nvidia.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v3,1/1] virtio-blk: add num_request_queues module parameter | expand |
> Sometimes a user would like to control the amount of request queues to > be created for a block device. For example, for limiting the memory > footprint of virtio-blk devices. > > Reviewed-by: Christoph Hellwig <hch@lst.de> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> > --- > > changes from v2: > - renamed num_io_queues to num_request_queues (from Stefan) > - added Reviewed-by signatures (from Stefan and Christoph) > > changes from v1: > - use param_set_uint_minmax (from Christoph) > - added "Should > 0" to module description > > Note: This commit apply on top of Jens's branch for-5.15/drivers > > --- > drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- > 1 file changed, 20 insertions(+), 1 deletion(-) > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c > index 4b49df2dfd23..aaa2833a4734 100644 > --- a/drivers/block/virtio_blk.c > +++ b/drivers/block/virtio_blk.c > @@ -24,6 +24,23 @@ > /* The maximum number of sg elements that fit into a virtqueue */ > #define VIRTIO_BLK_MAX_SG_ELEMS 32768 > > +static int virtblk_queue_count_set(const char *val, > + const struct kernel_param *kp) > +{ > + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); > +} > + > +static const struct kernel_param_ops queue_count_ops = { > + .set = virtblk_queue_count_set, > + .get = param_get_uint, > +}; > + > +static unsigned int num_request_queues; > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, > + 0644); > +MODULE_PARM_DESC(num_request_queues, > + "Number of request queues to use for blk device. Should > 0"); > + > static int major; > static DEFINE_IDA(vd_index_ida); > > @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) > if (err) > num_vqs = 1; > > - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); > + num_vqs = min_t(unsigned int, > + min_not_zero(num_request_queues, nr_cpu_ids), > + num_vqs); > > vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL); > if (!vblk->vqs) > -- Reviewed-by: Pankaj Gupta <pankaj.gupta@ionos.com>
On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote: > Sometimes a user would like to control the amount of request queues to > be created for a block device. For example, for limiting the memory > footprint of virtio-blk devices. > > Reviewed-by: Christoph Hellwig <hch@lst.de> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> > --- > > changes from v2: > - renamed num_io_queues to num_request_queues (from Stefan) > - added Reviewed-by signatures (from Stefan and Christoph) > > changes from v1: > - use param_set_uint_minmax (from Christoph) > - added "Should > 0" to module description > > Note: This commit apply on top of Jens's branch for-5.15/drivers > > --- > drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- > 1 file changed, 20 insertions(+), 1 deletion(-) > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c > index 4b49df2dfd23..aaa2833a4734 100644 > --- a/drivers/block/virtio_blk.c > +++ b/drivers/block/virtio_blk.c > @@ -24,6 +24,23 @@ > /* The maximum number of sg elements that fit into a virtqueue */ > #define VIRTIO_BLK_MAX_SG_ELEMS 32768 > > +static int virtblk_queue_count_set(const char *val, > + const struct kernel_param *kp) > +{ > + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); > +} > + > +static const struct kernel_param_ops queue_count_ops = { > + .set = virtblk_queue_count_set, > + .get = param_get_uint, > +}; > + > +static unsigned int num_request_queues; > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, > + 0644); > +MODULE_PARM_DESC(num_request_queues, > + "Number of request queues to use for blk device. Should > 0"); > + Won't it limit all virtio block devices to the same limit? It is very common to see multiple virtio-blk devices on the same system and they probably need different limits. Thanks
On 9/5/2021 12:46 AM, Leon Romanovsky wrote: >> +static unsigned int num_request_queues; >> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, >> + 0644); >> +MODULE_PARM_DESC(num_request_queues, >> + "Number of request queues to use for blk device. Should > 0"); >> + > Won't it limit all virtio block devices to the same limit? > > It is very common to see multiple virtio-blk devices on the same system > and they probably need different limits. > > Thanks Without looking into the code, that can be done adding a configfs interface and overriding a global value (module param) when it is set from configfs.
On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote: > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote: >>> +static unsigned int num_request_queues; >>> +module_param_cb(num_request_queues, &queue_count_ops, >>> &num_request_queues, >>> + 0644); >>> +MODULE_PARM_DESC(num_request_queues, >>> + "Number of request queues to use for blk device. Should > >>> 0"); >>> + >> Won't it limit all virtio block devices to the same limit? >> >> It is very common to see multiple virtio-blk devices on the same system >> and they probably need different limits. >> >> Thanks > > > Without looking into the code, that can be done adding a configfs > > interface and overriding a global value (module param) when it is set > from > > configfs. > > You have many ways to overcome this issue. For example: # ls -l /sys/block/vda/mq/ drwxr-xr-x 18 root root 0 Sep 5 12:14 0 drwxr-xr-x 18 root root 0 Sep 5 12:14 1 drwxr-xr-x 18 root root 0 Sep 5 12:14 2 drwxr-xr-x 18 root root 0 Sep 5 12:14 3 # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind # ls -l /sys/block/vda/mq/ drwxr-xr-x 10 root root 0 Sep 5 12:17 0 drwxr-xr-x 10 root root 0 Sep 5 12:17 1 drwxr-xr-x 10 root root 0 Sep 5 12:17 2 drwxr-xr-x 10 root root 0 Sep 5 12:17 3 drwxr-xr-x 10 root root 0 Sep 5 12:17 4 drwxr-xr-x 10 root root 0 Sep 5 12:17 5 drwxr-xr-x 10 root root 0 Sep 5 12:17 6 drwxr-xr-x 10 root root 0 Sep 5 12:17 7 -Max.
On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote: > > On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote: > > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote: > > > > +static unsigned int num_request_queues; > > > > +module_param_cb(num_request_queues, &queue_count_ops, > > > > &num_request_queues, > > > > + 0644); > > > > +MODULE_PARM_DESC(num_request_queues, > > > > + "Number of request queues to use for blk device. > > > > Should > 0"); > > > > + > > > Won't it limit all virtio block devices to the same limit? > > > > > > It is very common to see multiple virtio-blk devices on the same system > > > and they probably need different limits. > > > > > > Thanks > > > > > > Without looking into the code, that can be done adding a configfs > > > > interface and overriding a global value (module param) when it is set > > from > > > > configfs. > > > > > You have many ways to overcome this issue. > > For example: > > # ls -l /sys/block/vda/mq/ > drwxr-xr-x 18 root root 0 Sep 5 12:14 0 > drwxr-xr-x 18 root root 0 Sep 5 12:14 1 > drwxr-xr-x 18 root root 0 Sep 5 12:14 2 > drwxr-xr-x 18 root root 0 Sep 5 12:14 3 > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind > > # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues This is global to all virtio-blk devices. > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind > > # ls -l /sys/block/vda/mq/ > drwxr-xr-x 10 root root 0 Sep 5 12:17 0 > drwxr-xr-x 10 root root 0 Sep 5 12:17 1 > drwxr-xr-x 10 root root 0 Sep 5 12:17 2 > drwxr-xr-x 10 root root 0 Sep 5 12:17 3 > drwxr-xr-x 10 root root 0 Sep 5 12:17 4 > drwxr-xr-x 10 root root 0 Sep 5 12:17 5 > drwxr-xr-x 10 root root 0 Sep 5 12:17 6 > drwxr-xr-x 10 root root 0 Sep 5 12:17 7 > > -Max. > >
On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote: > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote: > > > +static unsigned int num_request_queues; > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, > > > + 0644); > > > +MODULE_PARM_DESC(num_request_queues, > > > + "Number of request queues to use for blk device. Should > 0"); > > > + > > Won't it limit all virtio block devices to the same limit? > > > > It is very common to see multiple virtio-blk devices on the same system > > and they probably need different limits. > > > > Thanks > > > Without looking into the code, that can be done adding a configfs > > interface and overriding a global value (module param) when it is set from > > configfs. So why should we do double work instead of providing one working interface from the beginning? Thanks > >
On 9/5/2021 1:19 PM, Leon Romanovsky wrote: > On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote: >> On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote: >>> On 9/5/2021 12:46 AM, Leon Romanovsky wrote: >>>>> +static unsigned int num_request_queues; >>>>> +module_param_cb(num_request_queues, &queue_count_ops, >>>>> &num_request_queues, >>>>> + 0644); >>>>> +MODULE_PARM_DESC(num_request_queues, >>>>> + "Number of request queues to use for blk device. >>>>> Should > 0"); >>>>> + >>>> Won't it limit all virtio block devices to the same limit? >>>> >>>> It is very common to see multiple virtio-blk devices on the same system >>>> and they probably need different limits. >>>> >>>> Thanks >>> >>> Without looking into the code, that can be done adding a configfs >>> >>> interface and overriding a global value (module param) when it is set >>> from >>> >>> configfs. >>> >>> >> You have many ways to overcome this issue. >> >> For example: >> >> # ls -l /sys/block/vda/mq/ >> drwxr-xr-x 18 root root 0 Sep 5 12:14 0 >> drwxr-xr-x 18 root root 0 Sep 5 12:14 1 >> drwxr-xr-x 18 root root 0 Sep 5 12:14 2 >> drwxr-xr-x 18 root root 0 Sep 5 12:14 3 >> >> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind >> >> # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues > This is global to all virtio-blk devices. You define a global module param but you bind/unbind a specific device. Do you have a better way to control it ? if the device is already probed, it's too late to change the queue_num. > >> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind >> >> # ls -l /sys/block/vda/mq/ >> drwxr-xr-x 10 root root 0 Sep 5 12:17 0 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 1 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 2 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 3 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 4 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 5 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 6 >> drwxr-xr-x 10 root root 0 Sep 5 12:17 7 >> >> -Max. >> >>
On Sun, Sep 05, 2021 at 02:16:58PM +0300, Max Gurtovoy wrote: > > On 9/5/2021 1:19 PM, Leon Romanovsky wrote: > > On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote: > > > On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote: > > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote: > > > > > > +static unsigned int num_request_queues; > > > > > > +module_param_cb(num_request_queues, &queue_count_ops, > > > > > > &num_request_queues, > > > > > > + 0644); > > > > > > +MODULE_PARM_DESC(num_request_queues, > > > > > > + "Number of request queues to use for blk device. > > > > > > Should > 0"); > > > > > > + > > > > > Won't it limit all virtio block devices to the same limit? > > > > > > > > > > It is very common to see multiple virtio-blk devices on the same system > > > > > and they probably need different limits. > > > > > > > > > > Thanks > > > > > > > > Without looking into the code, that can be done adding a configfs > > > > > > > > interface and overriding a global value (module param) when it is set > > > > from > > > > > > > > configfs. > > > > > > > > > > > You have many ways to overcome this issue. > > > > > > For example: > > > > > > # ls -l /sys/block/vda/mq/ > > > drwxr-xr-x 18 root root 0 Sep 5 12:14 0 > > > drwxr-xr-x 18 root root 0 Sep 5 12:14 1 > > > drwxr-xr-x 18 root root 0 Sep 5 12:14 2 > > > drwxr-xr-x 18 root root 0 Sep 5 12:14 3 > > > > > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind > > > > > > # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues > > This is global to all virtio-blk devices. > > You define a global module param but you bind/unbind a specific device. > > Do you have a better way to control it ? One of the possible solutions will be to extend virtio bus to allow setting of such pre-probe parameters. However I don't know if it is really worth doing it, > > if the device is already probed, it's too late to change the queue_num. > > > > > > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind > > > > > > # ls -l /sys/block/vda/mq/ > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 0 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 1 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 2 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 3 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 4 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 5 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 6 > > > drwxr-xr-x 10 root root 0 Sep 5 12:17 7 > > > > > > -Max. > > > > > >
On 9/5/2021 4:10 PM, Leon Romanovsky wrote: > On Sun, Sep 05, 2021 at 02:16:58PM +0300, Max Gurtovoy wrote: >> On 9/5/2021 1:19 PM, Leon Romanovsky wrote: >>> On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote: >>>> On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote: >>>>> On 9/5/2021 12:46 AM, Leon Romanovsky wrote: >>>>>>> +static unsigned int num_request_queues; >>>>>>> +module_param_cb(num_request_queues, &queue_count_ops, >>>>>>> &num_request_queues, >>>>>>> + 0644); >>>>>>> +MODULE_PARM_DESC(num_request_queues, >>>>>>> + "Number of request queues to use for blk device. >>>>>>> Should > 0"); >>>>>>> + >>>>>> Won't it limit all virtio block devices to the same limit? >>>>>> >>>>>> It is very common to see multiple virtio-blk devices on the same system >>>>>> and they probably need different limits. >>>>>> >>>>>> Thanks >>>>> Without looking into the code, that can be done adding a configfs >>>>> >>>>> interface and overriding a global value (module param) when it is set >>>>> from >>>>> >>>>> configfs. >>>>> >>>>> >>>> You have many ways to overcome this issue. >>>> >>>> For example: >>>> >>>> # ls -l /sys/block/vda/mq/ >>>> drwxr-xr-x 18 root root 0 Sep 5 12:14 0 >>>> drwxr-xr-x 18 root root 0 Sep 5 12:14 1 >>>> drwxr-xr-x 18 root root 0 Sep 5 12:14 2 >>>> drwxr-xr-x 18 root root 0 Sep 5 12:14 3 >>>> >>>> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind >>>> >>>> # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues >>> This is global to all virtio-blk devices. >> You define a global module param but you bind/unbind a specific device. >> >> Do you have a better way to control it ? > One of the possible solutions will be to extend virtio bus to allow > setting of such pre-probe parameters. However I don't know if it is > really worth doing it, So lets keep it as it is now. Thanks. >> if the device is already probed, it's too late to change the queue_num. >> >> >>>> # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind >>>> >>>> # ls -l /sys/block/vda/mq/ >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 0 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 1 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 2 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 3 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 4 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 5 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 6 >>>> drwxr-xr-x 10 root root 0 Sep 5 12:17 7 >>>> >>>> -Max. >>>> >>>>
On Sun, Sep 05, 2021 at 01:20:24PM +0300, Leon Romanovsky wrote: > On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote: > > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote: > > > > +static unsigned int num_request_queues; > > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, > > > > + 0644); > > > > +MODULE_PARM_DESC(num_request_queues, > > > > + "Number of request queues to use for blk device. Should > 0"); > > > > + > > > Won't it limit all virtio block devices to the same limit? > > > > > > It is very common to see multiple virtio-blk devices on the same system > > > and they probably need different limits. > > > > > > Thanks > > > > > > Without looking into the code, that can be done adding a configfs > > > > interface and overriding a global value (module param) when it is set from > > > > configfs. > > So why should we do double work instead of providing one working > interface from the beginning? > > Thanks > > > > > The main way to do it is really from the hypervisor. This one is a pretty blunt instrument, Max here says it's useful to reduce memory usage of the driver. If that's the usecase then a global limit seems sufficient.
On Sun, Sep 05, 2021 at 11:15:16AM -0400, Michael S. Tsirkin wrote: > On Sun, Sep 05, 2021 at 01:20:24PM +0300, Leon Romanovsky wrote: > > On Sun, Sep 05, 2021 at 01:49:46AM -0700, Chaitanya Kulkarni wrote: > > > > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote: > > > > > +static unsigned int num_request_queues; > > > > > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, > > > > > + 0644); > > > > > +MODULE_PARM_DESC(num_request_queues, > > > > > + "Number of request queues to use for blk device. Should > 0"); > > > > > + > > > > Won't it limit all virtio block devices to the same limit? > > > > > > > > It is very common to see multiple virtio-blk devices on the same system > > > > and they probably need different limits. > > > > > > > > Thanks > > > > > > > > > Without looking into the code, that can be done adding a configfs > > > > > > interface and overriding a global value (module param) when it is set from > > > > > > configfs. > > > > So why should we do double work instead of providing one working > > interface from the beginning? > > > > Thanks > > > > > > > > > > The main way to do it is really from the hypervisor. This one > is a pretty blunt instrument, Max here says it's useful to reduce > memory usage of the driver. If that's the usecase then a global limit > seems sufficient. How memory will you reduce? It is worth to write it in the commit message. Thanks > > -- > MST >
On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote: > Sometimes a user would like to control the amount of request queues to > be created for a block device. For example, for limiting the memory > footprint of virtio-blk devices. > > Reviewed-by: Christoph Hellwig <hch@lst.de> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> > --- > > changes from v2: > - renamed num_io_queues to num_request_queues (from Stefan) > - added Reviewed-by signatures (from Stefan and Christoph) > > changes from v1: > - use param_set_uint_minmax (from Christoph) > - added "Should > 0" to module description > > Note: This commit apply on top of Jens's branch for-5.15/drivers > > --- > drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- > 1 file changed, 20 insertions(+), 1 deletion(-) > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c > index 4b49df2dfd23..aaa2833a4734 100644 > --- a/drivers/block/virtio_blk.c > +++ b/drivers/block/virtio_blk.c > @@ -24,6 +24,23 @@ > /* The maximum number of sg elements that fit into a virtqueue */ > #define VIRTIO_BLK_MAX_SG_ELEMS 32768 > > +static int virtblk_queue_count_set(const char *val, > + const struct kernel_param *kp) > +{ > + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); > +} > + > +static const struct kernel_param_ops queue_count_ops = { > + .set = virtblk_queue_count_set, > + .get = param_get_uint, > +}; > + > +static unsigned int num_request_queues; > +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, > + 0644); > +MODULE_PARM_DESC(num_request_queues, > + "Number of request queues to use for blk device. Should > 0"); > + > static int major; > static DEFINE_IDA(vd_index_ida); > I wasn't happy with the message here so I tweaked it. Please look at it in linux-next and confirm. Thanks! > @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) > if (err) > num_vqs = 1; > > - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); > + num_vqs = min_t(unsigned int, > + min_not_zero(num_request_queues, nr_cpu_ids), > + num_vqs); > > vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL); > if (!vblk->vqs) > -- > 2.18.1
On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote: > On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote: >> Sometimes a user would like to control the amount of request queues to >> be created for a block device. For example, for limiting the memory >> footprint of virtio-blk devices. >> >> Reviewed-by: Christoph Hellwig <hch@lst.de> >> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> >> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> >> --- >> >> changes from v2: >> - renamed num_io_queues to num_request_queues (from Stefan) >> - added Reviewed-by signatures (from Stefan and Christoph) >> >> changes from v1: >> - use param_set_uint_minmax (from Christoph) >> - added "Should > 0" to module description >> >> Note: This commit apply on top of Jens's branch for-5.15/drivers >> >> --- >> drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- >> 1 file changed, 20 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c >> index 4b49df2dfd23..aaa2833a4734 100644 >> --- a/drivers/block/virtio_blk.c >> +++ b/drivers/block/virtio_blk.c >> @@ -24,6 +24,23 @@ >> /* The maximum number of sg elements that fit into a virtqueue */ >> #define VIRTIO_BLK_MAX_SG_ELEMS 32768 >> >> +static int virtblk_queue_count_set(const char *val, >> + const struct kernel_param *kp) >> +{ >> + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); >> +} >> + >> +static const struct kernel_param_ops queue_count_ops = { >> + .set = virtblk_queue_count_set, >> + .get = param_get_uint, >> +}; >> + >> +static unsigned int num_request_queues; >> +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, >> + 0644); >> +MODULE_PARM_DESC(num_request_queues, >> + "Number of request queues to use for blk device. Should > 0"); >> + >> static int major; >> static DEFINE_IDA(vd_index_ida); >> > I wasn't happy with the message here so I tweaked it. > > Please look at it in linux-next and confirm. Thanks! Looks good. > > >> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) >> if (err) >> num_vqs = 1; >> >> - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); >> + num_vqs = min_t(unsigned int, >> + min_not_zero(num_request_queues, nr_cpu_ids), >> + num_vqs); >> >> vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL); >> if (!vblk->vqs) >> -- >> 2.18.1
On 10/24/2021 10:19 AM, Max Gurtovoy wrote: > > On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote: >> On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote: >>> Sometimes a user would like to control the amount of request queues to >>> be created for a block device. For example, for limiting the memory >>> footprint of virtio-blk devices. >>> >>> Reviewed-by: Christoph Hellwig <hch@lst.de> >>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> >>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> >>> --- >>> >>> changes from v2: >>> - renamed num_io_queues to num_request_queues (from Stefan) >>> - added Reviewed-by signatures (from Stefan and Christoph) >>> >>> changes from v1: >>> - use param_set_uint_minmax (from Christoph) >>> - added "Should > 0" to module description >>> >>> Note: This commit apply on top of Jens's branch for-5.15/drivers >>> >>> --- >>> drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- >>> 1 file changed, 20 insertions(+), 1 deletion(-) >>> >>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c >>> index 4b49df2dfd23..aaa2833a4734 100644 >>> --- a/drivers/block/virtio_blk.c >>> +++ b/drivers/block/virtio_blk.c >>> @@ -24,6 +24,23 @@ >>> /* The maximum number of sg elements that fit into a virtqueue */ >>> #define VIRTIO_BLK_MAX_SG_ELEMS 32768 >>> +static int virtblk_queue_count_set(const char *val, >>> + const struct kernel_param *kp) >>> +{ >>> + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); >>> +} >>> + BTW, I've noticed in your new message you allow setting 0 so you might want to change the code to param_set_uint_minmax(val, kp, 0, nr_cpu_ids); to a case a user will load the module with num_request_queues=0. >>> +static const struct kernel_param_ops queue_count_ops = { >>> + .set = virtblk_queue_count_set, >>> + .get = param_get_uint, >>> +}; >>> + >>> +static unsigned int num_request_queues; >>> +module_param_cb(num_request_queues, &queue_count_ops, >>> &num_request_queues, >>> + 0644); >>> +MODULE_PARM_DESC(num_request_queues, >>> + "Number of request queues to use for blk device. Should > >>> 0"); >>> + >>> static int major; >>> static DEFINE_IDA(vd_index_ida); >> I wasn't happy with the message here so I tweaked it. >> >> Please look at it in linux-next and confirm. Thanks! > > Looks good. > > >> >> >>> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) >>> if (err) >>> num_vqs = 1; >>> - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); >>> + num_vqs = min_t(unsigned int, >>> + min_not_zero(num_request_queues, nr_cpu_ids), >>> + num_vqs); >>> vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), >>> GFP_KERNEL); >>> if (!vblk->vqs) >>> -- >>> 2.18.1
On Sun, Oct 24, 2021 at 11:12:26AM +0300, Max Gurtovoy wrote: > > On 10/24/2021 10:19 AM, Max Gurtovoy wrote: > > > > On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote: > > > On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote: > > > > Sometimes a user would like to control the amount of request queues to > > > > be created for a block device. For example, for limiting the memory > > > > footprint of virtio-blk devices. > > > > > > > > Reviewed-by: Christoph Hellwig <hch@lst.de> > > > > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> > > > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> > > > > --- > > > > > > > > changes from v2: > > > > - renamed num_io_queues to num_request_queues (from Stefan) > > > > - added Reviewed-by signatures (from Stefan and Christoph) > > > > > > > > changes from v1: > > > > - use param_set_uint_minmax (from Christoph) > > > > - added "Should > 0" to module description > > > > > > > > Note: This commit apply on top of Jens's branch for-5.15/drivers > > > > > > > > --- > > > > drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- > > > > 1 file changed, 20 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c > > > > index 4b49df2dfd23..aaa2833a4734 100644 > > > > --- a/drivers/block/virtio_blk.c > > > > +++ b/drivers/block/virtio_blk.c > > > > @@ -24,6 +24,23 @@ > > > > /* The maximum number of sg elements that fit into a virtqueue */ > > > > #define VIRTIO_BLK_MAX_SG_ELEMS 32768 > > > > +static int virtblk_queue_count_set(const char *val, > > > > + const struct kernel_param *kp) > > > > +{ > > > > + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); > > > > +} > > > > + > > BTW, I've noticed in your new message you allow setting 0 so you might want > to change the code to > > param_set_uint_minmax(val, kp, 0, nr_cpu_ids); > > to a case a user will load the module with num_request_queues=0. Oh. So as written the default forces 1 queue? Send a patch please! > > > > +static const struct kernel_param_ops queue_count_ops = { > > > > + .set = virtblk_queue_count_set, > > > > + .get = param_get_uint, > > > > +}; > > > > + > > > > +static unsigned int num_request_queues; > > > > +module_param_cb(num_request_queues, &queue_count_ops, > > > > &num_request_queues, > > > > + 0644); > > > > +MODULE_PARM_DESC(num_request_queues, > > > > + "Number of request queues to use for blk device. > > > > Should > 0"); > > > > + > > > > static int major; > > > > static DEFINE_IDA(vd_index_ida); > > > I wasn't happy with the message here so I tweaked it. > > > > > > Please look at it in linux-next and confirm. Thanks! > > > > Looks good. > > > > > > > > > > > > > > @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) > > > > if (err) > > > > num_vqs = 1; > > > > - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); > > > > + num_vqs = min_t(unsigned int, > > > > + min_not_zero(num_request_queues, nr_cpu_ids), > > > > + num_vqs); > > > > vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), > > > > GFP_KERNEL); > > > > if (!vblk->vqs) > > > > -- > > > > 2.18.1
On 10/24/2021 11:48 AM, Michael S. Tsirkin wrote: > On Sun, Oct 24, 2021 at 11:12:26AM +0300, Max Gurtovoy wrote: >> On 10/24/2021 10:19 AM, Max Gurtovoy wrote: >>> On 10/22/2021 12:30 PM, Michael S. Tsirkin wrote: >>>> On Thu, Sep 02, 2021 at 11:46:22PM +0300, Max Gurtovoy wrote: >>>>> Sometimes a user would like to control the amount of request queues to >>>>> be created for a block device. For example, for limiting the memory >>>>> footprint of virtio-blk devices. >>>>> >>>>> Reviewed-by: Christoph Hellwig <hch@lst.de> >>>>> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> >>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com> >>>>> --- >>>>> >>>>> changes from v2: >>>>> - renamed num_io_queues to num_request_queues (from Stefan) >>>>> - added Reviewed-by signatures (from Stefan and Christoph) >>>>> >>>>> changes from v1: >>>>> - use param_set_uint_minmax (from Christoph) >>>>> - added "Should > 0" to module description >>>>> >>>>> Note: This commit apply on top of Jens's branch for-5.15/drivers >>>>> >>>>> --- >>>>> drivers/block/virtio_blk.c | 21 ++++++++++++++++++++- >>>>> 1 file changed, 20 insertions(+), 1 deletion(-) >>>>> >>>>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c >>>>> index 4b49df2dfd23..aaa2833a4734 100644 >>>>> --- a/drivers/block/virtio_blk.c >>>>> +++ b/drivers/block/virtio_blk.c >>>>> @@ -24,6 +24,23 @@ >>>>> /* The maximum number of sg elements that fit into a virtqueue */ >>>>> #define VIRTIO_BLK_MAX_SG_ELEMS 32768 >>>>> +static int virtblk_queue_count_set(const char *val, >>>>> + const struct kernel_param *kp) >>>>> +{ >>>>> + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); >>>>> +} >>>>> + >> BTW, I've noticed in your new message you allow setting 0 so you might want >> to change the code to >> >> param_set_uint_minmax(val, kp, 0, nr_cpu_ids); >> >> to a case a user will load the module with num_request_queues=0. > Oh. So as written the default forces 1 queue? > Send a patch please! No. The default (if nobody touch this param) is 0 and everything works as today. but the user can't do "modprobe virtio_blk num_request_queues=0". My comment was right (should be set > 0). You changed and wrote "0 for no limit." but the user can't set to 0. So if you want the user to change to 0, please change the above code as I mentioned. > >>>>> +static const struct kernel_param_ops queue_count_ops = { >>>>> + .set = virtblk_queue_count_set, >>>>> + .get = param_get_uint, >>>>> +}; >>>>> + >>>>> +static unsigned int num_request_queues; >>>>> +module_param_cb(num_request_queues, &queue_count_ops, >>>>> &num_request_queues, >>>>> + 0644); >>>>> +MODULE_PARM_DESC(num_request_queues, >>>>> + "Number of request queues to use for blk device. >>>>> Should > 0"); >>>>> + >>>>> static int major; >>>>> static DEFINE_IDA(vd_index_ida); >>>> I wasn't happy with the message here so I tweaked it. >>>> >>>> Please look at it in linux-next and confirm. Thanks! >>> Looks good. >>> >>> >>>> >>>>> @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) >>>>> if (err) >>>>> num_vqs = 1; >>>>> - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); >>>>> + num_vqs = min_t(unsigned int, >>>>> + min_not_zero(num_request_queues, nr_cpu_ids), >>>>> + num_vqs); >>>>> vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), >>>>> GFP_KERNEL); >>>>> if (!vblk->vqs) >>>>> -- >>>>> 2.18.1
diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 4b49df2dfd23..aaa2833a4734 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -24,6 +24,23 @@ /* The maximum number of sg elements that fit into a virtqueue */ #define VIRTIO_BLK_MAX_SG_ELEMS 32768 +static int virtblk_queue_count_set(const char *val, + const struct kernel_param *kp) +{ + return param_set_uint_minmax(val, kp, 1, nr_cpu_ids); +} + +static const struct kernel_param_ops queue_count_ops = { + .set = virtblk_queue_count_set, + .get = param_get_uint, +}; + +static unsigned int num_request_queues; +module_param_cb(num_request_queues, &queue_count_ops, &num_request_queues, + 0644); +MODULE_PARM_DESC(num_request_queues, + "Number of request queues to use for blk device. Should > 0"); + static int major; static DEFINE_IDA(vd_index_ida); @@ -501,7 +518,9 @@ static int init_vq(struct virtio_blk *vblk) if (err) num_vqs = 1; - num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs); + num_vqs = min_t(unsigned int, + min_not_zero(num_request_queues, nr_cpu_ids), + num_vqs); vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL); if (!vblk->vqs)