| Message ID | 20200922105209.5284-2-sjpark@amazon.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | xen-blk(back|front): Let users disable persistent grants | expand |
On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote: > From: SeongJae Park <sjpark@amazon.de> > > Persistent grants feature provides high scalability. On some small > systems, however, it could incur data copy overheads[1] and thus it is > required to be disabled. But, there is no option to disable it. For > the reason, this commit adds a module parameter for disabling of the > feature. > > [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability > > Signed-off-by: Anthony Liguori <aliguori@amazon.com> > Signed-off-by: SeongJae Park <sjpark@amazon.de> > --- > .../ABI/testing/sysfs-driver-xen-blkback | 9 ++++++ > drivers/block/xen-blkback/xenbus.c | 28 ++++++++++++++----- > 2 files changed, 30 insertions(+), 7 deletions(-) > > diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback > index ecb7942ff146..ac2947b98950 100644 > --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback > +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback > @@ -35,3 +35,12 @@ Description: > controls the duration in milliseconds that blkback will not > cache any page not backed by a grant mapping. > The default is 10ms. > + > +What: /sys/module/xen_blkback/parameters/feature_persistent > +Date: September 2020 > +KernelVersion: 5.10 > +Contact: SeongJae Park <sjpark@amazon.de> > +Description: > + Whether to enable the persistent grants feature or not. Note > + that this option only takes effect on newly created backends. > + The default is Y (enable). > diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c > index b9aa5d1ac10b..8a95ddd08b13 100644 > --- a/drivers/block/xen-blkback/xenbus.c > +++ b/drivers/block/xen-blkback/xenbus.c > @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) > > /* ** Connection ** */ > > +/* Enable the persistent grants feature. */ > +static bool feature_persistent = true; > +module_param(feature_persistent, bool, 0644); > +MODULE_PARM_DESC(feature_persistent, > + "Enables the persistent grants feature"); > + > /* > * Write the physical details regarding the block device to the store, and > * switch to Connected state. > @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) > > xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); > > - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); > - if (err) { > - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", > - dev->nodename); > - goto abort; > + if (feature_persistent) { > + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", > + "%u", feature_persistent); > + if (err) { > + xenbus_dev_fatal(dev, err, > + "writing %s/feature-persistent", > + dev->nodename); > + goto abort; > + } > } > > err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", > @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) > xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); > return -ENOSYS; > } > - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", > - 0); > + if (feature_persistent) > + pers_grants = xenbus_read_unsigned(dev->otherend, > + "feature-persistent", 0); > + else > + pers_grants = 0; > + Sorry for not realizing earlier, but looking at it again I think you need to cache the value of feature_persistent when it's first used in the blkback state data, so that it's consistent. What would happen for example with the following flow (assume a persistent grants enabled frontend): feature_persistent = false connect(...) feature-persistent is not written to xenstore User changes feature_persistent = true connect_ring(...) pers_grants = true, because feature-persistent is set unconditionally by the frontend and feature_persistent variable is now true. Then blkback will try to use persistent grants and the whole connection will malfunction because the frontend won't. The other option is to prevent changing the variable when there are blkback instances already running. Thanks, Roger.
On Tue, 22 Sep 2020 13:12:59 +0200 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote: > > From: SeongJae Park <sjpark@amazon.de> > > > > Persistent grants feature provides high scalability. On some small > > systems, however, it could incur data copy overheads[1] and thus it is > > required to be disabled. But, there is no option to disable it. For > > the reason, this commit adds a module parameter for disabling of the > > feature. > > > > [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability > > > > Signed-off-by: Anthony Liguori <aliguori@amazon.com> > > Signed-off-by: SeongJae Park <sjpark@amazon.de> > > --- > > .../ABI/testing/sysfs-driver-xen-blkback | 9 ++++++ > > drivers/block/xen-blkback/xenbus.c | 28 ++++++++++++++----- > > 2 files changed, 30 insertions(+), 7 deletions(-) > > > > diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback > > index ecb7942ff146..ac2947b98950 100644 > > --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback > > +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback > > @@ -35,3 +35,12 @@ Description: > > controls the duration in milliseconds that blkback will not > > cache any page not backed by a grant mapping. > > The default is 10ms. > > + > > +What: /sys/module/xen_blkback/parameters/feature_persistent > > +Date: September 2020 > > +KernelVersion: 5.10 > > +Contact: SeongJae Park <sjpark@amazon.de> > > +Description: > > + Whether to enable the persistent grants feature or not. Note > > + that this option only takes effect on newly created backends. > > + The default is Y (enable). > > diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c > > index b9aa5d1ac10b..8a95ddd08b13 100644 > > --- a/drivers/block/xen-blkback/xenbus.c > > +++ b/drivers/block/xen-blkback/xenbus.c > > @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) > > > > /* ** Connection ** */ > > > > +/* Enable the persistent grants feature. */ > > +static bool feature_persistent = true; > > +module_param(feature_persistent, bool, 0644); > > +MODULE_PARM_DESC(feature_persistent, > > + "Enables the persistent grants feature"); > > + > > /* > > * Write the physical details regarding the block device to the store, and > > * switch to Connected state. > > @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) > > > > xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); > > > > - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); > > - if (err) { > > - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", > > - dev->nodename); > > - goto abort; > > + if (feature_persistent) { > > + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", > > + "%u", feature_persistent); > > + if (err) { > > + xenbus_dev_fatal(dev, err, > > + "writing %s/feature-persistent", > > + dev->nodename); > > + goto abort; > > + } > > } > > > > err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", > > @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) > > xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); > > return -ENOSYS; > > } > > - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", > > - 0); > > + if (feature_persistent) > > + pers_grants = xenbus_read_unsigned(dev->otherend, > > + "feature-persistent", 0); > > + else > > + pers_grants = 0; > > + > > Sorry for not realizing earlier, but looking at it again I think you > need to cache the value of feature_persistent when it's first used in > the blkback state data, so that it's consistent. > > What would happen for example with the following flow (assume a > persistent grants enabled frontend): > > feature_persistent = false > > connect(...) > feature-persistent is not written to xenstore > > User changes feature_persistent = true > > connect_ring(...) > pers_grants = true, because feature-persistent is set unconditionally > by the frontend and feature_persistent variable is now true. > > Then blkback will try to use persistent grants and the whole > connection will malfunction because the frontend won't. Ah, you're right. I should also catch this before but didn't, sorry. > > The other option is to prevent changing the variable when there are > blkback instances already running. I think storing the option value in xenstore would be simpler. That said, if you prefer this way, please let me know. Thanks, SeongJae Park
On Tue, Sep 22, 2020 at 01:26:38PM +0200, SeongJae Park wrote: > On Tue, 22 Sep 2020 13:12:59 +0200 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > > > On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote: > > > From: SeongJae Park <sjpark@amazon.de> > > > > > > Persistent grants feature provides high scalability. On some small > > > systems, however, it could incur data copy overheads[1] and thus it is > > > required to be disabled. But, there is no option to disable it. For > > > the reason, this commit adds a module parameter for disabling of the > > > feature. > > > > > > [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability > > > > > > Signed-off-by: Anthony Liguori <aliguori@amazon.com> > > > Signed-off-by: SeongJae Park <sjpark@amazon.de> > > > --- > > > .../ABI/testing/sysfs-driver-xen-blkback | 9 ++++++ > > > drivers/block/xen-blkback/xenbus.c | 28 ++++++++++++++----- > > > 2 files changed, 30 insertions(+), 7 deletions(-) > > > > > > diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback > > > index ecb7942ff146..ac2947b98950 100644 > > > --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback > > > +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback > > > @@ -35,3 +35,12 @@ Description: > > > controls the duration in milliseconds that blkback will not > > > cache any page not backed by a grant mapping. > > > The default is 10ms. > > > + > > > +What: /sys/module/xen_blkback/parameters/feature_persistent > > > +Date: September 2020 > > > +KernelVersion: 5.10 > > > +Contact: SeongJae Park <sjpark@amazon.de> > > > +Description: > > > + Whether to enable the persistent grants feature or not. Note > > > + that this option only takes effect on newly created backends. > > > + The default is Y (enable). > > > diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c > > > index b9aa5d1ac10b..8a95ddd08b13 100644 > > > --- a/drivers/block/xen-blkback/xenbus.c > > > +++ b/drivers/block/xen-blkback/xenbus.c > > > @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) > > > > > > /* ** Connection ** */ > > > > > > +/* Enable the persistent grants feature. */ > > > +static bool feature_persistent = true; > > > +module_param(feature_persistent, bool, 0644); > > > +MODULE_PARM_DESC(feature_persistent, > > > + "Enables the persistent grants feature"); > > > + > > > /* > > > * Write the physical details regarding the block device to the store, and > > > * switch to Connected state. > > > @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) > > > > > > xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); > > > > > > - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); > > > - if (err) { > > > - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", > > > - dev->nodename); > > > - goto abort; > > > + if (feature_persistent) { > > > + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", > > > + "%u", feature_persistent); > > > + if (err) { > > > + xenbus_dev_fatal(dev, err, > > > + "writing %s/feature-persistent", > > > + dev->nodename); > > > + goto abort; > > > + } > > > } > > > > > > err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", > > > @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) > > > xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); > > > return -ENOSYS; > > > } > > > - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", > > > - 0); > > > + if (feature_persistent) > > > + pers_grants = xenbus_read_unsigned(dev->otherend, > > > + "feature-persistent", 0); > > > + else > > > + pers_grants = 0; > > > + > > > > Sorry for not realizing earlier, but looking at it again I think you > > need to cache the value of feature_persistent when it's first used in > > the blkback state data, so that it's consistent. > > > > What would happen for example with the following flow (assume a > > persistent grants enabled frontend): > > > > feature_persistent = false > > > > connect(...) > > feature-persistent is not written to xenstore > > > > User changes feature_persistent = true > > > > connect_ring(...) > > pers_grants = true, because feature-persistent is set unconditionally > > by the frontend and feature_persistent variable is now true. > > > > Then blkback will try to use persistent grants and the whole > > connection will malfunction because the frontend won't. > > Ah, you're right. I should also catch this before but didn't, sorry. > > > > > The other option is to prevent changing the variable when there are > > blkback instances already running. > > I think storing the option value in xenstore would be simpler. That said, if > you prefer this way, please let me know. If possible I prefer to avoid reading from xenstore because it's slow, very slow compared to storing this somewhere in the blkback structure. Could you use the feature_gnt_persistent field in xen_vbd maybe as a place to cache the value before it's set with the final negotiated value between the frontend and the backend? Thanks, Roger.
On 22.09.20 13:26, SeongJae Park wrote: > On Tue, 22 Sep 2020 13:12:59 +0200 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > >> On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote: >>> From: SeongJae Park <sjpark@amazon.de> >>> >>> Persistent grants feature provides high scalability. On some small >>> systems, however, it could incur data copy overheads[1] and thus it is >>> required to be disabled. But, there is no option to disable it. For >>> the reason, this commit adds a module parameter for disabling of the >>> feature. >>> >>> [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability >>> >>> Signed-off-by: Anthony Liguori <aliguori@amazon.com> >>> Signed-off-by: SeongJae Park <sjpark@amazon.de> >>> --- >>> .../ABI/testing/sysfs-driver-xen-blkback | 9 ++++++ >>> drivers/block/xen-blkback/xenbus.c | 28 ++++++++++++++----- >>> 2 files changed, 30 insertions(+), 7 deletions(-) >>> >>> diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback >>> index ecb7942ff146..ac2947b98950 100644 >>> --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback >>> +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback >>> @@ -35,3 +35,12 @@ Description: >>> controls the duration in milliseconds that blkback will not >>> cache any page not backed by a grant mapping. >>> The default is 10ms. >>> + >>> +What: /sys/module/xen_blkback/parameters/feature_persistent >>> +Date: September 2020 >>> +KernelVersion: 5.10 >>> +Contact: SeongJae Park <sjpark@amazon.de> >>> +Description: >>> + Whether to enable the persistent grants feature or not. Note >>> + that this option only takes effect on newly created backends. >>> + The default is Y (enable). >>> diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c >>> index b9aa5d1ac10b..8a95ddd08b13 100644 >>> --- a/drivers/block/xen-blkback/xenbus.c >>> +++ b/drivers/block/xen-blkback/xenbus.c >>> @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) >>> >>> /* ** Connection ** */ >>> >>> +/* Enable the persistent grants feature. */ >>> +static bool feature_persistent = true; >>> +module_param(feature_persistent, bool, 0644); >>> +MODULE_PARM_DESC(feature_persistent, >>> + "Enables the persistent grants feature"); >>> + >>> /* >>> * Write the physical details regarding the block device to the store, and >>> * switch to Connected state. >>> @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) >>> >>> xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); >>> >>> - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); >>> - if (err) { >>> - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", >>> - dev->nodename); >>> - goto abort; >>> + if (feature_persistent) { >>> + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", >>> + "%u", feature_persistent); >>> + if (err) { >>> + xenbus_dev_fatal(dev, err, >>> + "writing %s/feature-persistent", >>> + dev->nodename); >>> + goto abort; >>> + } >>> } >>> >>> err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", >>> @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) >>> xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); >>> return -ENOSYS; >>> } >>> - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", >>> - 0); >>> + if (feature_persistent) >>> + pers_grants = xenbus_read_unsigned(dev->otherend, >>> + "feature-persistent", 0); >>> + else >>> + pers_grants = 0; >>> + >> >> Sorry for not realizing earlier, but looking at it again I think you >> need to cache the value of feature_persistent when it's first used in >> the blkback state data, so that it's consistent. >> >> What would happen for example with the following flow (assume a >> persistent grants enabled frontend): >> >> feature_persistent = false >> >> connect(...) >> feature-persistent is not written to xenstore >> >> User changes feature_persistent = true >> >> connect_ring(...) >> pers_grants = true, because feature-persistent is set unconditionally >> by the frontend and feature_persistent variable is now true. >> >> Then blkback will try to use persistent grants and the whole >> connection will malfunction because the frontend won't. > > Ah, you're right. I should also catch this before but didn't, sorry. > >> >> The other option is to prevent changing the variable when there are >> blkback instances already running. > > I think storing the option value in xenstore would be simpler. That said, if > you prefer this way, please let me know. No, Xenstore isn't the right place for that. This is a local implementation detail of blkback and shouldn't be exported to Xenstore. Juergen
On Tue, 22 Sep 2020 13:35:11 +0200 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > On Tue, Sep 22, 2020 at 01:26:38PM +0200, SeongJae Park wrote: > > On Tue, 22 Sep 2020 13:12:59 +0200 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > > > > > On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote: > > > > From: SeongJae Park <sjpark@amazon.de> > > > > > > > > Persistent grants feature provides high scalability. On some small > > > > systems, however, it could incur data copy overheads[1] and thus it is > > > > required to be disabled. But, there is no option to disable it. For > > > > the reason, this commit adds a module parameter for disabling of the > > > > feature. > > > > > > > > [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability > > > > > > > > Signed-off-by: Anthony Liguori <aliguori@amazon.com> > > > > Signed-off-by: SeongJae Park <sjpark@amazon.de> > > > > --- > > > > .../ABI/testing/sysfs-driver-xen-blkback | 9 ++++++ > > > > drivers/block/xen-blkback/xenbus.c | 28 ++++++++++++++----- > > > > 2 files changed, 30 insertions(+), 7 deletions(-) > > > > > > > > diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback > > > > index ecb7942ff146..ac2947b98950 100644 > > > > --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback > > > > +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback > > > > @@ -35,3 +35,12 @@ Description: > > > > controls the duration in milliseconds that blkback will not > > > > cache any page not backed by a grant mapping. > > > > The default is 10ms. > > > > + > > > > +What: /sys/module/xen_blkback/parameters/feature_persistent > > > > +Date: September 2020 > > > > +KernelVersion: 5.10 > > > > +Contact: SeongJae Park <sjpark@amazon.de> > > > > +Description: > > > > + Whether to enable the persistent grants feature or not. Note > > > > + that this option only takes effect on newly created backends. > > > > + The default is Y (enable). > > > > diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c > > > > index b9aa5d1ac10b..8a95ddd08b13 100644 > > > > --- a/drivers/block/xen-blkback/xenbus.c > > > > +++ b/drivers/block/xen-blkback/xenbus.c > > > > @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) > > > > > > > > /* ** Connection ** */ > > > > > > > > +/* Enable the persistent grants feature. */ > > > > +static bool feature_persistent = true; > > > > +module_param(feature_persistent, bool, 0644); > > > > +MODULE_PARM_DESC(feature_persistent, > > > > + "Enables the persistent grants feature"); > > > > + > > > > /* > > > > * Write the physical details regarding the block device to the store, and > > > > * switch to Connected state. > > > > @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) > > > > > > > > xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); > > > > > > > > - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); > > > > - if (err) { > > > > - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", > > > > - dev->nodename); > > > > - goto abort; > > > > + if (feature_persistent) { > > > > + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", > > > > + "%u", feature_persistent); > > > > + if (err) { > > > > + xenbus_dev_fatal(dev, err, > > > > + "writing %s/feature-persistent", > > > > + dev->nodename); > > > > + goto abort; > > > > + } > > > > } > > > > > > > > err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", > > > > @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) > > > > xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); > > > > return -ENOSYS; > > > > } > > > > - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", > > > > - 0); > > > > + if (feature_persistent) > > > > + pers_grants = xenbus_read_unsigned(dev->otherend, > > > > + "feature-persistent", 0); > > > > + else > > > > + pers_grants = 0; > > > > + > > > > > > Sorry for not realizing earlier, but looking at it again I think you > > > need to cache the value of feature_persistent when it's first used in > > > the blkback state data, so that it's consistent. > > > > > > What would happen for example with the following flow (assume a > > > persistent grants enabled frontend): > > > > > > feature_persistent = false > > > > > > connect(...) > > > feature-persistent is not written to xenstore > > > > > > User changes feature_persistent = true > > > > > > connect_ring(...) > > > pers_grants = true, because feature-persistent is set unconditionally > > > by the frontend and feature_persistent variable is now true. > > > > > > Then blkback will try to use persistent grants and the whole > > > connection will malfunction because the frontend won't. > > > > Ah, you're right. I should also catch this before but didn't, sorry. > > > > > > > > The other option is to prevent changing the variable when there are > > > blkback instances already running. > > > > I think storing the option value in xenstore would be simpler. That said, if > > you prefer this way, please let me know. > > If possible I prefer to avoid reading from xenstore because it's slow, > very slow compared to storing this somewhere in the blkback > structure. Agreed. > > Could you use the feature_gnt_persistent field in xen_vbd maybe as a > place to cache the value before it's set with the final negotiated > value between the frontend and the backend? Ok, I will do so in the next version. Thanks, SeongJae Park
On Tue, 22 Sep 2020 13:35:30 +0200 "Jürgen Groß" <jgross@suse.com> wrote: > On 22.09.20 13:26, SeongJae Park wrote: > > On Tue, 22 Sep 2020 13:12:59 +0200 "Roger Pau Monné" <roger.pau@citrix.com> wrote: > > > >> On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote: > >>> From: SeongJae Park <sjpark@amazon.de> > >>> > >>> Persistent grants feature provides high scalability. On some small > >>> systems, however, it could incur data copy overheads[1] and thus it is > >>> required to be disabled. But, there is no option to disable it. For > >>> the reason, this commit adds a module parameter for disabling of the > >>> feature. > >>> > >>> [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability > >>> > >>> Signed-off-by: Anthony Liguori <aliguori@amazon.com> > >>> Signed-off-by: SeongJae Park <sjpark@amazon.de> > >>> --- > >>> .../ABI/testing/sysfs-driver-xen-blkback | 9 ++++++ > >>> drivers/block/xen-blkback/xenbus.c | 28 ++++++++++++++----- > >>> 2 files changed, 30 insertions(+), 7 deletions(-) > >>> > >>> diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback > >>> index ecb7942ff146..ac2947b98950 100644 > >>> --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback > >>> +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback > >>> @@ -35,3 +35,12 @@ Description: > >>> controls the duration in milliseconds that blkback will not > >>> cache any page not backed by a grant mapping. > >>> The default is 10ms. > >>> + > >>> +What: /sys/module/xen_blkback/parameters/feature_persistent > >>> +Date: September 2020 > >>> +KernelVersion: 5.10 > >>> +Contact: SeongJae Park <sjpark@amazon.de> > >>> +Description: > >>> + Whether to enable the persistent grants feature or not. Note > >>> + that this option only takes effect on newly created backends. > >>> + The default is Y (enable). > >>> diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c > >>> index b9aa5d1ac10b..8a95ddd08b13 100644 > >>> --- a/drivers/block/xen-blkback/xenbus.c > >>> +++ b/drivers/block/xen-blkback/xenbus.c > >>> @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) > >>> > >>> /* ** Connection ** */ > >>> > >>> +/* Enable the persistent grants feature. */ > >>> +static bool feature_persistent = true; > >>> +module_param(feature_persistent, bool, 0644); > >>> +MODULE_PARM_DESC(feature_persistent, > >>> + "Enables the persistent grants feature"); > >>> + > >>> /* > >>> * Write the physical details regarding the block device to the store, and > >>> * switch to Connected state. > >>> @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) > >>> > >>> xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); > >>> > >>> - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); > >>> - if (err) { > >>> - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", > >>> - dev->nodename); > >>> - goto abort; > >>> + if (feature_persistent) { > >>> + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", > >>> + "%u", feature_persistent); > >>> + if (err) { > >>> + xenbus_dev_fatal(dev, err, > >>> + "writing %s/feature-persistent", > >>> + dev->nodename); > >>> + goto abort; > >>> + } > >>> } > >>> > >>> err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", > >>> @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) > >>> xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); > >>> return -ENOSYS; > >>> } > >>> - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", > >>> - 0); > >>> + if (feature_persistent) > >>> + pers_grants = xenbus_read_unsigned(dev->otherend, > >>> + "feature-persistent", 0); > >>> + else > >>> + pers_grants = 0; > >>> + > >> > >> Sorry for not realizing earlier, but looking at it again I think you > >> need to cache the value of feature_persistent when it's first used in > >> the blkback state data, so that it's consistent. > >> > >> What would happen for example with the following flow (assume a > >> persistent grants enabled frontend): > >> > >> feature_persistent = false > >> > >> connect(...) > >> feature-persistent is not written to xenstore > >> > >> User changes feature_persistent = true > >> > >> connect_ring(...) > >> pers_grants = true, because feature-persistent is set unconditionally > >> by the frontend and feature_persistent variable is now true. > >> > >> Then blkback will try to use persistent grants and the whole > >> connection will malfunction because the frontend won't. > > > > Ah, you're right. I should also catch this before but didn't, sorry. > > > >> > >> The other option is to prevent changing the variable when there are > >> blkback instances already running. > > > > I think storing the option value in xenstore would be simpler. That said, if > > you prefer this way, please let me know. > > No, Xenstore isn't the right place for that. This is a local > implementation detail of blkback and shouldn't be exported to Xenstore. Agreed. I will try using the 'feature_gnt_persistent' as Roger recommended. Thanks, SeongJae Park
diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback b/Documentation/ABI/testing/sysfs-driver-xen-blkback index ecb7942ff146..ac2947b98950 100644 --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback @@ -35,3 +35,12 @@ Description: controls the duration in milliseconds that blkback will not cache any page not backed by a grant mapping. The default is 10ms. + +What: /sys/module/xen_blkback/parameters/feature_persistent +Date: September 2020 +KernelVersion: 5.10 +Contact: SeongJae Park <sjpark@amazon.de> +Description: + Whether to enable the persistent grants feature or not. Note + that this option only takes effect on newly created backends. + The default is Y (enable). diff --git a/drivers/block/xen-blkback/xenbus.c b/drivers/block/xen-blkback/xenbus.c index b9aa5d1ac10b..8a95ddd08b13 100644 --- a/drivers/block/xen-blkback/xenbus.c +++ b/drivers/block/xen-blkback/xenbus.c @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device *dev) /* ** Connection ** */ +/* Enable the persistent grants feature. */ +static bool feature_persistent = true; +module_param(feature_persistent, bool, 0644); +MODULE_PARM_DESC(feature_persistent, + "Enables the persistent grants feature"); + /* * Write the physical details regarding the block device to the store, and * switch to Connected state. @@ -906,11 +912,15 @@ static void connect(struct backend_info *be) xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support); - err = xenbus_printf(xbt, dev->nodename, "feature-persistent", "%u", 1); - if (err) { - xenbus_dev_fatal(dev, err, "writing %s/feature-persistent", - dev->nodename); - goto abort; + if (feature_persistent) { + err = xenbus_printf(xbt, dev->nodename, "feature-persistent", + "%u", feature_persistent); + if (err) { + xenbus_dev_fatal(dev, err, + "writing %s/feature-persistent", + dev->nodename); + goto abort; + } } err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu", @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be) xenbus_dev_fatal(dev, err, "unknown fe protocol %s", protocol); return -ENOSYS; } - pers_grants = xenbus_read_unsigned(dev->otherend, "feature-persistent", - 0); + if (feature_persistent) + pers_grants = xenbus_read_unsigned(dev->otherend, + "feature-persistent", 0); + else + pers_grants = 0; + blkif->vbd.feature_gnt_persistent = pers_grants; blkif->vbd.overflow_max_grants = 0;