| Message ID | 1577959968-19427-2-git-send-email-anand.jain@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | readmirror feature (sysfs and in-memory only approach) | expand |
On 1/2/20 5:12 AM, Anand Jain wrote: > As of now we use %pid method to read stripped mirrored data. So > application's process id determines the stripe id to be read. This type > of read IO routing typically helps in a system with many small > independent applications tying to read random data. On the other hand > the %pid based read IO distribution policy is inefficient if there is a > single application trying to read large data and the overall disk > bandwidth remains under utilized. > > So this patch introduces a framework where we could add more readmirror > policies, such as routing the IO based on device's wait-queue or manual > when we have a read-preferred device or a policy based on the target > storage caching. > > Signed-off-by: Anand Jain <anand.jain@oracle.com> > --- > v2: Declare fs_devices::readmirror as u8 instead of atomic_t > A small change in comment and change log wordings. > > fs/btrfs/volumes.c | 16 +++++++++++++++- > fs/btrfs/volumes.h | 8 ++++++++ > 2 files changed, 23 insertions(+), 1 deletion(-) > > diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c > index c95e47aa84f8..e26af766f2b9 100644 > --- a/fs/btrfs/volumes.c > +++ b/fs/btrfs/volumes.c > @@ -1162,6 +1162,8 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, > fs_devices->opened = 1; > fs_devices->latest_bdev = latest_dev->bdev; > fs_devices->total_rw_bytes = 0; > + /* Set the default readmirror policy */ > + fs_devices->readmirror = BTRFS_READMIRROR_DEFAULT; > out: > return ret; > } > @@ -5300,7 +5302,19 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info, > else > num_stripes = map->num_stripes; > > - preferred_mirror = first + current->pid % num_stripes; > + switch (fs_info->fs_devices->readmirror) { > + case BTRFS_READMIRROR_BY_PID: > + preferred_mirror = first + current->pid % num_stripes; > + break; > + default: > + /* > + * Shouln't happen, just warn and use by_pid instead of failing. > + */ > + btrfs_warn_rl(fs_info, > + "unknown readmirror type %u, fallback to by_pid", > + fs_info->fs_devices->readmirror); > + preferred_mirror = first + current->pid % num_stripes; > + } > > if (dev_replace_is_ongoing && > fs_info->dev_replace.cont_reading_from_srcdev_mode == > diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h > index 68021d1ee216..f5f091f3c72b 100644 > --- a/fs/btrfs/volumes.h > +++ b/fs/btrfs/volumes.h > @@ -209,6 +209,12 @@ struct btrfs_device { > BTRFS_DEVICE_GETSET_FUNCS(disk_total_bytes); > BTRFS_DEVICE_GETSET_FUNCS(bytes_used); > > +/* readmirror_policy types */ > +#define BTRFS_READMIRROR_DEFAULT BTRFS_READMIRROR_BY_PID > +enum btrfs_readmirror_policy_type { > + BTRFS_READMIRROR_BY_PID, > +}; > + > struct btrfs_fs_devices { > u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */ > u8 metadata_uuid[BTRFS_FSID_SIZE]; > @@ -260,6 +266,8 @@ struct btrfs_fs_devices { > struct kobject *devices_kobj; > struct kobject *devinfo_kobj; > struct completion kobj_unregister; > + > + u8 readmirror; The only valid values for this are the enum, so make this enum btrfs_readmirror_policy_type readmirror; Thanks, Josef
On 02/01/2020 10:12, Anand Jain wrote: > As of now we use %pid method to read stripped mirrored data. So > application's process id determines the stripe id to be read. This type > of read IO routing typically helps in a system with many small > independent applications tying to read random data. On the other hand > the %pid based read IO distribution policy is inefficient if there is a > single application trying to read large data and the overall disk > bandwidth remains under utilized. > > So this patch introduces a framework where we could add more readmirror > policies, such as routing the IO based on device's wait-queue or manual > when we have a read-preferred device or a policy based on the target > storage caching. I think the idea is good but that it would be cleaner if the tunable was named read_policy rather than readmirror as it's more obvious that it contains a policy tunable. Do you envisage allowing more than one policy to be active for a filesystem? If not, what about using the same structure as the CPU frequency and block IO schedulers with the format #cat /sys/block/sda/queue/scheduler noop [deadline] cfq Such that btrfs would (eventually) have something like #cat /sys/fs/btrfs/UUID/read_policy by_pid [user_defined_device] by_shortest_queue And the policy would be changed by echo'ing the new policy name to the read_policy kobject. Steve
On 3/1/20 12:24 AM, Josef Bacik wrote: > On 1/2/20 5:12 AM, Anand Jain wrote: >> As of now we use %pid method to read stripped mirrored data. So >> application's process id determines the stripe id to be read. This type >> of read IO routing typically helps in a system with many small >> independent applications tying to read random data. On the other hand >> the %pid based read IO distribution policy is inefficient if there is a >> single application trying to read large data and the overall disk >> bandwidth remains under utilized. >> >> So this patch introduces a framework where we could add more readmirror >> policies, such as routing the IO based on device's wait-queue or manual >> when we have a read-preferred device or a policy based on the target >> storage caching. >> >> Signed-off-by: Anand Jain <anand.jain@oracle.com> >> --- >> v2: Declare fs_devices::readmirror as u8 instead of atomic_t >> A small change in comment and change log wordings. >> >> fs/btrfs/volumes.c | 16 +++++++++++++++- >> fs/btrfs/volumes.h | 8 ++++++++ >> 2 files changed, 23 insertions(+), 1 deletion(-) >> >> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c >> index c95e47aa84f8..e26af766f2b9 100644 >> --- a/fs/btrfs/volumes.c >> +++ b/fs/btrfs/volumes.c >> @@ -1162,6 +1162,8 @@ static int open_fs_devices(struct >> btrfs_fs_devices *fs_devices, >> fs_devices->opened = 1; >> fs_devices->latest_bdev = latest_dev->bdev; >> fs_devices->total_rw_bytes = 0; >> + /* Set the default readmirror policy */ >> + fs_devices->readmirror = BTRFS_READMIRROR_DEFAULT; >> out: >> return ret; >> } >> @@ -5300,7 +5302,19 @@ static int find_live_mirror(struct >> btrfs_fs_info *fs_info, >> else >> num_stripes = map->num_stripes; >> - preferred_mirror = first + current->pid % num_stripes; >> + switch (fs_info->fs_devices->readmirror) { >> + case BTRFS_READMIRROR_BY_PID: >> + preferred_mirror = first + current->pid % num_stripes; >> + break; >> + default: >> + /* >> + * Shouln't happen, just warn and use by_pid instead of failing. >> + */ >> + btrfs_warn_rl(fs_info, >> + "unknown readmirror type %u, fallback to by_pid", >> + fs_info->fs_devices->readmirror); >> + preferred_mirror = first + current->pid % num_stripes; >> + } >> if (dev_replace_is_ongoing && >> fs_info->dev_replace.cont_reading_from_srcdev_mode == >> diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h >> index 68021d1ee216..f5f091f3c72b 100644 >> --- a/fs/btrfs/volumes.h >> +++ b/fs/btrfs/volumes.h >> @@ -209,6 +209,12 @@ struct btrfs_device { >> BTRFS_DEVICE_GETSET_FUNCS(disk_total_bytes); >> BTRFS_DEVICE_GETSET_FUNCS(bytes_used); >> +/* readmirror_policy types */ >> +#define BTRFS_READMIRROR_DEFAULT BTRFS_READMIRROR_BY_PID >> +enum btrfs_readmirror_policy_type { >> + BTRFS_READMIRROR_BY_PID, >> +}; >> + >> struct btrfs_fs_devices { >> u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */ >> u8 metadata_uuid[BTRFS_FSID_SIZE]; >> @@ -260,6 +266,8 @@ struct btrfs_fs_devices { >> struct kobject *devices_kobj; >> struct kobject *devinfo_kobj; >> struct completion kobj_unregister; >> + >> + u8 readmirror; > > The only valid values for this are the enum, so make this > > enum btrfs_readmirror_policy_type readmirror; Oh. Ok. Thanks, Anand > Thanks, > > Josef
On 3/1/20 3:32 AM, Steven Davies wrote: > On 02/01/2020 10:12, Anand Jain wrote: >> As of now we use %pid method to read stripped mirrored data. So >> application's process id determines the stripe id to be read. This type >> of read IO routing typically helps in a system with many small >> independent applications tying to read random data. On the other hand >> the %pid based read IO distribution policy is inefficient if there is a >> single application trying to read large data and the overall disk >> bandwidth remains under utilized. >> >> So this patch introduces a framework where we could add more readmirror >> policies, such as routing the IO based on device's wait-queue or manual >> when we have a read-preferred device or a policy based on the target >> storage caching. > > I think the idea is good but that it would be cleaner if the tunable was > named read_policy rather than readmirror as it's more obvious that it > contains a policy tunable. Um. 'read_policy' sounds good, but I hope it is clear enough to indicate that we are talking about read for only mirrored-chunks. Will rename to read_policy. > Do you envisage allowing more than one policy to be active for a > filesystem? If not, what about using the same structure as the CPU > frequency and block IO schedulers with the format > > #cat /sys/block/sda/queue/scheduler > noop [deadline] cfq > > Such that btrfs would (eventually) have something like > > #cat /sys/fs/btrfs/UUID/read_policy > by_pid [user_defined_device] by_shortest_queue > And in case of user_defined_device, the device for the read shall be specified in cat /sys/fs/btrfs/UUID/devinfo/<devid>/read_preferred 0 = unset, 1 = set. (devinfo patches are in the ML [1] open for comment) [1] [PATCH v3 4/4] btrfs: sysfs, add devid/dev_state kobject and attribute > And the policy would be changed by echo'ing the new policy name to the > read_policy kobject. I like this approach, will change it to use this format. Thanks, Anand > Steve
On 03/01/2020 10:28, Anand Jain wrote: > On 3/1/20 3:32 AM, Steven Davies wrote: >> On 02/01/2020 10:12, Anand Jain wrote: >>> So this patch introduces a framework where we could add more readmirror >>> policies, such as routing the IO based on device's wait-queue or manual >>> when we have a read-preferred device or a policy based on the target >>> storage caching. >> >> I think the idea is good but that it would be cleaner if the tunable >> was named read_policy rather than readmirror as it's more obvious that >> it contains a policy tunable. > > Um. 'read_policy' sounds good, but I hope it is clear enough to > indicate that we are talking about read for only mirrored-chunks. > Will rename to read_policy. I think it would be obvious that the policy will only apply to mirrored chunks; after all, you can only read a chunk from a device that contains it. >> Do you envisage allowing more than one policy to be active for a >> filesystem? If not, what about using the same structure as the CPU >> frequency and block IO schedulers with the format >> >> #cat /sys/block/sda/queue/scheduler >> noop [deadline] cfq >> >> Such that btrfs would (eventually) have something like >> >> #cat /sys/fs/btrfs/UUID/read_policy >> by_pid [user_defined_device] by_shortest_queue >> > > And in case of user_defined_device, the device for the read shall be > specified in > > cat /sys/fs/btrfs/UUID/devinfo/<devid>/read_preferred > > 0 = unset, 1 = set. > > (devinfo patches are in the ML [1] open for comment) > [1] > [PATCH v3 4/4] btrfs: sysfs, add devid/dev_state kobject and attribute I remember seeing that patch and I think the approach is logical. Steve
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index c95e47aa84f8..e26af766f2b9 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -1162,6 +1162,8 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices, fs_devices->opened = 1; fs_devices->latest_bdev = latest_dev->bdev; fs_devices->total_rw_bytes = 0; + /* Set the default readmirror policy */ + fs_devices->readmirror = BTRFS_READMIRROR_DEFAULT; out: return ret; } @@ -5300,7 +5302,19 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info, else num_stripes = map->num_stripes; - preferred_mirror = first + current->pid % num_stripes; + switch (fs_info->fs_devices->readmirror) { + case BTRFS_READMIRROR_BY_PID: + preferred_mirror = first + current->pid % num_stripes; + break; + default: + /* + * Shouln't happen, just warn and use by_pid instead of failing. + */ + btrfs_warn_rl(fs_info, + "unknown readmirror type %u, fallback to by_pid", + fs_info->fs_devices->readmirror); + preferred_mirror = first + current->pid % num_stripes; + } if (dev_replace_is_ongoing && fs_info->dev_replace.cont_reading_from_srcdev_mode == diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h index 68021d1ee216..f5f091f3c72b 100644 --- a/fs/btrfs/volumes.h +++ b/fs/btrfs/volumes.h @@ -209,6 +209,12 @@ struct btrfs_device { BTRFS_DEVICE_GETSET_FUNCS(disk_total_bytes); BTRFS_DEVICE_GETSET_FUNCS(bytes_used); +/* readmirror_policy types */ +#define BTRFS_READMIRROR_DEFAULT BTRFS_READMIRROR_BY_PID +enum btrfs_readmirror_policy_type { + BTRFS_READMIRROR_BY_PID, +}; + struct btrfs_fs_devices { u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */ u8 metadata_uuid[BTRFS_FSID_SIZE]; @@ -260,6 +266,8 @@ struct btrfs_fs_devices { struct kobject *devices_kobj; struct kobject *devinfo_kobj; struct completion kobj_unregister; + + u8 readmirror; }; #define BTRFS_BIO_INLINE_CSUM_SIZE 64
As of now we use %pid method to read stripped mirrored data. So application's process id determines the stripe id to be read. This type of read IO routing typically helps in a system with many small independent applications tying to read random data. On the other hand the %pid based read IO distribution policy is inefficient if there is a single application trying to read large data and the overall disk bandwidth remains under utilized. So this patch introduces a framework where we could add more readmirror policies, such as routing the IO based on device's wait-queue or manual when we have a read-preferred device or a policy based on the target storage caching. Signed-off-by: Anand Jain <anand.jain@oracle.com> --- v2: Declare fs_devices::readmirror as u8 instead of atomic_t A small change in comment and change log wordings. fs/btrfs/volumes.c | 16 +++++++++++++++- fs/btrfs/volumes.h | 8 ++++++++ 2 files changed, 23 insertions(+), 1 deletion(-)