[v2,1/3] btrfs: add readmirror type framework
diff mbox series

Message ID 1577959968-19427-2-git-send-email-anand.jain@oracle.com
State New
Headers show
Series
  • readmirror feature (sysfs and in-memory only approach)
Related show

Commit Message

Anand Jain Jan. 2, 2020, 10:12 a.m. UTC
As of now we use %pid method to read stripped mirrored data. So
application's process id determines the stripe id to be read. This type
of read IO routing typically helps in a system with many small
independent applications tying to read random data. On the other hand
the %pid based read IO distribution policy is inefficient if there is a
single application trying to read large data and the overall disk
bandwidth remains under utilized.

So this patch introduces a framework where we could add more readmirror
policies, such as routing the IO based on device's wait-queue or manual
when we have a read-preferred device or a policy based on the target
storage caching.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
v2: Declare fs_devices::readmirror as u8 instead of atomic_t
    A small change in comment and change log wordings.

 fs/btrfs/volumes.c | 16 +++++++++++++++-
 fs/btrfs/volumes.h |  8 ++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)

Comments

Josef Bacik Jan. 2, 2020, 4:24 p.m. UTC | #1
On 1/2/20 5:12 AM, Anand Jain wrote:
> As of now we use %pid method to read stripped mirrored data. So
> application's process id determines the stripe id to be read. This type
> of read IO routing typically helps in a system with many small
> independent applications tying to read random data. On the other hand
> the %pid based read IO distribution policy is inefficient if there is a
> single application trying to read large data and the overall disk
> bandwidth remains under utilized.
> 
> So this patch introduces a framework where we could add more readmirror
> policies, such as routing the IO based on device's wait-queue or manual
> when we have a read-preferred device or a policy based on the target
> storage caching.
> 
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> ---
> v2: Declare fs_devices::readmirror as u8 instead of atomic_t
>      A small change in comment and change log wordings.
> 
>   fs/btrfs/volumes.c | 16 +++++++++++++++-
>   fs/btrfs/volumes.h |  8 ++++++++
>   2 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index c95e47aa84f8..e26af766f2b9 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -1162,6 +1162,8 @@ static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
>   	fs_devices->opened = 1;
>   	fs_devices->latest_bdev = latest_dev->bdev;
>   	fs_devices->total_rw_bytes = 0;
> +	/* Set the default readmirror policy */
> +	fs_devices->readmirror = BTRFS_READMIRROR_DEFAULT;
>   out:
>   	return ret;
>   }
> @@ -5300,7 +5302,19 @@ static int find_live_mirror(struct btrfs_fs_info *fs_info,
>   	else
>   		num_stripes = map->num_stripes;
>   
> -	preferred_mirror = first + current->pid % num_stripes;
> +	switch (fs_info->fs_devices->readmirror) {
> +	case BTRFS_READMIRROR_BY_PID:
> +		preferred_mirror = first + current->pid % num_stripes;
> +		break;
> +	default:
> +		/*
> +		 * Shouln't happen, just warn and use by_pid instead of failing.
> +		 */
> +		btrfs_warn_rl(fs_info,
> +			      "unknown readmirror type %u, fallback to by_pid",
> +			      fs_info->fs_devices->readmirror);
> +		preferred_mirror = first + current->pid % num_stripes;
> +	}
>   
>   	if (dev_replace_is_ongoing &&
>   	    fs_info->dev_replace.cont_reading_from_srcdev_mode ==
> diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
> index 68021d1ee216..f5f091f3c72b 100644
> --- a/fs/btrfs/volumes.h
> +++ b/fs/btrfs/volumes.h
> @@ -209,6 +209,12 @@ struct btrfs_device {
>   BTRFS_DEVICE_GETSET_FUNCS(disk_total_bytes);
>   BTRFS_DEVICE_GETSET_FUNCS(bytes_used);
>   
> +/* readmirror_policy types */
> +#define BTRFS_READMIRROR_DEFAULT	BTRFS_READMIRROR_BY_PID
> +enum btrfs_readmirror_policy_type {
> +	BTRFS_READMIRROR_BY_PID,
> +};
> +
>   struct btrfs_fs_devices {
>   	u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */
>   	u8 metadata_uuid[BTRFS_FSID_SIZE];
> @@ -260,6 +266,8 @@ struct btrfs_fs_devices {
>   	struct kobject *devices_kobj;
>   	struct kobject *devinfo_kobj;
>   	struct completion kobj_unregister;
> +
> +	u8 readmirror;

The only valid values for this are the enum, so make this

enum btrfs_readmirror_policy_type readmirror;

Thanks,

Josef
Steven Davies Jan. 2, 2020, 7:32 p.m. UTC | #2
On 02/01/2020 10:12, Anand Jain wrote:
> As of now we use %pid method to read stripped mirrored data. So
> application's process id determines the stripe id to be read. This type
> of read IO routing typically helps in a system with many small
> independent applications tying to read random data. On the other hand
> the %pid based read IO distribution policy is inefficient if there is a
> single application trying to read large data and the overall disk
> bandwidth remains under utilized.
> 
> So this patch introduces a framework where we could add more readmirror
> policies, such as routing the IO based on device's wait-queue or manual
> when we have a read-preferred device or a policy based on the target
> storage caching.

I think the idea is good but that it would be cleaner if the tunable was 
named read_policy rather than readmirror as it's more obvious that it 
contains a policy tunable.

Do you envisage allowing more than one policy to be active for a 
filesystem? If not, what about using the same structure as the CPU 
frequency and block IO schedulers with the format

#cat /sys/block/sda/queue/scheduler
noop [deadline] cfq

Such that btrfs would (eventually) have something like

#cat /sys/fs/btrfs/UUID/read_policy
by_pid [user_defined_device] by_shortest_queue

And the policy would be changed by echo'ing the new policy name to the 
read_policy kobject.

Steve
Anand Jain Jan. 3, 2020, 9:57 a.m. UTC | #3
On 3/1/20 12:24 AM, Josef Bacik wrote:
> On 1/2/20 5:12 AM, Anand Jain wrote:
>> As of now we use %pid method to read stripped mirrored data. So
>> application's process id determines the stripe id to be read. This type
>> of read IO routing typically helps in a system with many small
>> independent applications tying to read random data. On the other hand
>> the %pid based read IO distribution policy is inefficient if there is a
>> single application trying to read large data and the overall disk
>> bandwidth remains under utilized.
>>
>> So this patch introduces a framework where we could add more readmirror
>> policies, such as routing the IO based on device's wait-queue or manual
>> when we have a read-preferred device or a policy based on the target
>> storage caching.
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>> ---
>> v2: Declare fs_devices::readmirror as u8 instead of atomic_t
>>      A small change in comment and change log wordings.
>>
>>   fs/btrfs/volumes.c | 16 +++++++++++++++-
>>   fs/btrfs/volumes.h |  8 ++++++++
>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
>> index c95e47aa84f8..e26af766f2b9 100644
>> --- a/fs/btrfs/volumes.c
>> +++ b/fs/btrfs/volumes.c
>> @@ -1162,6 +1162,8 @@ static int open_fs_devices(struct 
>> btrfs_fs_devices *fs_devices,
>>       fs_devices->opened = 1;
>>       fs_devices->latest_bdev = latest_dev->bdev;
>>       fs_devices->total_rw_bytes = 0;
>> +    /* Set the default readmirror policy */
>> +    fs_devices->readmirror = BTRFS_READMIRROR_DEFAULT;
>>   out:
>>       return ret;
>>   }
>> @@ -5300,7 +5302,19 @@ static int find_live_mirror(struct 
>> btrfs_fs_info *fs_info,
>>       else
>>           num_stripes = map->num_stripes;
>> -    preferred_mirror = first + current->pid % num_stripes;
>> +    switch (fs_info->fs_devices->readmirror) {
>> +    case BTRFS_READMIRROR_BY_PID:
>> +        preferred_mirror = first + current->pid % num_stripes;
>> +        break;
>> +    default:
>> +        /*
>> +         * Shouln't happen, just warn and use by_pid instead of failing.
>> +         */
>> +        btrfs_warn_rl(fs_info,
>> +                  "unknown readmirror type %u, fallback to by_pid",
>> +                  fs_info->fs_devices->readmirror);
>> +        preferred_mirror = first + current->pid % num_stripes;
>> +    }
>>       if (dev_replace_is_ongoing &&
>>           fs_info->dev_replace.cont_reading_from_srcdev_mode ==
>> diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
>> index 68021d1ee216..f5f091f3c72b 100644
>> --- a/fs/btrfs/volumes.h
>> +++ b/fs/btrfs/volumes.h
>> @@ -209,6 +209,12 @@ struct btrfs_device {
>>   BTRFS_DEVICE_GETSET_FUNCS(disk_total_bytes);
>>   BTRFS_DEVICE_GETSET_FUNCS(bytes_used);
>> +/* readmirror_policy types */
>> +#define BTRFS_READMIRROR_DEFAULT    BTRFS_READMIRROR_BY_PID
>> +enum btrfs_readmirror_policy_type {
>> +    BTRFS_READMIRROR_BY_PID,
>> +};
>> +
>>   struct btrfs_fs_devices {
>>       u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */
>>       u8 metadata_uuid[BTRFS_FSID_SIZE];
>> @@ -260,6 +266,8 @@ struct btrfs_fs_devices {
>>       struct kobject *devices_kobj;
>>       struct kobject *devinfo_kobj;
>>       struct completion kobj_unregister;
>> +
>> +    u8 readmirror;
> 
> The only valid values for this are the enum, so make this
> 
> enum btrfs_readmirror_policy_type readmirror;

  Oh. Ok.

Thanks, Anand


> Thanks,
> 
> Josef
Anand Jain Jan. 3, 2020, 10:28 a.m. UTC | #4
On 3/1/20 3:32 AM, Steven Davies wrote:
> On 02/01/2020 10:12, Anand Jain wrote:
>> As of now we use %pid method to read stripped mirrored data. So
>> application's process id determines the stripe id to be read. This type
>> of read IO routing typically helps in a system with many small
>> independent applications tying to read random data. On the other hand
>> the %pid based read IO distribution policy is inefficient if there is a
>> single application trying to read large data and the overall disk
>> bandwidth remains under utilized.
>>
>> So this patch introduces a framework where we could add more readmirror
>> policies, such as routing the IO based on device's wait-queue or manual
>> when we have a read-preferred device or a policy based on the target
>> storage caching.
> 
> I think the idea is good but that it would be cleaner if the tunable was 
> named read_policy rather than readmirror as it's more obvious that it 
> contains a policy tunable.

  Um. 'read_policy' sounds good, but I hope it is clear enough to
  indicate that we are talking about read for only mirrored-chunks.
  Will rename to read_policy.

> Do you envisage allowing more than one policy to be active for a 
> filesystem? If not, what about using the same structure as the CPU 
> frequency and block IO schedulers with the format
> 
> #cat /sys/block/sda/queue/scheduler
> noop [deadline] cfq
> 
> Such that btrfs would (eventually) have something like
> 
> #cat /sys/fs/btrfs/UUID/read_policy
> by_pid [user_defined_device] by_shortest_queue
> 

  And in case of user_defined_device, the device for the read shall be
  specified in

   cat /sys/fs/btrfs/UUID/devinfo/<devid>/read_preferred

   0 = unset, 1 = set.

   (devinfo patches are in the ML [1] open for comment)
   [1]
   [PATCH v3 4/4] btrfs: sysfs, add devid/dev_state kobject and attribute

> And the policy would be changed by echo'ing the new policy name to the 
> read_policy kobject.

  I like this approach, will change it to use this format.

Thanks, Anand

> Steve
Steven Davies Jan. 3, 2020, 2:51 p.m. UTC | #5
On 03/01/2020 10:28, Anand Jain wrote:
> On 3/1/20 3:32 AM, Steven Davies wrote:
>> On 02/01/2020 10:12, Anand Jain wrote:

>>> So this patch introduces a framework where we could add more readmirror
>>> policies, such as routing the IO based on device's wait-queue or manual
>>> when we have a read-preferred device or a policy based on the target
>>> storage caching.
>>
>> I think the idea is good but that it would be cleaner if the tunable 
>> was named read_policy rather than readmirror as it's more obvious that 
>> it contains a policy tunable.
> 
>   Um. 'read_policy' sounds good, but I hope it is clear enough to
>   indicate that we are talking about read for only mirrored-chunks.
>   Will rename to read_policy.

I think it would be obvious that the policy will only apply to mirrored 
chunks; after all, you can only read a chunk from a device that contains it.

>> Do you envisage allowing more than one policy to be active for a 
>> filesystem? If not, what about using the same structure as the CPU 
>> frequency and block IO schedulers with the format
>>
>> #cat /sys/block/sda/queue/scheduler
>> noop [deadline] cfq
>>
>> Such that btrfs would (eventually) have something like
>>
>> #cat /sys/fs/btrfs/UUID/read_policy
>> by_pid [user_defined_device] by_shortest_queue
>>
> 
>   And in case of user_defined_device, the device for the read shall be
>   specified in
> 
>    cat /sys/fs/btrfs/UUID/devinfo/<devid>/read_preferred
> 
>    0 = unset, 1 = set.
> 
>    (devinfo patches are in the ML [1] open for comment)
>    [1]
>    [PATCH v3 4/4] btrfs: sysfs, add devid/dev_state kobject and attribute

I remember seeing that patch and I think the approach is logical.

Steve

Patch
diff mbox series

diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index c95e47aa84f8..e26af766f2b9 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -1162,6 +1162,8 @@  static int open_fs_devices(struct btrfs_fs_devices *fs_devices,
 	fs_devices->opened = 1;
 	fs_devices->latest_bdev = latest_dev->bdev;
 	fs_devices->total_rw_bytes = 0;
+	/* Set the default readmirror policy */
+	fs_devices->readmirror = BTRFS_READMIRROR_DEFAULT;
 out:
 	return ret;
 }
@@ -5300,7 +5302,19 @@  static int find_live_mirror(struct btrfs_fs_info *fs_info,
 	else
 		num_stripes = map->num_stripes;
 
-	preferred_mirror = first + current->pid % num_stripes;
+	switch (fs_info->fs_devices->readmirror) {
+	case BTRFS_READMIRROR_BY_PID:
+		preferred_mirror = first + current->pid % num_stripes;
+		break;
+	default:
+		/*
+		 * Shouln't happen, just warn and use by_pid instead of failing.
+		 */
+		btrfs_warn_rl(fs_info,
+			      "unknown readmirror type %u, fallback to by_pid",
+			      fs_info->fs_devices->readmirror);
+		preferred_mirror = first + current->pid % num_stripes;
+	}
 
 	if (dev_replace_is_ongoing &&
 	    fs_info->dev_replace.cont_reading_from_srcdev_mode ==
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 68021d1ee216..f5f091f3c72b 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -209,6 +209,12 @@  struct btrfs_device {
 BTRFS_DEVICE_GETSET_FUNCS(disk_total_bytes);
 BTRFS_DEVICE_GETSET_FUNCS(bytes_used);
 
+/* readmirror_policy types */
+#define BTRFS_READMIRROR_DEFAULT	BTRFS_READMIRROR_BY_PID
+enum btrfs_readmirror_policy_type {
+	BTRFS_READMIRROR_BY_PID,
+};
+
 struct btrfs_fs_devices {
 	u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */
 	u8 metadata_uuid[BTRFS_FSID_SIZE];
@@ -260,6 +266,8 @@  struct btrfs_fs_devices {
 	struct kobject *devices_kobj;
 	struct kobject *devinfo_kobj;
 	struct completion kobj_unregister;
+
+	u8 readmirror;
 };
 
 #define BTRFS_BIO_INLINE_CSUM_SIZE	64