[9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror
diff mbox

Message ID 20170417032642.30770-10-quwenruo@cn.fujitsu.com
State New
Headers show

Commit Message

Qu Wenruo April 17, 2017, 3:26 a.m. UTC
Introduce a new option, --pattern, to specify the pattern to fill
mirror.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 Documentation/btrfs-modify.asciidoc |  5 +++++
 modify/mirror.c                     | 11 ++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)

Comments

Lakshmipathi.G April 18, 2017, 7:05 a.m. UTC | #1
Nice. With this new option, it should be possible to achieve something
like RAID56 corruption test-script say,
https://patchwork.kernel.org/patch/9583455/  right?
----
Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org


On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> Introduce a new option, --pattern, to specify the pattern to fill
> mirror.
>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
>  Documentation/btrfs-modify.asciidoc |  5 +++++
>  modify/mirror.c                     | 11 ++++++++++-
>  2 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc
> index ae2ada65..d82fb0df 100644
> --- a/Documentation/btrfs-modify.asciidoc
> +++ b/Documentation/btrfs-modify.asciidoc
> @@ -41,6 +41,11 @@ stripe number 1.
>  P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
>  Q for RAID6 2nd parity stripe.
>
> +--pattern <pattern>::::
> +Specify to pattern to modify the mirror data5f536652.
> ++
> +Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic "_BHRfS_M").
> +
>  EXIT STATUS
>  -----------
>  *btrfs-modify* returns a zero exit status if all its operations succeed.
> diff --git a/modify/mirror.c b/modify/mirror.c
> index c89927f6..dd717fbe 100644
> --- a/modify/mirror.c
> +++ b/modify/mirror.c
> @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
>         char *device;
>         u64 length = (u64)-1;
>         u64 logical = (u64)-1;
> +       u32 pattern = 0x66524842;
>         int stripe = STRIPE_UNINITILIZED;
>         int ret;
>
>         while (1) {
>                 int c;
>                 enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
> -                       GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
> +                       GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET,
> +                       GETOPT_VAL_PATTERN };
>                 static const struct option long_options[] = {
>                         { "logical", required_argument, NULL,
>                                 GETOPT_VAL_LOGICAL },
> @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
>                         { "stripe", required_argument, NULL, GETOPT_VAL_STRIPE },
>                         { "root-ino-offset", required_argument, NULL,
>                                 GETOPT_VAL_ROOT_INO_OFFSET},
> +                       { "pattern", required_argument, NULL,
> +                               GETOPT_VAL_PATTERN},
>                         { NULL, 0, NULL, 0 }
>                 };
>
> @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
>                 case GETOPT_VAL_ROOT_INO_OFFSET:
>                         parse_root_ino_offset(&dest, optarg);
>                         break;
> +               case GETOPT_VAL_PATTERN:
> +                       pattern = arg_strtou32(optarg);
> +                       break;
>                 case '?':
>                 case 'h':
>                         usage(modify_mirror_usage);
> @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
>                 stripe = 0;
>         }
>
> +       memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern));
> +
>         fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
>         if (!fs_info) {
>                 error("failed to open btrfs on device %s\n", device);
> --
> 2.12.2
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lakshmipathi.G April 23, 2017, 7:42 a.m. UTC | #2
Thanks for the example and details. I understood some and need to
re-read couple of more times to understand the remaining.

btw, I created a corruption framework(with previous org), the sample
usage and example is below. It looks similar to Btrfs corruption tool.
thanks.

--
corrupt.py --help
Usage: corrupt.py [options] <arg1> <arg2> <arg3> ...]

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -t object, --object=object
                        The object type to be corrupted [ choose from (inode,

l_inner,l_master,l_leaf,fileinode_inner,fileinode_leaf,dirinode_in
                        ner,dirinode_leaf,data_block,fec_block,baddr)
default: inode]
  -p /fs/path, --path=/fs/path
                        The file or directory path in /fs
  -q optype, --optype=optype
                        The corruption operation on the object [choose from
                        (set,xor,add) default: set]
  -o offset, --offset=offset
                        The valid offset value of the object [default:0]
  -m mirror, --mirror=mirror
                        The mirror number [default: 1]
  -s size, --size=size  The offset field/byte to be corrupted [default: 1]
  -c fixcrc, --fixcrc=fixcrc
                        This fixes the CRC's of the corrupted objects
  -n lbn, --lbn=lbn     The lbn of the data block to be corrupted
  -b baddr, --baddr=baddr
                        The baddr to be corrupted
  --log=LOG_FILE        This logs the result into the log file

Example usage:
corrupt.py --object=inode --offset=0 --size=4 --path=/fs/path/to/file.txt
Will corrupt first 4 bytes of inode structure.

corrupt.py --object=dirinode_inner --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode structure.

corrupt.py --object=dirinode_leaf --offset=0 --size=4
--path=/fs/path/to/file.txt
Will corrupt first 4 bytes of directory-inode leaf structure.
(file.txt should be
large enough to contain leafs).

corrupt.py --object=inode --offset=0 --size=4
--path=/fs/path/to/file.txt  --mirror=2
Will corrupt first 4 bytes of inode structure that reside on Node/drive-2.

--optype=optype ==> default is set. It sets '0' to specified offset.
add => Will try given value to existing value. For example.
offset=(nlink's offset) --optype=add will increase the nlink count by
1.
(running fsck, expected to find this and fix it)

where as optype xor => should flip the bits at given offset.
--

----
Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org

On Thu, Apr 20, 2017 at 6:30 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
>
> At 04/19/2017 03:46 PM, Lakshmipathi.G wrote:
>>
>> Thanks. This should make easier to create corruption test scripts.
>>
>> when using --root-ino-offset 5,257,0  Does final entry 0 has to be default value ? Does it represent any value at
>> this moment?
>
>
> For case 5,257,0, it means the start point of the corruption is root 5, inode 257 and file offset 0.
>
> So let's take the following file layout as an example (a little complex one, to show off all the ability)
>
> Root 5 inode 257:
>
>        \/ This is file offset
> Extent 0:   disk bytenr X, disk len 4K, nr_bytes 4K offset 0  plain
> Extent 4K:  disk bytenr Y, disk len 8K, nr_bytes 4K offset 0  plain
> Extent 8K:  disk bytenr Z, disk len 8K, nr_bytes 4K offset 4K plain
>
> And if you specified --root-ino-offset 5,257,0 and --length 12288
>
> Then the following on-disk range will be corrupted:
> [X, X+4K)
> [Y, Y+4K)
> [Z+4K, Z+8K)
>
> Making all the 0~12K of root5, inode257 get corrupted, while unrelated ranges like [Z, Z+4K) and [Y+4K, Y+8K) not modified.
>
> While for RAID1/RAID10/DUP usage, just create a file on a new fs, then pass --root-ino-offset 5,257,0 with --length <file_size> will corrupt the whole file for you, allowing us to test recovery.
>
>>
>> One more thing, does btrfs-modify command has option to print out existing parity value (all parity of a file or specific a parity)?
>> something like get_parity option?
>
>
> 1) For print parity value
> I didn't see the need to print parity value.
> As if you want to verify if the parity is good or not, just use offline scrub.
> It will detect parity/data corruption.
>
> 2) For printing/corrupting value of a file in RAID56 profile
> Not as easy as one may think.
>
> Think about the following case:
>
> Root 5 ino 257:
> Extent 0: disk bytenr X(rounded to stripe boundary), disk len 32K, nr_bytes 32K, offset 0.
> Extent 1: disk bytenr X+32K, disk len 64K, nr_bytes 64K, offset 0
>
> On disk it will be:
>
> |    Dev  1       |       Dev   2    |       Dev 3      |
> X                 X+64K              X+128K             X+192K
> |<-Ext 0->|<----- Ext 1---->|        |///// Parity /////|
>
> For Extent 0, it's OK to just print out/corrupt its value, as it's completely inside one data stripe.
>
> But for Extent 1, it crosses 2 device.
> If we corrupt on disk data of Extent1, we will corrupt both data stripe, leaving the full stripe unrecoverable.
>
> So for P/Q corrupting, I'm considering to record which full stripe we have corrupted and never corrupt any full stripe already damaged in next version.
>
> Thanks,
> Qu
>>
>>
>>
>>
>> ----
>> Cheers,
>> Lakshmipathi.G
>> FOSS Programmer.
>> http://www.giis.co.in http://www.webminal.org
>>
>> On Tue, Apr 18, 2017 at 12:43 PM, Qu Wenruo <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>> wrote:
>>
>>
>>
>>     At 04/18/2017 03:05 PM, Lakshmipathi.G wrote:
>>
>>         Nice. With this new option, it should be possible to achieve
>>         something
>>         like RAID56 corruption test-script say,
>>         https://patchwork.kernel.org/patch/9583455/
>>         <https://patchwork.kernel.org/patch/9583455/>  right?
>>
>>
>>     Yes, just for your use case!
>>
>>     It would be much more simpler to craft such recovery test case.
>>
>>     Just create a file in default subvolume (5), write large enough
>>     data, get the file ino (257 for first inode).
>>
>>     Then pass --root-ino-offset 5,257,0 --length 64k --stripe 0 to
>>     corrupt data stripe.
>>     Or change to --stripe P to corrupt parity.
>>
>>     Although it's still a little tricky to use current "--stripe" option
>>     to only corrupt data stripe (above 64K may crosses 2 data stripes).
>>     It shouldn't to be hard to add a new parameter --raid-stripe for
>>     such usage.
>>
>>     But it should be good enough for DUP/RAID1/RAID10.
>>
>>     Thanks,
>>     Qu
>>
>>
>>         ----
>>         Cheers,
>>         Lakshmipathi.G
>>         FOSS Programmer.
>>         http://www.giis.co.in http://www.webminal.org
>>
>>
>>         On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo
>>         <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>> wrote:
>>
>>             Introduce a new option, --pattern, to specify the pattern to
>>             fill
>>             mirror.
>>
>>             Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com
>>             <mailto:quwenruo@cn.fujitsu.com>>
>>
>>             ---
>>                Documentation/btrfs-modify.asciidoc |  5 +++++
>>                modify/mirror.c                     | 11 ++++++++++-
>>                2 files changed, 15 insertions(+), 1 deletion(-)
>>
>>             diff --git a/Documentation/btrfs-modify.asciidoc
>>             b/Documentation/btrfs-modify.asciidoc
>>             index ae2ada65..d82fb0df 100644
>>             --- a/Documentation/btrfs-modify.asciidoc
>>             +++ b/Documentation/btrfs-modify.asciidoc
>>             @@ -41,6 +41,11 @@ stripe number 1.
>>                P for RAID56 1st parity stripe. Only RAID5 and RAID6
>>             support stripe number P.
>>                Q for RAID6 2nd parity stripe.
>>
>>             +--pattern <pattern>::::
>>             +Specify to pattern to modify the mirror data5f536652.
>>             ++
>>             +Default value is "0x66524842" (ascii "BHRf", part of the
>>             btrfs magic "_BHRfS_M").
>>             +
>>                EXIT STATUS
>>                -----------
>>                *btrfs-modify* returns a zero exit status if all its
>>             operations succeed.
>>             diff --git a/modify/mirror.c b/modify/mirror.c
>>             index c89927f6..dd717fbe 100644
>>             --- a/modify/mirror.c
>>             +++ b/modify/mirror.c
>>             @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
>>                       char *device;
>>                       u64 length = (u64)-1;
>>                       u64 logical = (u64)-1;
>>             +       u32 pattern = 0x66524842;
>>                       int stripe = STRIPE_UNINITILIZED;
>>                       int ret;
>>
>>                       while (1) {
>>                               int c;
>>                               enum { GETOPT_VAL_LOGICAL = 257,
>>             GETOPT_VAL_LENGTH,
>>             -                       GETOPT_VAL_STRIPE,
>>             GETOPT_VAL_ROOT_INO_OFFSET };
>>             +                       GETOPT_VAL_STRIPE,
>>             GETOPT_VAL_ROOT_INO_OFFSET,
>>             +                       GETOPT_VAL_PATTERN };
>>                               static const struct option long_options[] = {
>>                                       { "logical", required_argument, NULL,
>>                                               GETOPT_VAL_LOGICAL },
>>             @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
>>                                       { "stripe", required_argument,
>>             NULL, GETOPT_VAL_STRIPE },
>>                                       { "root-ino-offset",
>>             required_argument, NULL,
>>                                               GETOPT_VAL_ROOT_INO_OFFSET},
>>             +                       { "pattern", required_argument, NULL,
>>             +                               GETOPT_VAL_PATTERN},
>>                                       { NULL, 0, NULL, 0 }
>>                               };
>>
>>             @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
>>                               case GETOPT_VAL_ROOT_INO_OFFSET:
>>                                       parse_root_ino_offset(&dest, optarg);
>>                                       break;
>>             +               case GETOPT_VAL_PATTERN:
>>             +                       pattern = arg_strtou32(optarg);
>>             +                       break;
>>                               case '?':
>>                               case 'h':
>>                                       usage(modify_mirror_usage);
>>             @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
>>                               stripe = 0;
>>                       }
>>
>>             +       memset(write_buf, pattern, sizeof(write_buf) /
>>             sizeof(pattern));
>>             +
>>                       fs_info = open_ctree_fs_info(device, 0, 0, 0,
>>             OPEN_CTREE_WRITES);
>>                       if (!fs_info) {
>>                               error("failed to open btrfs on device
>>             %s\n", device);
>>             --
>>             2.12.2
>>
>>
>>
>>             --
>>             To unsubscribe from this list: send the line "unsubscribe
>>             linux-btrfs" in
>>             the body of a message to majordomo@vger.kernel.org
>>             <mailto:majordomo@vger.kernel.org>
>>             More majordomo info at
>>             http://vger.kernel.org/majordomo-info.html
>>             <http://vger.kernel.org/majordomo-info.html>
>>
>>         --
>>         To unsubscribe from this list: send the line "unsubscribe
>>         linux-btrfs" in
>>         the body of a message to majordomo@vger.kernel.org
>>         <mailto:majordomo@vger.kernel.org>
>>         More majordomo info at
>>         http://vger.kernel.org/majordomo-info.html
>>         <http://vger.kernel.org/majordomo-info.html>
>>
>>
>>
>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Sterba May 4, 2017, 4:53 p.m. UTC | #3
On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
> Thanks for the example and details. I understood some and need to
> re-read couple of more times to understand the remaining.
> 
> btw, I created a corruption framework(with previous org), the sample
> usage and example is below. It looks similar to Btrfs corruption tool.
> thanks.
> 
> --
> corrupt.py --help
[...]

Interesting, can you please share the script? This is another
alternative that seems more plausible for rapid prototyping of various
corruption scenarios. The C utility (either existing btrfs-corrupt-block
or the proposed btrfs-modify) can become tedious to change, but can be
compiled and distributed without the python dependency.

I wanted to use something python-based for tests when Hans announced the
python-btrfs project, but it has broader goals than just the testsuite
needs.  So we could have our own corrupt.py, just for our internal use.

I'm not sure if a compiled tool like btrfs-modify is really needed, but
why we can't have both.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lakshmipathi.G May 6, 2017, 3:31 p.m. UTC | #4
okay, but I'm not sure whether it can be useful here other than the
sample cli usage. The reason, this tool relies heavily on Python/C
layer[1]. The framework works like this:

C-api(fs-progs) <--> Python/C api <--> Python program.

C-api is typical fs library (like libext2fs/libbtrfs) and C/Python
used  to export every on-disk fs structure as an python-object along
with few fs-library functions. Finally in python script we can perform
stuffs like:

--
import fslib as fs

disk_object=fs.get_inode("/file/path")
#set few values finally make a call
retval = fs.corrupt_object(disk_object, offset, size, optype, fixcrc)
---

at C-python layer above python-object converted into C appropriate
data-type and passed on to C-layer.

hard-part will be troubleshooting issues, tracking them is difficult
because both gdb and pdb can help only upto certain point.

[1]: https://docs.python.org/2/extending/extending.html

Cheers.
Lakshmipathi.G


On 5/5/17, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> At 05/05/2017 12:53 AM, David Sterba wrote:
>> On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
>>> Thanks for the example and details. I understood some and need to
>>> re-read couple of more times to understand the remaining.
>>>
>>> btw, I created a corruption framework(with previous org), the sample
>>> usage and example is below. It looks similar to Btrfs corruption tool.
>>> thanks.
>>>
>>> --
>>> corrupt.py --help
>> [...]
>>
>> Interesting, can you please share the script? This is another
>> alternative that seems more plausible for rapid prototyping of various
>> corruption scenarios. The C utility (either existing btrfs-corrupt-block
>> or the proposed btrfs-modify) can become tedious to change, but can be
>> compiled and distributed without the python dependency.
>>
>> I wanted to use something python-based for tests when Hans announced the
>> python-btrfs project, but it has broader goals than just the testsuite
>> needs.  So we could have our own corrupt.py, just for our internal use.
>>
>> I'm not sure if a compiled tool like btrfs-modify is really needed, but
>> why we can't have both.
>>
>>
> Python based tool is always a good idea.
>
> If python based lib can provide the same capability of current C
> facilities (btrfs_search_slot() at least) and have better encapsulation,
> I'm totally committed to python based one.
>
> Although I'm a little afraid that the python script is not using the low
> level btrfs_search_slot(), but just getting chunk layout then do
> mathematics to get stripe location.
>
> Thanks,
> Qu
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch
diff mbox

diff --git a/Documentation/btrfs-modify.asciidoc b/Documentation/btrfs-modify.asciidoc
index ae2ada65..d82fb0df 100644
--- a/Documentation/btrfs-modify.asciidoc
+++ b/Documentation/btrfs-modify.asciidoc
@@ -41,6 +41,11 @@  stripe number 1.
 P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
 Q for RAID6 2nd parity stripe.
 
+--pattern <pattern>::::
+Specify to pattern to modify the mirror data5f536652.
++
+Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic "_BHRfS_M").
+
 EXIT STATUS
 -----------
 *btrfs-modify* returns a zero exit status if all its operations succeed.
diff --git a/modify/mirror.c b/modify/mirror.c
index c89927f6..dd717fbe 100644
--- a/modify/mirror.c
+++ b/modify/mirror.c
@@ -373,13 +373,15 @@  int modify_mirror(int argc, char **argv)
 	char *device;
 	u64 length = (u64)-1;
 	u64 logical = (u64)-1;
+	u32 pattern = 0x66524842;
 	int stripe = STRIPE_UNINITILIZED;
 	int ret;
 
 	while (1) {
 		int c;
 		enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
-			GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
+			GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET,
+			GETOPT_VAL_PATTERN };
 		static const struct option long_options[] = {
 			{ "logical", required_argument, NULL,
 				GETOPT_VAL_LOGICAL },
@@ -388,6 +390,8 @@  int modify_mirror(int argc, char **argv)
 			{ "stripe", required_argument, NULL, GETOPT_VAL_STRIPE },
 			{ "root-ino-offset", required_argument, NULL,
 				GETOPT_VAL_ROOT_INO_OFFSET},
+			{ "pattern", required_argument, NULL,
+				GETOPT_VAL_PATTERN},
 			{ NULL, 0, NULL, 0 }
 		};
 
@@ -407,6 +411,9 @@  int modify_mirror(int argc, char **argv)
 		case GETOPT_VAL_ROOT_INO_OFFSET:
 			parse_root_ino_offset(&dest, optarg);
 			break;
+		case GETOPT_VAL_PATTERN:
+			pattern = arg_strtou32(optarg);
+			break;
 		case '?':
 		case 'h':
 			usage(modify_mirror_usage);
@@ -439,6 +446,8 @@  int modify_mirror(int argc, char **argv)
 		stripe = 0;
 	}
 
+	memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern));
+
 	fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
 	if (!fs_info) {
 		error("failed to open btrfs on device %s\n", device);