| Message ID | 20240209160204.1471421-1-sergei.shtepa@linux.dev (mailing list archive) |
|---|---|
| Headers | show |
| Series | filtering and snapshots of a block devices | expand |
Il 09/02/2024 17:01, Sergei Shtepa ha scritto: > Hi all. > > I am happy to offer an improved version of the block device filtering > mechanism (blkfilter) and module for creating snapshots of block devices > (blksnap). > > The filtering block device mechanism is implemented in the block layer. > This allows to attach and detach block device filters. Filters extend the > functionality of the block layer. See more in > Documentation/block/blkfilter.rst. > > The main purpose of snapshots of block devices is to provide backups of > them. See more in Documentation/block/blksnap.rst. The tool, library and > tests for working with blksnap can be found on github. > Link: https://github.com/veeam/blksnap/tree/stable-v2.0 > There is also documentation from which you can learn how to manage the > module using the library and the console tool. > > Based on LK v6.8-rc3 with Christoph's patchset "clean up blk_mq_submit_bio". > Link: https://lore.kernel.org/linux-block/50fbe76b-d77d-4a7e-bda4-3a3b754fbd7e@kernel.org/T/#t > > I express my appreciation and gratitude to Christoph. Thanks to his > attention to the project, it was possible to raise the quality of the code. > I probably wouldn't have made version 7 if it wasn't for his help. > I am sure that the blksnap module will improve the quality of backup tools > for Linux. > > v7 changes: > - The location of the filtering of I/O units has been changed. This made it > possible to remove the additional call bio_queue_enter(). > - Remove configs BLKSNAP_DIFF_BLKDEV and BLKSNAP_CHUNK_DIFF_BIO_SYNC. > - To process the ioctl, the switch statement is used instead of a table > with functions. > - Instead of a file descriptor, the module gets a path on the file system. > This allows the kernel module to correctly open a file or block device > with exclusive access rights. > - Fixed a bio leaking bugs. Hi, looking https://lore.kernel.org/all/20240209160204.1471421-1-sergei.shtepa@linux.dev/ I don't see any response to this latest version of blksnap even though a lot of time has passed, and this seems strange to me. From what I see recipients were less to avoid being blocked by antispam and double shipments like previous versions, and there was an error in subjects from patch 3/8 to patch 8/8, where I think it should be "blksnap:" or "block/blksnap:" instead of "block:" at the beginning of the subjects. However, the error in the subjects does not seem serious to me to avoid possible replies, or am I wrong? Isn't there anyone who wants to comment or review? I want to thank Sergei Shtepa for all his work and others people that helped with previous versions review and advices, especially Christoph Hellwig who helped so much, and I hope the project can proceed. > > v6 changes: > - The difference storage has been changed. > In the previous version, the file was created only to reserve sector > ranges on a block device. The data was stored directly to the block > device in these sector ranges. Now saving and reading data is done using > 'VFS' using vfs_iter_write() and vfs_iter_read() functions. This allows > not to depend on the filesystem and use, for example, tmpfs. Using an > unnamed temporary file allows hiding it from other processes and > automatically release it when the snapshot is closed. > However, now the module does not allow adding a block device to the > snapshot on which the difference storage is located. There is no way to > ensure the immutability of file metadata when writing data to a file. > This means that the metadata of the filesystem may change, which may > cause damage to the snapshot. > - _IOW and _IOR were mixed up - fixed. > - Protection against the use of the snapshots for block devices with > hardware inline encryption and data integrity was implemented. > Compatibility with them was not planned and has not been tested at the > moment. > > v5 changes: > - Rebase for "kernel/git/axboe/linux-block.git" branch "for-6.5/block". > Link: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-6.5/block > > v4 changes: > - Structures for describing the state of chunks are allocated dynamically. > This reduces memory consumption, since the struct chunk is allocated only > for those blocks for which the snapshot image state differs from the > original block device. > - The algorithm for calculating the chunk size depending on the size of the > block device has been changed. For large block devices, it is now > possible to allocate a larger number of chunks, and their size is smaller. > - For block devices, a 'filter' file has been added to /sys/block/<device>. > It displays the name of the filter that is attached to the block device. > - Fixed a problem with the lack of protection against re-adding a block > device to a snapshot. > - Fixed a bug in the algorithm of allocating the next bio for a chunk. > This problem was occurred on large disks, for which a chunk consists of > at least two bio. > - The ownership mechanism of the diff_area structure has been changed. > This fixed the error of prematurely releasing the diff_area structure > when destroying the snapshot. > - Documentation corrected. > - The Sparse analyzer is passed. > - Use __u64 type instead pointers in UAPI. > > v3 changes: > - New block device I/O controls BLKFILTER_ATTACH and BLKFILTER_DETACH allow > to attach and detach filters. > - New block device I/O control BLKFILTER_CTL allow sending command to > attached block device filter. > - The copy-on-write algorithm for processing I/O units has been optimized > and has become asynchronous. > - The snapshot image reading algorithm has been optimized and has become > asynchronous. > - Optimized the finite state machine for processing chunks. > - Fixed a tracking block size calculation bug. > > v2 changes: > - Added documentation for Block Device Filtering Mechanism. > - Added documentation for Block Devices Snapshots Module (blksnap). > - The MAINTAINERS file has been updated. > - Optimized queue code for snapshot images. > - Fixed comments, log messages and code for better readability. > > v1 changes: > - Forgotten "static" declarations have been added. > - The text of the comments has been corrected. > - It is possible to connect only one filter, since there are no others in > upstream. > - Do not have additional locks for attach/detach filter. > - blksnap.h moved to include/uapi/. > - #pragma once and commented code removed. > - uuid_t removed from user API. > - Removed default values for module parameters from the configuration file. > - The debugging code for tracking memory leaks has been removed. > - Simplified Makefile. > - Optimized work with large memory buffers, CBT tables are now in virtual > memory. > - The allocation code of minor numbers has been optimized. > - The implementation of the snapshot image block device has been > simplified, now it is a bio-based block device. > - Removed initialization of global variables with null values. > - only one bio is used to copy one chunk. > - Checked on ppc64le. > > Sergei Shtepa (8): > documentation: filtering and snapshots of a block devices > block: filtering of a block devices > block: header file of the blksnap module interface > block: module management interface functions > block: handling and tracking I/O units > block: difference storage implementation > block: snapshot and snapshot image block device > block: Kconfig, Makefile and MAINTAINERS files > > Documentation/block/blkfilter.rst | 66 ++ > Documentation/block/blksnap.rst | 351 ++++++++++ > Documentation/block/index.rst | 2 + > .../userspace-api/ioctl/ioctl-number.rst | 1 + > MAINTAINERS | 17 + > block/Makefile | 3 +- > block/bdev.c | 2 + > block/blk-core.c | 26 +- > block/blk-filter.c | 257 +++++++ > block/blk-mq.c | 7 +- > block/blk-mq.h | 2 +- > block/blk.h | 11 + > block/genhd.c | 10 + > block/ioctl.c | 7 + > block/partitions/core.c | 9 + > drivers/block/Kconfig | 2 + > drivers/block/Makefile | 2 + > drivers/block/blksnap/Kconfig | 12 + > drivers/block/blksnap/Makefile | 15 + > drivers/block/blksnap/cbt_map.c | 225 +++++++ > drivers/block/blksnap/cbt_map.h | 90 +++ > drivers/block/blksnap/chunk.c | 631 ++++++++++++++++++ > drivers/block/blksnap/chunk.h | 134 ++++ > drivers/block/blksnap/diff_area.c | 577 ++++++++++++++++ > drivers/block/blksnap/diff_area.h | 175 +++++ > drivers/block/blksnap/diff_buffer.c | 114 ++++ > drivers/block/blksnap/diff_buffer.h | 37 + > drivers/block/blksnap/diff_storage.c | 290 ++++++++ > drivers/block/blksnap/diff_storage.h | 103 +++ > drivers/block/blksnap/event_queue.c | 81 +++ > drivers/block/blksnap/event_queue.h | 64 ++ > drivers/block/blksnap/main.c | 481 +++++++++++++ > drivers/block/blksnap/params.h | 16 + > drivers/block/blksnap/snapimage.c | 135 ++++ > drivers/block/blksnap/snapimage.h | 10 + > drivers/block/blksnap/snapshot.c | 462 +++++++++++++ > drivers/block/blksnap/snapshot.h | 65 ++ > drivers/block/blksnap/tracker.c | 369 ++++++++++ > drivers/block/blksnap/tracker.h | 78 +++ > include/linux/blk-filter.h | 72 ++ > include/linux/blk_types.h | 1 + > include/linux/sched.h | 1 + > include/uapi/linux/blk-filter.h | 35 + > include/uapi/linux/blksnap.h | 384 +++++++++++ > include/uapi/linux/fs.h | 3 + > 45 files changed, 5430 insertions(+), 5 deletions(-) > create mode 100644 Documentation/block/blkfilter.rst > create mode 100644 Documentation/block/blksnap.rst > create mode 100644 block/blk-filter.c > create mode 100644 drivers/block/blksnap/Kconfig > create mode 100644 drivers/block/blksnap/Makefile > create mode 100644 drivers/block/blksnap/cbt_map.c > create mode 100644 drivers/block/blksnap/cbt_map.h > create mode 100644 drivers/block/blksnap/chunk.c > create mode 100644 drivers/block/blksnap/chunk.h > create mode 100644 drivers/block/blksnap/diff_area.c > create mode 100644 drivers/block/blksnap/diff_area.h > create mode 100644 drivers/block/blksnap/diff_buffer.c > create mode 100644 drivers/block/blksnap/diff_buffer.h > create mode 100644 drivers/block/blksnap/diff_storage.c > create mode 100644 drivers/block/blksnap/diff_storage.h > create mode 100644 drivers/block/blksnap/event_queue.c > create mode 100644 drivers/block/blksnap/event_queue.h > create mode 100644 drivers/block/blksnap/main.c > create mode 100644 drivers/block/blksnap/params.h > create mode 100644 drivers/block/blksnap/snapimage.c > create mode 100644 drivers/block/blksnap/snapimage.h > create mode 100644 drivers/block/blksnap/snapshot.c > create mode 100644 drivers/block/blksnap/snapshot.h > create mode 100644 drivers/block/blksnap/tracker.c > create mode 100644 drivers/block/blksnap/tracker.h > create mode 100644 include/linux/blk-filter.h > create mode 100644 include/uapi/linux/blk-filter.h > create mode 100644 include/uapi/linux/blksnap.h >