From patchwork Fri Sep 6 22:57:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E111E6FE49 for ; Fri, 6 Sep 2024 22:57:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A28B86B0098; Fri, 6 Sep 2024 18:57:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93B3A6B009A; Fri, 6 Sep 2024 18:57:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7661D6B0099; Fri, 6 Sep 2024 18:57:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 54B3F6B0096 for ; Fri, 6 Sep 2024 18:57:08 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CF668A134F for ; Fri, 6 Sep 2024 22:57:07 +0000 (UTC) X-FDA: 82535825694.19.710DF5A Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) by imf22.hostedemail.com (Postfix) with ESMTP id D8B51C0007 for ; Fri, 6 Sep 2024 22:57:05 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OWN26DZA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of asml.silence@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=asml.silence@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725663353; a=rsa-sha256; cv=none; b=kBN/euSN1zG14xBNKhVAddQ3WcbT88j7NDMwM80Nvi6pYnYyP9HTK+oUamwuV7pf58m5pl i5yJ+W0j6uBcttx5J2HTsXOcnT6DwWIEEWUGKlxqlr7RByRLoCGxjcvCc2q97aUtfcCCKH 6NpdWbz7an2Fgy8TNA30GIlWt8aUZKg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OWN26DZA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of asml.silence@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=asml.silence@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725663353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v7dFMMDbVFAIV5pvquHCkne/t7NzjwwosACmczIEiUs=; b=5bx/QYMCS4hFTxltncUoJaJpiyQufnnFsVOGK6T2QN+wSN+qhJrzt3RJYpXGHRzmRjNQAE 834Eihp6/63axNOQh+QjIq6chD2bxErbKfDILy6cpCDlDLq677Un9DI1SsbEL6DCYgIEkT 1yuZ0UjIDW7UCXCWO7PZa8BKb7/pi40= Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-5365a9574b6so1532263e87.1 for ; Fri, 06 Sep 2024 15:57:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663424; x=1726268224; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v7dFMMDbVFAIV5pvquHCkne/t7NzjwwosACmczIEiUs=; b=OWN26DZA5yBzndqpXA9Lk7D8sXURH8LfNn7rB2zeE5kJKrmuscz40RDXWBSWWRuJ1C VttvLuHXepNzIPjfeAdUo81WkzEf5h3zoO5aqUoJ/wlSizQp9BlLopHIExUBKt61SIkP +UkRKcuKRcFVK09xaTJ441bL4CAQCEsIzzGQBxAKYKs5vmaF26ZDP4HEzTf74xGAGZuS jHAslcxhhZURj8bxPZQ5hkxgXcR4ieetEbMMiO2ud51khrNyDL6dJmlPtiEcm9LnEEWk poxkS3SWMOXvk52ytCL6XjNtEh/q8BfewyBqb69gKDaXpl9IizJoixB7uoPbDk7yjg9Z KPfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663424; x=1726268224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v7dFMMDbVFAIV5pvquHCkne/t7NzjwwosACmczIEiUs=; b=HIayw2T52Cn3nH85qc1Y9i0u2AgOyR42fL7HXGCWJeiG1mM7H4ARATusg5jCAj39PO 6thSOMgJWUArwd8jHW6chq6vmJlqirH50lcluKEy1POiHSYSqPwpSF0F+l5hzzeR0hab 4pCEfVv5aJWS3OEi9sChuKdHDC8QHfLA5fShq29NjxKq8BOQMs35wgr2kUN5H/OoVEfY PZZbcL4ZhpfmCBtC259mTQxUal9p94gnbp7Z7/RnCjNCuMhwDwK+C0ym8UFz8jMU5jT0 M30h1cDwrqjBX+krR9UoFsmTNoOcQL0Hpgogsz2SEYJJkk7iZ8BURO3Xh1+CDt9Nr6SD vCtw== X-Forwarded-Encrypted: i=1; AJvYcCX9JnkzCDhCXTkXI7vSA0NFt9r7UCiX6Nk8Hrnd7psQiIXl0wI9evtoo29sEFqWR1e1GJMp351cQA==@kvack.org X-Gm-Message-State: AOJu0YxZVjlcMkzKsSoGX9ELmpw5stf8j+vCSV0zXDN7iWzIBZLAyuI6 QXE0FV/Z6XPXua2ThrjYZBiUm89mER40IqNxnbrcpUNTXzgLZrqo X-Google-Smtp-Source: AGHT+IHt+QYHmRu0nquydclberbtpO0LZ0vL2z0xJXuhII8HVhOpdcMo8Pgh0RIOwsY85+0VzuhDqw== X-Received: by 2002:a05:6512:b88:b0:536:53d1:850d with SMTP id 2adb3069b0e04-536587f848emr3490089e87.39.1725663423587; Fri, 06 Sep 2024 15:57:03 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.57.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:57:03 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 5/8] block: implement async discard as io_uring cmd Date: Fri, 6 Sep 2024 23:57:22 +0100 Message-ID: <7fc0a61ae29190a42e958eddfefd6d44cdf372ad.1725621577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: D8B51C0007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: fbakeesbpa6tyju6np1sde71dwnkgf1y X-HE-Tag: 1725663425-563009 X-HE-Meta: U2FsdGVkX195NlmikL/ZGdV40W9ykCq1msAsNkSOV6jUeY2O0qszqD83C7f9FTWj0Q3b9XMlbWi4kmlBdq8xJT2Lu+/IwJCsQIE2P9JTu+31VN8zETdkgzlBetjkH623EPCV/pTEEz7tdhh00LEWfi7cvKmVuTyrGoeBLKG+HWnelOWo5VUgykZetgTTp5iRmeYBGoM/zSj/RQLHuIKbkGV4CVeqHitYq8v3AtxnVpcNwTtqpj/kdBqRIDFCpiNDEztZvZ3CGBv406eBDQkkakBxr71P5p3W2PPSikKcv6ST2sQ9i7xV2T5IJhjFyVJPN1novevOainXRkHevXRJJKYriV4oVt8iSwNrcETNuxWJ7TRYqvZJ2zsKUa6mpaqasuxXDhZlaqTjy1iqrXPaueiwK8dhxjDdLXggagzcsvE+GPgGPheMpziTSkiQe51veyv1CnevHSazKYzGQnxREuT19mZ2/v6xo0FdVRK5UIDB5CbarZwuxIAY2NPFZiAgl0tSN40q1arhiP42+fI6IJvYxpEFF2TfFkZ4G7T9oi4K81RejEPjZUq0wpzxNgYZYczwR03SfW9DyZBf1uQSBIol96p1Hrg3jCC56OrvAUtRhqrL7a9xBh6NbfbUyeVRW7NWFDOLnmByCNOoFHCV4ifi9ifDKia7mWPhX82iIjim9uuOWb2s3XSkcK2HPCXfGBlHizxWKP1GBBCs+1ri/a/lXavDiDT3L4EHXSSnIsTXXIt5Q9rcgobjJ7+Q4SYuRXzgrGgmuZdZ0xgbq5iPzo98CNIAfHrdxSqjs8XnCT6VxAcL84gThz7/4etjBTFAlK2KGNPDp3FSflY+S2tYfyLlDul5Qhdqge8mz2BuPz4Y4gVwhu0MWai2V5+enxLS7nqVGgz9YcnnfIYEVD4pWTqCLj6mPgl6M+o2nYldeqAR9Zm377QkHw2mHWml23ipz8i/jyYUROwTWGASere r3QPCqZD xgzWHBByBQ4zVCgUYP7Gj+p3Fa6/UfFjb4n/7wtPPhS+Im5+gKr+Twye74NITAFUDU/GEWuMdB3JRS4MOGy5BID1YdSwsUsWUYx2CMB/jor61BfL3htOiTxP67yRpDg8kiTOheIKadr1DKR3DdH7pWL3RyGY5laJte+nmMpAXdeplkp01wHHQWbPpP8TB5PLsiiZmGJF08aeCWtnZNqO1bDSNWhqz3h07Ozsn9beVjDwciH8ZfuWnD9gRbHxLbGrbGWN4v1Rkum6uuKitclUEwwxmT0oiyZL8E7A81n1t+6wdPSeECk9u6DFy8Ml60Y9cI2HIj72Qh4Er1KIsa18P6lOOivayemkF6MUQlucPgQce/To5FbI2Qwbpo4g/yUCrtSkTEitaJKv5PYexvxnNFi9vlOHwk640fgYS4/uBkn680uZhv/aCLbBUFg0JbRwh4FnuAyEF/cZXOfpeAwAfYKwBnuSu25xBfJt7JNrQztRIhP+2IijLSvZARbI6imYUfcYS5p5ak7v65EHr30eVABKveRxKUT3+bvq1vm2nNTqnhlM6dKpn9zE4dWVIlR0x8eYu9GE4/+FThWg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: io_uring allows to implement custom file specific operations via fops->uring_cmd callback. Use it to wire up asynchronous discard commands. Normally, first it tries to do a non-blocking issue, and if fails we'd retry from a blocking context by returning -EAGAIN to core io_uring. Note, unlike ioctl(BLKDISCARD) with stronger guarantees against races, we only do a best effort attempt to invalidate page cache, and it can race with any writes and reads and leave page cache stale. It's the same kind of races we allow to direct writes. Suggested-by: Conrad Meyer Signed-off-by: Pavel Begunkov --- block/blk-lib.c | 3 +- block/blk.h | 1 + block/fops.c | 2 + block/ioctl.c | 102 ++++++++++++++++++++++++++++++++++++++++ include/linux/bio.h | 2 + include/uapi/linux/fs.h | 2 + 6 files changed, 111 insertions(+), 1 deletion(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index 83eb7761c2bf..c94c67a75f7e 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -10,7 +10,8 @@ #include "blk.h" -static sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) +/* The maximum size of a discard that can be issued from a given sector. */ +sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) { unsigned int discard_granularity = bdev_discard_granularity(bdev); sector_t granularity_aligned_sector; diff --git a/block/blk.h b/block/blk.h index 32f4e9f630a3..1a1a18d118f7 100644 --- a/block/blk.h +++ b/block/blk.h @@ -605,6 +605,7 @@ blk_mode_t file_to_blk_mode(struct file *file); int truncate_bdev_range(struct block_device *bdev, blk_mode_t mode, loff_t lstart, loff_t lend); long blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); +int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags); long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); extern const struct address_space_operations def_blk_aops; diff --git a/block/fops.c b/block/fops.c index 9825c1713a49..8154b10b5abf 100644 --- a/block/fops.c +++ b/block/fops.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "blk.h" static inline struct inode *bdev_file_inode(struct file *file) @@ -873,6 +874,7 @@ const struct file_operations def_blk_fops = { .splice_read = filemap_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .uring_cmd = blkdev_uring_cmd, .fop_flags = FOP_BUFFER_RASYNC, }; diff --git a/block/ioctl.c b/block/ioctl.c index a820f692dd1c..19fba8332eee 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -11,6 +11,8 @@ #include #include #include +#include +#include #include "blk.h" static int blkpg_do_ioctl(struct block_device *bdev, @@ -742,3 +744,103 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) return ret; } #endif + +struct blk_iou_cmd { + int res; + bool nowait; +}; + +static void blk_cmd_complete(struct io_uring_cmd *cmd, unsigned int issue_flags) +{ + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + + if (bic->res == -EAGAIN && bic->nowait) + io_uring_cmd_issue_blocking(cmd); + else + io_uring_cmd_done(cmd, bic->res, 0, issue_flags); +} + +static void bio_cmd_bio_end_io(struct bio *bio) +{ + struct io_uring_cmd *cmd = bio->bi_private; + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + + if (unlikely(bio->bi_status) && !bic->res) + bic->res = blk_status_to_errno(bio->bi_status); + + io_uring_cmd_do_in_task_lazy(cmd, blk_cmd_complete); + bio_put(bio); +} + +static int blkdev_cmd_discard(struct io_uring_cmd *cmd, + struct block_device *bdev, + uint64_t start, uint64_t len, bool nowait) +{ + gfp_t gfp = nowait ? GFP_NOWAIT : GFP_KERNEL; + sector_t sector = start >> SECTOR_SHIFT; + sector_t nr_sects = len >> SECTOR_SHIFT; + struct bio *prev = NULL, *bio; + int err; + + if (!bdev_max_discard_sectors(bdev)) + return -EOPNOTSUPP; + + if (!(file_to_blk_mode(cmd->file) & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; + + /* + * Don't allow multi-bio non-blocking submissions as subsequent bios + * may fail but we won't get a direct indication of that. Normally, + * the caller should retry from a blocking context. + */ + if (nowait && nr_sects > bio_discard_limit(bdev, sector)) + return -EAGAIN; + + err = filemap_invalidate_pages(bdev->bd_mapping, start, + start + len - 1, nowait); + if (err) + return err; + + while ((bio = blk_alloc_discard_bio(bdev, §or, &nr_sects, gfp))) { + if (nowait) + bio->bi_opf |= REQ_NOWAIT; + prev = bio_chain_and_submit(prev, bio); + } + if (!prev) + return -EAGAIN; + + prev->bi_private = cmd; + prev->bi_end_io = bio_cmd_bio_end_io; + submit_bio(prev); + return -EIOCBQUEUED; +} + +int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) +{ + struct block_device *bdev = I_BDEV(cmd->file->f_mapping->host); + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + const struct io_uring_sqe *sqe = cmd->sqe; + u32 cmd_op = cmd->cmd_op; + uint64_t start, len; + + if (unlikely(sqe->ioprio || sqe->__pad1 || sqe->len || + sqe->rw_flags || sqe->file_index)) + return -EINVAL; + + bic->res = 0; + bic->nowait = issue_flags & IO_URING_F_NONBLOCK; + + start = READ_ONCE(sqe->addr); + len = READ_ONCE(sqe->addr3); + + switch (cmd_op) { + case BLOCK_URING_CMD_DISCARD: + return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); + } + return -EINVAL; +} diff --git a/include/linux/bio.h b/include/linux/bio.h index faceadb040f9..78ead424484c 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -684,4 +684,6 @@ struct bio *bio_chain_and_submit(struct bio *prev, struct bio *new); struct bio *blk_alloc_discard_bio(struct block_device *bdev, sector_t *sector, sector_t *nr_sects, gfp_t gfp_mask); +sector_t bio_discard_limit(struct block_device *bdev, sector_t sector); + #endif /* __LINUX_BIO_H */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..7ea41ca97158 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -208,6 +208,8 @@ struct fsxattr { * (see uapi/linux/blkzoned.h) */ +#define BLOCK_URING_CMD_DISCARD _IO(0x12,137) + #define BMAP_IOCTL 1 /* obsolete - kept for compatibility */ #define FIBMAP _IO(0x00,1) /* bmap access */ #define FIGETBSZ _IO(0x00,2) /* get the block size used for bmap */