From patchwork Fri Sep 6 22:57:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794849 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B739B15B0F2; Fri, 6 Sep 2024 22:56:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663419; cv=none; b=hNzKIkaRhUQx8tBdZYqU56vvh2fCACvNkgjvPyZOogicq5Q0ytrpPSIC1MijlisguVwoM9CBTOsh88WkfZosJjvkE7YtGAaVbtj+DT2hm39inD79sJ5h6cVKVuIOwZ90Qeiij38E9wMzXjj52SmBBKZ20gREictEiGxdctVtQ+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663419; c=relaxed/simple; bh=2vI0bIvl9+D8oswmBaa6Rx1hzhaGczPhBrLqovoOrl0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O03UiZUSTEuFNn7Zv29VUJNoS81fNMM3wykHCYmsKPl2NbJSx0fOj9BKqbFBWEanJ2jH8bZ5JPlfPa+vWsXOAmAr4+pmZv+iqv+m0D799TCFucuBK0xq0ebaMM3thzt4gbqxAK2fBWIfR+A3q1gKm4CiT8Sq2scaj6h/wVW+5M4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SwR8lEJ5; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SwR8lEJ5" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a86910caf9cso656010866b.1; Fri, 06 Sep 2024 15:56:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663415; x=1726268215; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KZ8pkxfeOzRC4kA801xaO2cK98vlGCE9d9luTt+J2Kg=; b=SwR8lEJ5ondmHg/1e9TsKsQFdULJuXaDKO/CJ1PX6Aa+RnORvkZqOwG7iYcXRs+7XM mtLxrm/mMz5b2lsucTjTGnMVbUCP1ixBvumAB8lBvZfqf1OyTk4MNQKdsABelDMhYlAB LDn/ariGe1ewP1jSa8NF+TeGgvdItFIIUx95MObdLBME2lIoLnI0xBDUUvEdWSm1xRHQ 8RODGEThzPXSTlNdMAO/e7GxkR5UyVxPPimeC1jH26hds6Wf/2Hj8D4J7HE5LNvNyAOe Zd81IevGzueFq79TfHzms3sxIfr8rZtyngGMrA7r1bdeqHU7he848wYw3FgK4Md1XB2f Mg7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663415; x=1726268215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KZ8pkxfeOzRC4kA801xaO2cK98vlGCE9d9luTt+J2Kg=; b=NeBJokr4yywidkmVUG6GOp7+/9ruUs7AoVhPppGaNn+m22bD8H1tXVt+ovf10CQ8EC rUN28Z+PHJbUW7ZciRdxtxCDwSnMgdLJA0dZLm2DtHqzZIzdNZEP7uV5dxKnoiHNOc5w 4RV52mRu1HYefv5iz/IWz4oTq31dOvz9rPMp1MbpjW4FbnBXh9W+D5OhJffizmdJnSur 0VGLYvTJHsoRCX/hJwUV3NiedWZiHC94/8O/rINuyltL90Sl3VVtDBdwBtXWclaiul7l O6gkUxZvCYc9uEmPklBgBwLCsVeK7pV63GBj0TXJQGybTQ1vEhQtJgrpPWhZHXnS6lqu 1uAA== X-Forwarded-Encrypted: i=1; AJvYcCV/9iraSgTK1hMx23znzXVCxXFm0gRHJ/hn8GoGi8ay6H3YO8fVB4tv7FouaJo81VUGnhA/vuIt4x5C8w==@vger.kernel.org X-Gm-Message-State: AOJu0Yx8Dc8rzJ+ny5zFjRq7Qxq+gRx5pnl9sE9H8gEAHnVDlO0eMjqo lKnjCctQa0nZ/QwgxdMqMGO+2TVhrQpIk1ekDMr3ibjRM5uaRQhWZ32ISPSb X-Google-Smtp-Source: AGHT+IHyXvN1ue7B3ODRoTm6YInUy1vTPF5mt5XLGTBYTGrDKLCROG5nKHfbHCNql8pi+pUWW0OV0g== X-Received: by 2002:a17:907:2d2b:b0:a8b:58e3:ac1f with SMTP id a640c23a62f3a-a8b595ddc9bmr313555066b.12.1725663415255; Fri, 06 Sep 2024 15:56:55 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.56.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:56:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 1/8] io_uring/cmd: expose iowq to cmds Date: Fri, 6 Sep 2024 23:57:18 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When an io_uring request needs blocking context we offload it to the io_uring's thread pool called io-wq. We can get there off ->uring_cmd by returning -EAGAIN, but there is no straightforward way of doing that from an asynchronous callback. Add a helper that would transfer a command to a blocking context. Note, we do an extra hop via task_work before io_queue_iowq(), that's a limitation of io_uring infra we have that can likely be lifted later if that would ever become a problem. Signed-off-by: Pavel Begunkov --- include/linux/io_uring/cmd.h | 6 ++++++ io_uring/io_uring.c | 11 +++++++++++ io_uring/io_uring.h | 1 + io_uring/uring_cmd.c | 7 +++++++ 4 files changed, 25 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 447fbfd32215..86ceb3383e49 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -48,6 +48,9 @@ void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags); +/* Execute the request from a blocking context */ +void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd); + #else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) @@ -67,6 +70,9 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags) { } +static inline void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd) +{ +} #endif /* diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 1aca501efaf6..86cf31902841 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -533,6 +533,17 @@ static void io_queue_iowq(struct io_kiocb *req) io_queue_linked_timeout(link); } +static void io_req_queue_iowq_tw(struct io_kiocb *req, struct io_tw_state *ts) +{ + io_queue_iowq(req); +} + +void io_req_queue_iowq(struct io_kiocb *req) +{ + req->io_task_work.func = io_req_queue_iowq_tw; + io_req_task_work_add(req); +} + static __cold void io_queue_deferred(struct io_ring_ctx *ctx) { while (!list_empty(&ctx->defer_list)) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 65078e641390..9d70b2cf7b1e 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -94,6 +94,7 @@ int io_uring_alloc_task_context(struct task_struct *task, int io_ring_add_registered_file(struct io_uring_task *tctx, struct file *file, int start, int end); +void io_req_queue_iowq(struct io_kiocb *req); int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts); int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr); diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 8391c7c7c1ec..39c3c816ec78 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -277,6 +277,13 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, } EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed); +void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd) +{ + struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); + + io_req_queue_iowq(req); +} + static inline int io_uring_cmd_getsockopt(struct socket *sock, struct io_uring_cmd *cmd, unsigned int issue_flags) From patchwork Fri Sep 6 22:57:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794850 Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A61B15B0F2; Fri, 6 Sep 2024 22:56:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663421; cv=none; b=lRMs5w/dr8aD0RxadU6Nt7AFgvrM7WCebWzmuLEgMF+/3ysKPnyn6Ly3NKeG7yLT42ANubMc3V7N7gI29bTwnnG+JuM86dtKdaMgWiWxfLw3Y0PjLiAahFoUP6IQQz001L5eCpQwJjQ3vsyY6zGj4pbawtLdjJL2dd85o6rbIXs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663421; c=relaxed/simple; bh=xgZK/jw8ZlQXj8cIxR0W7ybGVnlcP1lJNS+9Q/l9a0s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FsJDqgB9nUXxpXTWynxhKmUA6tv/8A5wepdf4eEFUQ+J6wU/BMVbTLooqa+BuW7SpG5IcoWtsNdeWZjS9u1V70QJlLGT3mzv3qGrtTNNjc6uCvrWNN+jp+ggknD/0kjXDoDt/kNY+FeOR17hscpEydEn1GdbN/IGCPiKNdaIqnc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eZnYdF5d; arc=none smtp.client-ip=209.85.208.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eZnYdF5d" Received: by mail-lj1-f175.google.com with SMTP id 38308e7fff4ca-2f4f2868621so28672331fa.0; Fri, 06 Sep 2024 15:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663418; x=1726268218; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yyZRXxiAI12SAMIbkwT8bn9l7S8e/TKEaLOeKBVDL1c=; b=eZnYdF5ddEmpuMKLoK+QH0VpgqshABqPTCaD5pt9rBQxBCd7QqekXcXHY4dAeS5FhT b18YnkUQvzZwPO+uvzVybr1ErE4z8hrSIVFZQ2AAbhehC6/D6X3QwwtsXB0bBjt+Dtr4 Z/OtJystuUTLXRRPhocoISHQ0c2uTYb8d9sSmLqeHpRrtoGi3LKlfTwhXlucoc0Gvxo5 dd9+9wexEKgd/WNpJQvLoMEhcvZHpylixUHpwT11iArm+VvZiHx85DexsWgm/WXyhWl1 f72lyLbYaXx2+rR6WGCLSEpmmYd1ar1gt8FNH48PYgblK01XY+uauP+sN9dn2pLels3F xx0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663418; x=1726268218; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yyZRXxiAI12SAMIbkwT8bn9l7S8e/TKEaLOeKBVDL1c=; b=qfk9PyCie5RHvHd16UHWjNWnQnbu57fVmAjLTpJ33LRMSGXPmL60mnVGpYbNGju23/ FlYKCZ/ffCNFUOFXtyBad01MMSPn6VWIFWMnGHihcvvGBiDEkn1gygBzbG3Jq9yyeEcZ 2BLXz/NAdMU0rF00fJqnOz7BNW0DuXHVPeBU/YHgzztWMYgaImaR8oVZoM9MpuzFSgJz Vq0WKqTgy/BtWMtl+lii4NvPZ+r4xBZ4bRLv1buv5wdKlMzEpMCecw8A8iFldp6Xgh4y DKYsL5uZhitVh6KFThjZA63jNudUA7Q+omFP0BBEfJd9aEx7+s9WRaiv41WAKPYk9EWl hHFA== X-Forwarded-Encrypted: i=1; AJvYcCUNvOeE9rRWGwbz+XLjWgxxZVoNVf5BeZuv3AXYVgRkwv8UiNhovoW0eVGRMNLX6XOFwdobJ1UjqFeQHA==@vger.kernel.org X-Gm-Message-State: AOJu0YymCq9q05oH5u9mPCBzib+B6qWptTVPhnLE8GAjc/VLnBNhNDFd peHkNY9Z3p++P5wZE5pzwrQwMmdiiCgfhGTa9u9dKpR5CyXnm847oPNbOvQ4 X-Google-Smtp-Source: AGHT+IHXnXfbkaF7u3VX1p6hb15jHuT8Sz6pGMKxR+NPrjTVAbIbg44Z7wW3wwKJC6QGwAd3IjyH4Q== X-Received: by 2002:a05:6512:e88:b0:52e:7542:f471 with SMTP id 2adb3069b0e04-536587b8ec7mr3170548e87.29.1725663416985; Fri, 06 Sep 2024 15:56:56 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.56.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:56:56 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 2/8] io_uring/cmd: give inline space in request to cmds Date: Fri, 6 Sep 2024 23:57:19 +0100 Message-ID: <7ca779a61ee5e166e535d70df9c7f07b15d8a0ce.1725621577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Some io_uring commands can use some inline space in io_kiocb. We have 32 bytes in struct io_uring_cmd, expose it. Signed-off-by: Pavel Begunkov --- include/linux/io_uring/cmd.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 86ceb3383e49..c189d36ad55e 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -23,6 +23,15 @@ static inline const void *io_uring_sqe_cmd(const struct io_uring_sqe *sqe) return sqe->cmd; } +static inline void io_uring_cmd_private_sz_check(size_t cmd_sz) +{ + BUILD_BUG_ON(cmd_sz > sizeof_field(struct io_uring_cmd, pdu)); +} +#define io_uring_cmd_to_pdu(cmd, pdu_type) ( \ + io_uring_cmd_private_sz_check(sizeof(pdu_type)), \ + ((pdu_type *)&(cmd)->pdu) \ +) + #if defined(CONFIG_IO_URING) int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); From patchwork Fri Sep 6 22:57:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794851 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D258A15B0E5; Fri, 6 Sep 2024 22:57:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663423; cv=none; b=eNlLvY8C5GGOvfCaRN+3YWYy50u5Mf2pvAnFZRQrEsineY1k5ikdxibo09no2HA7FjYWsHckcQBHP7qYNraYNjJXibR+6+yAak4E3pEZQAVsPaScRzGzug0+toikdrab5eWe00FBYR2URjq8MSzsatqduFPc0g+eDoinjoO4Eww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663423; c=relaxed/simple; bh=JrAOgMM2iAIABQmCU3ZV4QTcl0u9rm5SIWFUspGZWXA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z1KNswRmmUDPi1pz37w3twHRwj6Prq4E2rh9d2MKXNZ1L4e3mTyGAV54HwXrP4hEiR84Lny1udZBVnck+z/MV5FUh8eqoAvVoKYdW7GP45xZ+eSJk/8AH9v5xB2+MU86Jn/7xNPqY5Aqhjqytm1tXgZpP1Ss17S5z2vzWlqYmr4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Eh5RCxPc; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eh5RCxPc" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a8692bbec79so346671966b.3; Fri, 06 Sep 2024 15:57:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663420; x=1726268220; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2XjW/1z5M+e5L1HlSBo37JIC76JYeYPBxc+HZavMul8=; b=Eh5RCxPcAWOksrt+up8VqwN5sKhrFP9opupmWubBNT/yNN/IeXw4XB+6sW2uwMeu7k UiR2jcEHYmps69zK3EY7RrUO5UX8pvr4jJB5MUcvdY9BWpHzNCYs1jKn/Noi9vJygVIT 65QANLR1nwyaMOlk6REOYtrvIMiYTzBEQCevzVIYAdKZuDvqMScxV6yqTHe57O0Hkqjj MqODOfdJyAnPUt7mFOtE4RqImdQMwO2v4cj9abJtiT1SbPrzF96AiMkxjQAlBCKY2YCM 4V49hlgwfuwds4TMVP0c4m64MRli46JKkkqh0woLREDigqAvDHVVjZfBpGNvAZbn9Hcy 88zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663420; x=1726268220; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2XjW/1z5M+e5L1HlSBo37JIC76JYeYPBxc+HZavMul8=; b=qjskSgRgH0SQA0bFQnaap4uVsb3uSK95tlE3/pHnrwTykyRCl5HVyn1xIII8YJDtni JBNAwrjXnTMyVYIzzg4TOoPDlaLbPIDrNYfQAn8QCe4SabRfjcmhULWcEyzIc5vWo2P4 +UHqgn2rCbR73a9X5BGHzYcZPqoJpa+yUd1wm//xIMWoRe5MqCDawtruUlnlLcUf0G/X qh1NRmzE4RGMSuSbQXTQtG2/qbW+ReZp/SXtusCg/kz/tTadv7j3jIGMgiZZPK7s+e0G bj4c3fEtDkL2u65vbtdbxApGSmNUeUcC/6y96GU/rwHWJON8bjXox/lMbM6/smv2JAIK xCRw== X-Forwarded-Encrypted: i=1; AJvYcCUXm1TCKXFJDxLMdjx6QuFZNMq3kRYUea5H9ZtUafboO8ZMJnnXJLP3b0WgJzaxMND5aDJgIlPcTqB87g==@vger.kernel.org X-Gm-Message-State: AOJu0YxI4QKPpyUuc8m455B1l08yX67U/68c457momy5qyVeriLWXe1B MwUESiLNq+1xUqUil9gzYfGQQvOdIbRE4nhJ7Z26mbUjpk2hOqYde7P2E9Zd X-Google-Smtp-Source: AGHT+IElFIAZCS4JhFrJt5ylxD7009i496r/jtdVHC3NBh65yo7l8qVQPQsC4GCBaH1S1XbN+0jHTA== X-Received: by 2002:a17:906:f59f:b0:a86:b32f:eee6 with SMTP id a640c23a62f3a-a8a88858b21mr338191766b.54.1725663419302; Fri, 06 Sep 2024 15:56:59 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.56.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:56:58 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 3/8] filemap: introduce filemap_invalidate_pages Date: Fri, 6 Sep 2024 23:57:20 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 kiocb_invalidate_pages() is useful for the write path, however not everything is backed by kiocb and we want to reuse the function for bio based discard implementation. Extract and and reuse a new helper called filemap_invalidate_pages(), which takes a argument indicating whether it should be non-blocking and might return -EAGAIN. Signed-off-by: Pavel Begunkov --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d9c7edb6422b..e39c3a7ce33c 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -32,6 +32,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, pgoff_t start, pgoff_t end); int kiocb_invalidate_pages(struct kiocb *iocb, size_t count); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); +int filemap_invalidate_pages(struct address_space *mapping, + loff_t pos, loff_t end, bool nowait); int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/filemap.c b/mm/filemap.c index d62150418b91..6843ed4847d4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2712,14 +2712,12 @@ int kiocb_write_and_wait(struct kiocb *iocb, size_t count) } EXPORT_SYMBOL_GPL(kiocb_write_and_wait); -int kiocb_invalidate_pages(struct kiocb *iocb, size_t count) +int filemap_invalidate_pages(struct address_space *mapping, + loff_t pos, loff_t end, bool nowait) { - struct address_space *mapping = iocb->ki_filp->f_mapping; - loff_t pos = iocb->ki_pos; - loff_t end = pos + count - 1; int ret; - if (iocb->ki_flags & IOCB_NOWAIT) { + if (nowait) { /* we could block if there are any pages in the range */ if (filemap_range_has_page(mapping, pos, end)) return -EAGAIN; @@ -2738,6 +2736,15 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count) return invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT, end >> PAGE_SHIFT); } + +int kiocb_invalidate_pages(struct kiocb *iocb, size_t count) +{ + struct address_space *mapping = iocb->ki_filp->f_mapping; + + return filemap_invalidate_pages(mapping, iocb->ki_pos, + iocb->ki_pos + count - 1, + iocb->ki_flags & IOCB_NOWAIT); +} EXPORT_SYMBOL_GPL(kiocb_invalidate_pages); /** From patchwork Fri Sep 6 22:57:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794852 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 045DA15B15D; Fri, 6 Sep 2024 22:57:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663425; cv=none; b=Y+Bz6AgAzeC6RIrurQov9pvVn4ezSXugtmZQltPnBIarnHObV/ZiKMKZRhjpXPsO9GDo35RQv/PPi4IqQUwysVFlB3y6xqxIhxrUGNu2wVXdjkSWwvOm72BzuSJTHF1REgZNmhNl1MkqjT+MUszkpVRTk40FqYjgEQ20PsKJyp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663425; c=relaxed/simple; bh=hjZDHvkTsBuwkkkqnazOc/370IgWi6PLEmiOKefcDVg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=szyA+e5f1brehlQ6vcZyIxb8gDRKSOvJw33llhDTuV2FIE+LA2RvBx+1RBtQY51CQXihCyu7MPXiSR6giolDnDTluGs4LH66nTF+jcDD0PAn2j8o7G8BIZfk1+waEIwYADKIoThiVLQZfELE6AMf0yWs7go5/RN/6Qw6c+KJoRo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ie/dnWpC; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ie/dnWpC" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a8682bb5e79so362391366b.2; Fri, 06 Sep 2024 15:57:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663422; x=1726268222; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gCQ1sFR6q4N5GvvlFRQ23FRxGveAlTZ5FLsIiX1fOqY=; b=Ie/dnWpCgiuoK5/tddvQ7LKGAgsbLtoCf6mgGbtcT71iHHnQGVQp1WtIRTygI0fC3Z oegirP1HlNV4Z+WLqLkA4ohe2/C41t+iJGlgE1egfiuVIDeVrp2JPkYDkOt+kNBUo7qy aaZjuqUdcg1ikjcayU/Tr5vRh9u72V4lL03VWfJaATZkB5h1rNAfA9MNG8R2q61Orbno DVP2pFWbKCt8ltD7tlWhEA4nbEtNFUl+wrrHi0N339SRDrJIOwjikjHfeP/fR/a7/zIA k5z7Q7jyHOoF7qZzBovlCjElHxcG2GiJjyFjdRyXsjE4R4zjc4lUCskd4MZuB1FF4JnZ FJAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663422; x=1726268222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gCQ1sFR6q4N5GvvlFRQ23FRxGveAlTZ5FLsIiX1fOqY=; b=A5P0DdcT+PrnWau9T/GDtKwb0bSbAa+YjW3dQh0e91VCaNnS8SiUhUQ5o3CVEm/M7P Sbv39hEmZwJtgCweCEHLbiACT7z+EM7A4G7EROIDLiNm/yQpx5jodg3FOTwA7bFAQ0gj 8ZptpSqwfacdYJNyogfkxajh6MsUWBE3D8XlJXJtq1ivzQ0lsSL6ii+JgLmp7g3vMLtn wcLYTBp9x3fvOqPn9KI2bBzcIAl9odnmSwh+PA5TTuZwvyqkB5tGrUEn8iq9eMyu1Hxi Co8oXRi6HNJdh1vof5r6JeiIzSGiJFue+FRs05m6xI0AqItylb8g2lcftWNbIdFDMEX+ 0Apw== X-Forwarded-Encrypted: i=1; AJvYcCVLt8fpM6Q4iArNmLvAvtJ4NHE+uo/S1LLp8sWHCigGQM+xuqPUOnohnoESBfF5fAXNjqLL2Cpob/x5sQ==@vger.kernel.org X-Gm-Message-State: AOJu0YyIZ0nQTqItd9M96xpOucAMFwvtU7G9tLRQK9CX4cXPBGmsEG9N aApxlWbYKKD64IleGgNTs1PjUloJ5ZPGGV7nFxOLra8iPmiwfJHB8I5OqlvY X-Google-Smtp-Source: AGHT+IGJ6gzQkF2S9wNkzcDzctZY5nEoDtVuCYWlHsBqBW5skC54eWB3Kt2NmCi50PZKCSauBvcXGg== X-Received: by 2002:a17:907:7b92:b0:a7a:9a78:4b5a with SMTP id a640c23a62f3a-a8a887fcdb3mr350202066b.52.1725663421631; Fri, 06 Sep 2024 15:57:01 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.56.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:57:01 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 4/8] block: introduce blk_validate_byte_range() Date: Fri, 6 Sep 2024 23:57:21 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to further changes extract a helper function out of blk_ioctl_discard() that validates if we can do IO against the given range of disk byte addresses. Signed-off-by: Pavel Begunkov --- block/ioctl.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index e8e4a4190f18..a820f692dd1c 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -92,38 +92,46 @@ static int compat_blkpg_ioctl(struct block_device *bdev, } #endif +static int blk_validate_byte_range(struct block_device *bdev, + uint64_t start, uint64_t len) +{ + unsigned int bs_mask = bdev_logical_block_size(bdev) - 1; + uint64_t end; + + if ((start | len) & bs_mask) + return -EINVAL; + if (!len) + return -EINVAL; + if (check_add_overflow(start, len, &end) || end > bdev_nr_bytes(bdev)) + return -EINVAL; + + return 0; +} + static int blk_ioctl_discard(struct block_device *bdev, blk_mode_t mode, unsigned long arg) { - unsigned int bs_mask = bdev_logical_block_size(bdev) - 1; - uint64_t range[2], start, len, end; + uint64_t range[2], start, len; struct bio *prev = NULL, *bio; sector_t sector, nr_sects; struct blk_plug plug; int err; - if (!(mode & BLK_OPEN_WRITE)) - return -EBADF; - - if (!bdev_max_discard_sectors(bdev)) - return -EOPNOTSUPP; - if (bdev_read_only(bdev)) - return -EPERM; - if (copy_from_user(range, (void __user *)arg, sizeof(range))) return -EFAULT; - start = range[0]; len = range[1]; - if (!len) - return -EINVAL; - if ((start | len) & bs_mask) - return -EINVAL; + if (!bdev_max_discard_sectors(bdev)) + return -EOPNOTSUPP; - if (check_add_overflow(start, len, &end) || - end > bdev_nr_bytes(bdev)) - return -EINVAL; + if (!(mode & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; filemap_invalidate_lock(bdev->bd_mapping); err = truncate_bdev_range(bdev, mode, start, start + len - 1); From patchwork Fri Sep 6 22:57:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794853 Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E15C158DC3; Fri, 6 Sep 2024 22:57:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663428; cv=none; b=GhACAaQQ4WAjVVXdDGrFXWarMl5pxFSGKNHoa1Y0d5gtclUJcaPBArGUrOHj+2S1JOjAWbhOU0dnSdk7KUICy0yHSrJq2g3W36yTJivAKQQwzb77Mb26DDCW2rzfXtLmLHn09Wu+IBxjFtBwVrnAjdzoLxgMJ31JKP+aj6K6YIs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663428; c=relaxed/simple; bh=j+7zM8nY7uAZN93ODz2NwhZQYv08k76ZVh99E9m/LeE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SRUu6jE0jWKY87neyzS/YDzc6JLsC1ke5KixI6NRIS8zWrjUfxFyUashVsSipEj1wz6c7z3gHUsZN5BzsBCX6xp1Z5rsOw1aYdfaHGYbkvlH6IJ11cosLwuaT8Momhd1rqMOmrthWHNMQ0XYHrWoQXQ3kUYfGXHUhvrElcKV/Ds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fTjdm3X7; arc=none smtp.client-ip=209.85.167.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fTjdm3X7" Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-5365aa568ceso1286937e87.0; Fri, 06 Sep 2024 15:57:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663424; x=1726268224; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v7dFMMDbVFAIV5pvquHCkne/t7NzjwwosACmczIEiUs=; b=fTjdm3X79fX4NWw0EGm/jSeeQufTJLkgFtpCGHjwwFfq/w3v8TDjXOu5/SwyCT97Rt ptZ5/Y0UBuLunK57ypC+roFURLgJYvOg+MI+8nMBphBNYEkePMxVbh+s90R3IRgWJhOi +zuAjuHwO74PUzBEhtQyJObV3wrAJW62MehsSdhcaup7a0skUU3l7G0rDORM6zu1UYj3 nUkX3jKnfnAwXmngP0XyPOBjtlJ29gAiA8gUOnS8gA1cVj/tEMvz1A1D5DGGUCnsbJGM FgSBwEmcZ8K2K5GsIuUnF1RKOB8y9Br7H+3MZIpnItett77dVmv7ZdR+0fiCJp/K0Gen 8ehw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663424; x=1726268224; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v7dFMMDbVFAIV5pvquHCkne/t7NzjwwosACmczIEiUs=; b=AGxwpmefcy6mHT49mpynXFhkYOmDt4BHGWfff7ogLFAg/NmCBFlHs/PeQoCSYh9u1o wIQaxwzA6IZv5upe8mvBSj+meS8R2omnVt3XrIKHvfZV2Kpvw1TCLAHZG5Mn2GH+ObbO S6kUy2RFzM3nsCN2zTwzeESZBETrvtIAmZ8HdYLkqa+eX/04R2sJ2pEzUcjOUBd2hAe1 UMd5Ier7OpPmzGlYwOe3iMTYQVqIUyoaW45IJ3ZYXtc69RemK4p4jas1hewqjpxm4HsN EmH4lyyl2UfFQKz0X/rJDCgffORUccEe3gyqJDDFlLOGpfPrj32ftdfvlWkjCpggbrJf CgpQ== X-Forwarded-Encrypted: i=1; AJvYcCW+3sd1IodeyHvMgyoJs2XlIF3OyzFhFojkI8oOCymCTrjYiBsi9cgFKNdg0ol4ZN1m6KNh3aNVZocuTA==@vger.kernel.org X-Gm-Message-State: AOJu0YwxSQhuyF3d554xFuDk4pvu254XeKit1s9S73HswY2FCtrZCXx+ nASESGxjI5YEc9AczI4GxxTV3sX38ghwmRZWLYKm+i18hf8Q/Wa4xcfndUny X-Google-Smtp-Source: AGHT+IHt+QYHmRu0nquydclberbtpO0LZ0vL2z0xJXuhII8HVhOpdcMo8Pgh0RIOwsY85+0VzuhDqw== X-Received: by 2002:a05:6512:b88:b0:536:53d1:850d with SMTP id 2adb3069b0e04-536587f848emr3490089e87.39.1725663423587; Fri, 06 Sep 2024 15:57:03 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.57.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:57:03 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 5/8] block: implement async discard as io_uring cmd Date: Fri, 6 Sep 2024 23:57:22 +0100 Message-ID: <7fc0a61ae29190a42e958eddfefd6d44cdf372ad.1725621577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring allows to implement custom file specific operations via fops->uring_cmd callback. Use it to wire up asynchronous discard commands. Normally, first it tries to do a non-blocking issue, and if fails we'd retry from a blocking context by returning -EAGAIN to core io_uring. Note, unlike ioctl(BLKDISCARD) with stronger guarantees against races, we only do a best effort attempt to invalidate page cache, and it can race with any writes and reads and leave page cache stale. It's the same kind of races we allow to direct writes. Suggested-by: Conrad Meyer Signed-off-by: Pavel Begunkov --- block/blk-lib.c | 3 +- block/blk.h | 1 + block/fops.c | 2 + block/ioctl.c | 102 ++++++++++++++++++++++++++++++++++++++++ include/linux/bio.h | 2 + include/uapi/linux/fs.h | 2 + 6 files changed, 111 insertions(+), 1 deletion(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index 83eb7761c2bf..c94c67a75f7e 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -10,7 +10,8 @@ #include "blk.h" -static sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) +/* The maximum size of a discard that can be issued from a given sector. */ +sector_t bio_discard_limit(struct block_device *bdev, sector_t sector) { unsigned int discard_granularity = bdev_discard_granularity(bdev); sector_t granularity_aligned_sector; diff --git a/block/blk.h b/block/blk.h index 32f4e9f630a3..1a1a18d118f7 100644 --- a/block/blk.h +++ b/block/blk.h @@ -605,6 +605,7 @@ blk_mode_t file_to_blk_mode(struct file *file); int truncate_bdev_range(struct block_device *bdev, blk_mode_t mode, loff_t lstart, loff_t lend); long blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); +int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags); long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); extern const struct address_space_operations def_blk_aops; diff --git a/block/fops.c b/block/fops.c index 9825c1713a49..8154b10b5abf 100644 --- a/block/fops.c +++ b/block/fops.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "blk.h" static inline struct inode *bdev_file_inode(struct file *file) @@ -873,6 +874,7 @@ const struct file_operations def_blk_fops = { .splice_read = filemap_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .uring_cmd = blkdev_uring_cmd, .fop_flags = FOP_BUFFER_RASYNC, }; diff --git a/block/ioctl.c b/block/ioctl.c index a820f692dd1c..19fba8332eee 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -11,6 +11,8 @@ #include #include #include +#include +#include #include "blk.h" static int blkpg_do_ioctl(struct block_device *bdev, @@ -742,3 +744,103 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) return ret; } #endif + +struct blk_iou_cmd { + int res; + bool nowait; +}; + +static void blk_cmd_complete(struct io_uring_cmd *cmd, unsigned int issue_flags) +{ + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + + if (bic->res == -EAGAIN && bic->nowait) + io_uring_cmd_issue_blocking(cmd); + else + io_uring_cmd_done(cmd, bic->res, 0, issue_flags); +} + +static void bio_cmd_bio_end_io(struct bio *bio) +{ + struct io_uring_cmd *cmd = bio->bi_private; + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + + if (unlikely(bio->bi_status) && !bic->res) + bic->res = blk_status_to_errno(bio->bi_status); + + io_uring_cmd_do_in_task_lazy(cmd, blk_cmd_complete); + bio_put(bio); +} + +static int blkdev_cmd_discard(struct io_uring_cmd *cmd, + struct block_device *bdev, + uint64_t start, uint64_t len, bool nowait) +{ + gfp_t gfp = nowait ? GFP_NOWAIT : GFP_KERNEL; + sector_t sector = start >> SECTOR_SHIFT; + sector_t nr_sects = len >> SECTOR_SHIFT; + struct bio *prev = NULL, *bio; + int err; + + if (!bdev_max_discard_sectors(bdev)) + return -EOPNOTSUPP; + + if (!(file_to_blk_mode(cmd->file) & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; + + /* + * Don't allow multi-bio non-blocking submissions as subsequent bios + * may fail but we won't get a direct indication of that. Normally, + * the caller should retry from a blocking context. + */ + if (nowait && nr_sects > bio_discard_limit(bdev, sector)) + return -EAGAIN; + + err = filemap_invalidate_pages(bdev->bd_mapping, start, + start + len - 1, nowait); + if (err) + return err; + + while ((bio = blk_alloc_discard_bio(bdev, §or, &nr_sects, gfp))) { + if (nowait) + bio->bi_opf |= REQ_NOWAIT; + prev = bio_chain_and_submit(prev, bio); + } + if (!prev) + return -EAGAIN; + + prev->bi_private = cmd; + prev->bi_end_io = bio_cmd_bio_end_io; + submit_bio(prev); + return -EIOCBQUEUED; +} + +int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) +{ + struct block_device *bdev = I_BDEV(cmd->file->f_mapping->host); + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + const struct io_uring_sqe *sqe = cmd->sqe; + u32 cmd_op = cmd->cmd_op; + uint64_t start, len; + + if (unlikely(sqe->ioprio || sqe->__pad1 || sqe->len || + sqe->rw_flags || sqe->file_index)) + return -EINVAL; + + bic->res = 0; + bic->nowait = issue_flags & IO_URING_F_NONBLOCK; + + start = READ_ONCE(sqe->addr); + len = READ_ONCE(sqe->addr3); + + switch (cmd_op) { + case BLOCK_URING_CMD_DISCARD: + return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); + } + return -EINVAL; +} diff --git a/include/linux/bio.h b/include/linux/bio.h index faceadb040f9..78ead424484c 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -684,4 +684,6 @@ struct bio *bio_chain_and_submit(struct bio *prev, struct bio *new); struct bio *blk_alloc_discard_bio(struct block_device *bdev, sector_t *sector, sector_t *nr_sects, gfp_t gfp_mask); +sector_t bio_discard_limit(struct block_device *bdev, sector_t sector); + #endif /* __LINUX_BIO_H */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..7ea41ca97158 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -208,6 +208,8 @@ struct fsxattr { * (see uapi/linux/blkzoned.h) */ +#define BLOCK_URING_CMD_DISCARD _IO(0x12,137) + #define BMAP_IOCTL 1 /* obsolete - kept for compatibility */ #define FIBMAP _IO(0x00,1) /* bmap access */ #define FIGETBSZ _IO(0x00,2) /* get the block size used for bmap */ From patchwork Fri Sep 6 22:57:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794854 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7B1D15B10B; Fri, 6 Sep 2024 22:57:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663429; cv=none; b=BE0+n3Cv3B8/ct+vyRtgSat7K97YKLaHLxWvF2krVq5E3cNf3eCXgpYhmXTn2csxZU3xYvC/l2wwQSjrQDkGsDKmC4YM5B84fTPUKyRptPQ9eHG7hVUon9ocWnfISKcW7iJjCHSz3jMcLqab53yR2RmQePQ0aFk+/IeN9IFbYM0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663429; c=relaxed/simple; bh=lyDccPclJxGZedhLYadtZmxYdaswgU6YsCGNLhOcKgM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DxMgGmAscrtnuAkMY+J6jFqjuz21SYpo3P6fsHpDa83uwxLXxg9MFkHpAj5Jd5pXTKys4DRmP/E9Vp//LxGO6j20cmXr2lxY0FvLFSC2X+1cSrOidDYFJoTnMgTcea5PBY6114Wjli+U2vBYYzmUhIwI9beQfaSfPk7qsB2O5JY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gpizstF0; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gpizstF0" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a868b739cd9so336002466b.2; Fri, 06 Sep 2024 15:57:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663426; x=1726268226; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sVNRBhtechMDGkg+NH1UKI/XUcjXx3fSa2oGaERaZjA=; b=gpizstF0vvM1yQQkTvSDZes8KRn6U+zPNVRbOqmOadV1qwfNi69Y0dvwLzzbUMteRq ickWom9bpeVno/N8S4T7Pv/o9CYFA2y421o2NvX3ZOnBOS+5WkKEaavk4YPiohXHubzW vr/BGiwXVKcFjLUwm+vnR4zYqtMRzVk2DtPjLLLydLp0ye7coz8cq6Zar+M5DlpDOwmZ EVP+pzoeedOjeYJzxWIoC3/nkVBdh0jCgKgho2EZUqOrxw2EBdb5RfHebHkqYNFs1YkH YGznCFGJpl/etPkwbNTzqIIRBF1LSLfMLLmsXU7sHR3fZlSpJY6omWB0L98lUNzjPMee xwsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663426; x=1726268226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sVNRBhtechMDGkg+NH1UKI/XUcjXx3fSa2oGaERaZjA=; b=PNYU0Rz09vnwsSM77wDEW5rdVehVvq4F7etNiv/z2IlNV5WxQZRqXxSTa8/tKiN2/V 8zwrVk9C8qad5DW8NTn9PeD1o6yrLmq0bAm5E58PPK2EZ5iqnZ8SxKmy5yP8XMr3/gum e/POW9kVWocN8qtU76c3GUEZvJEjRk7pdHnrME3S0XtxUXBSUWV6jxJVpwSQuEph+F/y n6kPSP/wc1j9yVM8ZBgeOQO0lky3qqp92/SGK4nU1UH95GVfYIZjoAQlb6nEGbjM8nZp qU654sx2REuqF1Nvt9q5WwRAWcd/4IqPb+WHdTY6oPxE2XBKA+Gu+U6H30prLyqsP5tM LKGA== X-Forwarded-Encrypted: i=1; AJvYcCVLf01gYazF45YizydxE5J0k04F3tJNsOUHkQesuMQrJ7/kyugkuSsY5mts4PeWC4BVXUR1CXfA4e0gdA==@vger.kernel.org X-Gm-Message-State: AOJu0YzQHV0f63qdKogFyFzCw1xaJKrzyAM2jQi2gFiUxsnMLYvj2Trp d8ib0ciQNPaSUKPd5Aw1XBe+5cAy0Mrg/gGrpV5rQhsUktzb0XlVBhm4u+3w X-Google-Smtp-Source: AGHT+IGwtkNX+b7B+F+E65xCqdvf83CYwy/QQoEkioql8r4UWUgpbreyku1f9KDVfE9jAVw9Or4nJQ== X-Received: by 2002:a17:907:968b:b0:a86:b46b:860a with SMTP id a640c23a62f3a-a8d1c738eedmr53296966b.54.1725663425531; Fri, 06 Sep 2024 15:57:05 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.57.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:57:05 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 6/8] block: implement async write zeroes command Date: Fri, 6 Sep 2024 23:57:23 +0100 Message-ID: <2a99dd14c2e0c1fced433822a13ff00735a84816.1725621577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add another io_uring cmd for block layer implementing asynchronous write zeroes. It reuses helpers we've added for async discards, and inherits the code structure as well as all considerations in regards to page cache races. Suggested-by: Conrad Meyer Signed-off-by: Pavel Begunkov --- block/ioctl.c | 64 +++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/fs.h | 1 + 2 files changed, 65 insertions(+) diff --git a/block/ioctl.c b/block/ioctl.c index 19fba8332eee..ef4b2a90ad79 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -772,6 +772,67 @@ static void bio_cmd_bio_end_io(struct bio *bio) bio_put(bio); } +static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, + struct block_device *bdev, + uint64_t start, uint64_t len, bool nowait) +{ + + sector_t bs_mask = (bdev_logical_block_size(bdev) >> SECTOR_SHIFT) - 1; + sector_t limit = bdev_write_zeroes_sectors(bdev); + sector_t sector = start >> SECTOR_SHIFT; + sector_t nr_sects = len >> SECTOR_SHIFT; + struct bio *prev = NULL, *bio; + gfp_t gfp = nowait ? GFP_NOWAIT : GFP_KERNEL; + int err; + + if (!(file_to_blk_mode(cmd->file) & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; + + if (!limit) + return -EOPNOTSUPP; + /* + * Don't allow multi-bio non-blocking submissions as subsequent bios + * may fail but we won't get a direct indication of that. Normally, + * the caller should retry from a blocking context. + */ + if (nowait && nr_sects > limit) + return -EAGAIN; + + err = filemap_invalidate_pages(bdev->bd_mapping, start, + start + len - 1, nowait); + if (err) + return err; + + limit = min(limit, (UINT_MAX >> SECTOR_SHIFT) & ~bs_mask); + while (nr_sects) { + sector_t bio_sects = min(nr_sects, limit); + + bio = bio_alloc(bdev, 0, REQ_OP_WRITE_ZEROES|REQ_NOUNMAP, gfp); + if (!bio) + break; + if (nowait) + bio->bi_opf |= REQ_NOWAIT; + bio->bi_iter.bi_sector = sector; + bio->bi_iter.bi_size = bio_sects << SECTOR_SHIFT; + sector += bio_sects; + nr_sects -= bio_sects; + + prev = bio_chain_and_submit(prev, bio); + } + if (!prev) + return -EAGAIN; + + prev->bi_private = cmd; + prev->bi_end_io = bio_cmd_bio_end_io; + submit_bio(prev); + return -EIOCBQUEUED; +} + static int blkdev_cmd_discard(struct io_uring_cmd *cmd, struct block_device *bdev, uint64_t start, uint64_t len, bool nowait) @@ -841,6 +902,9 @@ int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) switch (cmd_op) { case BLOCK_URING_CMD_DISCARD: return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); + case BLOCK_URING_CMD_WRITE_ZEROES: + return blkdev_cmd_write_zeroes(cmd, bdev, start, len, + bic->nowait); } return -EINVAL; } diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 7ea41ca97158..68b0fccebf92 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -209,6 +209,7 @@ struct fsxattr { */ #define BLOCK_URING_CMD_DISCARD _IO(0x12,137) +#define BLOCK_URING_CMD_WRITE_ZEROES _IO(0x12,138) #define BMAP_IOCTL 1 /* obsolete - kept for compatibility */ #define FIBMAP _IO(0x00,1) /* bmap access */ From patchwork Fri Sep 6 22:57:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794855 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49D6715B54E; Fri, 6 Sep 2024 22:57:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663430; cv=none; b=uw1UmvuuvtMQ2uxC/zDAmEgNTMjc6EN9bPO7z/PlNfI5j+ny6xOuxwUM5FrJAejVTyQoX1Sh5yQB11bluEvH+gqvnqf3xFxZXxbtjhPYx8cmUZoZin9CN7aGkF/ehHXitI7BgPnJM8B9aEBWMWY7pMIHEP1M4f/ciq0sIN6w/I8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663430; c=relaxed/simple; bh=78UR0qruQh9KW0yalCw2O7xPrgksBC9fYQyOrc8Aubg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DKJ6lVg/12mdT0tWwRobTL2G39PEkOqqY4TISP+zq8ZYGjdfMMAguSgQCTL+sdjCS6JgMnq233D/v7ZPwZI8zRuJ6WeL2gfC+FXhKGQ8Jq70BgFqtuWpXSbGb+9ARdse/BbEvDjyrmRhNZII/kRzK2/NwiUSndaRPk9TKwXyrWE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iPLODqSO; arc=none smtp.client-ip=209.85.218.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iPLODqSO" Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-a8ce5db8668so112530966b.1; Fri, 06 Sep 2024 15:57:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663427; x=1726268227; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jYezJHCo7v489eblMcj1bTcVFXeBrX/vk5R7qG2A1Yk=; b=iPLODqSO95xJ8LezNfwQmV0CBQjTWMC86BPKjs76QLXZIIhUlOr7UH4WWiF6Q53Cr/ PwOznYDnmR5AIWkmxQiKn2PUcuipWdN91jI7l1+MeQrw5hxD7b6lfNTKjdOwUOG3AALo +7rO15Kpb5WXztBDykinIl8O2asGikK13uuqsewntJR0C7u+h2SjuEgD5piQsdfqmajQ coq1RjK/t3KaVS/V/4xpV45qur14rFcVadJxVkoA8hjuhwG/I4+YRrColMMWkvl2+TJT jwIf6PUU2CuYyQXQfGl7ic9Yq0835Kz0GbYi8VocRmteC6UtHD3ou4szjHrpAYsQd9Uo Zm0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663427; x=1726268227; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jYezJHCo7v489eblMcj1bTcVFXeBrX/vk5R7qG2A1Yk=; b=bjUZWjbV4U8vWPfG0ALFKqm1hM4BWZDM+81zSPND2t5NpR5KFowTlezjZj3cteTaNU INX3peqC8LuE/848Tx+3CGod7CiS5htBMHktX/m2WFgf9HPn38pwK8urJb2GYT4mxycq 5+C2unDkCwdkSlPhfTwYBvjJidehPSy7QUoJmrPjTutHgFiOCkNnrYtxu+p0wu5oSPxH 5EyCmhM1mPAhRGVBsOKTfkBB/M45ZwsOHdx+DZENbJmoAvL06Skggsp0ApYsf6fzu6nM ucS0qUxx9D4UR3hF477DiGqfWN9M4Ri6YBqEJxs+XenSaoNDOmMc0/F2/SwXe6LOvIBT 0xYg== X-Forwarded-Encrypted: i=1; AJvYcCXSpxRuTMjW551MfX4H6VPwIZR1v+6MdL9Y1PJKMbrF5yqdwoVS9ibu+g5Bz014F6egQogr+todyhS7cw==@vger.kernel.org X-Gm-Message-State: AOJu0YyuMF069s1Thhzd8xUASER/wKld77CJ0vfbCdB6/rYuMR4jgXCA OQ/i9hvKW7dXzFx31uys8LATt22h/w64+L8kAHgdTHJYBGI7/I0j++LEmfWh X-Google-Smtp-Source: AGHT+IE/3M7b/K3pyseCsX+NiZT9J56A5VyhoP1o7o5GWtiHew5dHLC0T6ZxFj2yq0dOHss7K5NZbA== X-Received: by 2002:a17:907:3f20:b0:a8d:caa:7fee with SMTP id a640c23a62f3a-a8d1bf75ec5mr67443566b.7.1725663427142; Fri, 06 Sep 2024 15:57:07 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.57.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:57:06 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 7/8] block: add nowait flag for __blkdev_issue_zero_pages Date: Fri, 6 Sep 2024 23:57:24 +0100 Message-ID: <387caa9b7a23061f19034a8afd1dfcf017d8fe35.1725621577.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To reuse __blkdev_issue_zero_pages() in the following patch, we need to make it work with non-blocking requests. Add a new nowait flag we can pass inside. Return errors if something went wrong, and check bio_alloc() for failures, which wasn't supposed to happen before because of what gfp flags the callers are passing. Note that there might be a bio passed back even when the function returned an error. To limit the scope of the patch, don't add return code handling to callers, that can be deferred to a follow up. Signed-off-by: Pavel Begunkov --- block/blk-lib.c | 24 +++++++++++++++++++----- include/linux/bio.h | 4 ++++ include/linux/blkdev.h | 1 + 3 files changed, 24 insertions(+), 5 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index c94c67a75f7e..0d8f1b93b4c3 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -193,20 +193,32 @@ static unsigned int __blkdev_sectors_to_bio_pages(sector_t nr_sects) return min(pages, (sector_t)BIO_MAX_VECS); } -static void __blkdev_issue_zero_pages(struct block_device *bdev, +int blkdev_issue_zero_pages_bio(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned int flags) { + blk_opf_t opf = REQ_OP_WRITE; + + if (flags & BLKDEV_ZERO_PAGES_NOWAIT) { + sector_t max_bio_sectors = BIO_MAX_VECS << PAGE_SECTORS_SHIFT; + + if (nr_sects > max_bio_sectors) + return -EAGAIN; + opf |= REQ_NOWAIT; + } + while (nr_sects) { unsigned int nr_vecs = __blkdev_sectors_to_bio_pages(nr_sects); struct bio *bio; - bio = bio_alloc(bdev, nr_vecs, REQ_OP_WRITE, gfp_mask); + bio = bio_alloc(bdev, nr_vecs, opf, gfp_mask); + if (!bio) + return -ENOMEM; bio->bi_iter.bi_sector = sector; if ((flags & BLKDEV_ZERO_KILLABLE) && fatal_signal_pending(current)) - break; + return -EINTR; do { unsigned int len, added; @@ -223,6 +235,8 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev, *biop = bio_chain_and_submit(*biop, bio); cond_resched(); } + + return 0; } static int blkdev_issue_zero_pages(struct block_device *bdev, sector_t sector, @@ -236,7 +250,7 @@ static int blkdev_issue_zero_pages(struct block_device *bdev, sector_t sector, return -EOPNOTSUPP; blk_start_plug(&plug); - __blkdev_issue_zero_pages(bdev, sector, nr_sects, gfp, &bio, flags); + blkdev_issue_zero_pages_bio(bdev, sector, nr_sects, gfp, &bio, flags); if (bio) { if ((flags & BLKDEV_ZERO_KILLABLE) && fatal_signal_pending(current)) { @@ -286,7 +300,7 @@ int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, } else { if (flags & BLKDEV_ZERO_NOFALLBACK) return -EOPNOTSUPP; - __blkdev_issue_zero_pages(bdev, sector, nr_sects, gfp_mask, + blkdev_issue_zero_pages_bio(bdev, sector, nr_sects, gfp_mask, biop, flags); } return 0; diff --git a/include/linux/bio.h b/include/linux/bio.h index 78ead424484c..87d85b326e1e 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -686,4 +686,8 @@ struct bio *blk_alloc_discard_bio(struct block_device *bdev, sector_t bio_discard_limit(struct block_device *bdev, sector_t sector); +int blkdev_issue_zero_pages_bio(struct block_device *bdev, + sector_t sector, sector_t nr_sects, gfp_t gfp_mask, + struct bio **biop, unsigned int flags); + #endif /* __LINUX_BIO_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 643c9020a35a..bf1aa951fda2 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1098,6 +1098,7 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ #define BLKDEV_ZERO_KILLABLE (1 << 2) /* interruptible by fatal signals */ +#define BLKDEV_ZERO_PAGES_NOWAIT (1 << 3) /* non-blocking submission */ extern int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, From patchwork Fri Sep 6 22:57:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13794856 Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7563615B57C; Fri, 6 Sep 2024 22:57:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663433; cv=none; b=pyhX3R8sX83NhvKnXq702YwJuC+ni7cdW1V/nkz0AIJAoYgZl6bOAeVCeVNfD6Rp0+fNSkHNj/KUIW5o4n3xwfPTbj7DxcWOUMHnwJqDOR+qIXErjqKHVS8nOyfVUDW0N0Jkk+bqQu2kS2eRcTXmeFwbQQUtrqoYuMR0x1rLpy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725663433; c=relaxed/simple; bh=fX6prqf2VFD2UV5gx7IfCiSBWUdjjihLSk5KGUumqJA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZNyJNKctalR/RJDPPPN2cD97r01HOcurwVyiDg4EZovrHT+HzGNLAlBmcRjlsK0jFNvyKSGL5VjEs9jMT2VHio6Lc/lG76OccEHrGrVFc8Eus1JDDp+4S3Pyflk43zMEQbf+P+kFI/JeQ/2A8YQRG1mncwUtUNIqwKEbHuZoFJ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CDIUu+Ao; arc=none smtp.client-ip=209.85.167.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CDIUu+Ao" Received: by mail-lf1-f54.google.com with SMTP id 2adb3069b0e04-535be093a43so3180061e87.3; Fri, 06 Sep 2024 15:57:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725663429; x=1726268229; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=llWa+KY1CsE699NO6/Qz9OdhajA1Gh72FplwVQHN8Os=; b=CDIUu+Aox8Z3H8bOdSttwSPxdkc00prGvp90oFfJRsB4F9gphuR7MTmCX+hgBEBaAe PamvlXyfAzCzM8dc+tW91Mw+ei1zoQg8OyP2GkSHIrXQ37BgcKufbFr99WwxxrGHuSCe ZAHt95jc7uujescyQtei8SNqJKnTw0t1fIH0Ga01PHtgO3ONl2qzO/NsUaFiu2tJ9VpJ AkN2MBMX4Ee2UYjHcVNd9OE/BYGYRTMTm2S/IBOjdFB60qGNPUlxZMSjHoimDG6QkEGR FJKP7Z9UAnnj2QVxea9jkhtSep/nNH89vDcGOSnjLL7jiwTgLT1M297YBoSasHvuPE+1 3zLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725663429; x=1726268229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=llWa+KY1CsE699NO6/Qz9OdhajA1Gh72FplwVQHN8Os=; b=kfGIOuFF2urkmb0TXdywkMIhNOHeRB+A/LLxgNL+ZT6pupsBhLu+Ke4QPRN95azygz /BqeiLooZl155xYgi1YDRGAekXUVEMMh/LWYTt1+3o2irsINavaVE2bGOHlRkGnK85St mNRUZeRZsaZfm90PQJoz4qQekHz/CXMqJwisOzADb+2T8qyw17dJpsggA451v0Dv4W78 MH2jboDr/EjjB5Hzkr5uGsci3G80O4yz57UvMFDQyuLfIhbRWM2fEAYjB4qwUBWUMaBj 73GqbmXQiYQeZqc0sfB0J7+LSImoXAgjpsNGdcf+FpRsTUbl2VgJh+3l0uVD7/mxEXrm +7dA== X-Forwarded-Encrypted: i=1; AJvYcCUyBBO0Gz7J6fj8nD5JyFs4AkCpm/XqgL57pHzVxk0FFszlgcne61tq1CHGs2oQYSvIUgreBi6miY2otA==@vger.kernel.org X-Gm-Message-State: AOJu0YzLUPJ+8kawVyv5QpXZtWXW83jMMDecCXjOYoxs7LFXUcHKFdOo vsz3Zuwkq/VhxOt7vav+FDcyqIBH59YvgFSjrS4gHCQhV+c6b5LVOf5F/xAQ X-Google-Smtp-Source: AGHT+IF7dy6Fwnb2jmgqk1DE1zvsMyOI+fNBEs5xezgv4QpyDJkwby70G9EHhdLSnvBeBbSkACl5HA== X-Received: by 2002:a05:6512:10d0:b0:52c:d904:d26e with SMTP id 2adb3069b0e04-536587abee5mr2790225e87.21.1725663429138; Fri, 06 Sep 2024 15:57:09 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.146.236]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25d54978sm2679566b.199.2024.09.06.15.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Sep 2024 15:57:08 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Conrad Meyer , linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v4 8/8] block: implement async write zero pages command Date: Fri, 6 Sep 2024 23:57:25 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a command that writes the zero page to the drive. Apart from passing the zero page instead of actual data it uses the normal write path and doesn't do any further acceleration, nor it requires any special hardware support. The indended use is to have a fallback when BLOCK_URING_CMD_WRITE_ZEROES is not supported. Signed-off-by: Pavel Begunkov --- block/ioctl.c | 24 +++++++++++++++++++++--- include/uapi/linux/fs.h | 1 + 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index ef4b2a90ad79..3cb479192023 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -774,7 +774,8 @@ static void bio_cmd_bio_end_io(struct bio *bio) static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, struct block_device *bdev, - uint64_t start, uint64_t len, bool nowait) + uint64_t start, uint64_t len, + bool nowait, bool zero_pages) { sector_t bs_mask = (bdev_logical_block_size(bdev) >> SECTOR_SHIFT) - 1; @@ -793,6 +794,20 @@ static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, if (err) return err; + if (zero_pages) { + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, + struct blk_iou_cmd); + + err = blkdev_issue_zero_pages_bio(bdev, sector, nr_sects, + gfp, &prev, + BLKDEV_ZERO_PAGES_NOWAIT); + if (!prev) + return -EAGAIN; + if (err) + bic->res = err; + goto out_submit; + } + if (!limit) return -EOPNOTSUPP; /* @@ -826,7 +841,7 @@ static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, } if (!prev) return -EAGAIN; - +out_submit: prev->bi_private = cmd; prev->bi_end_io = bio_cmd_bio_end_io; submit_bio(prev); @@ -904,7 +919,10 @@ int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); case BLOCK_URING_CMD_WRITE_ZEROES: return blkdev_cmd_write_zeroes(cmd, bdev, start, len, - bic->nowait); + bic->nowait, false); + case BLOCK_URING_CMD_WRITE_ZERO_PAGE: + return blkdev_cmd_write_zeroes(cmd, bdev, start, len, + bic->nowait, true); } return -EINVAL; } diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 68b0fccebf92..f4337b87d846 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -210,6 +210,7 @@ struct fsxattr { #define BLOCK_URING_CMD_DISCARD _IO(0x12,137) #define BLOCK_URING_CMD_WRITE_ZEROES _IO(0x12,138) +#define BLOCK_URING_CMD_WRITE_ZERO_PAGE _IO(0x12,139) #define BMAP_IOCTL 1 /* obsolete - kept for compatibility */ #define FIBMAP _IO(0x00,1) /* bmap access */