From patchwork Wed Sep 11 16:34:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800896 Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 976971AB6DA; Wed, 11 Sep 2024 16:34:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072467; cv=none; b=NUTyH5yjjJp+ktpDOKQhGWdojRirrqz7pV8nz+d20VTEBdpY979RIIg72Y88upnaXn4BANo3gTEZtBtK1ER979AyIgyJbGTVZ+oumC/E6n2RgNi33kQI1qKlSkfrPHIvGTe47Ryz6/ew6kZBTcWlVlx7qeGDgq2ZPsHgdq9dEJk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072467; c=relaxed/simple; bh=2vI0bIvl9+D8oswmBaa6Rx1hzhaGczPhBrLqovoOrl0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PRXRm//JtF2yydHWc5tb5zY4SkNNySMkCHDak9aVcJvZOvZRigIRAg+vIlGYOuzc63au2rtntHyEll8za2B3zXrfmXhkFMvxHi5QhDqjnaDBGyM2zUAR3uHPeZFqBg/kuSh+5+roA+BWp4LZ36gYXFCOpPTSst7oYJzooaVYRhM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=K8m9LrkG; arc=none smtp.client-ip=209.85.167.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="K8m9LrkG" Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-5365aa568ceso14564e87.0; Wed, 11 Sep 2024 09:34:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072463; x=1726677263; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KZ8pkxfeOzRC4kA801xaO2cK98vlGCE9d9luTt+J2Kg=; b=K8m9LrkGqbyVhXQheilOU/ZAIWUGKen+T5lGhdBjtZfLDJ2ijYC169s6jjl/N1Whpn uMob1BMS18bwK/jhKsRadhlSKbsW9FzDQ5nlVFRNUlO7W207ANDw406K8mDXn3owZFpt YYV2TRW1SstgukfdIR0xlRDOe/tEsC2K/FnZIIQz3bXhy5/Uw3f5GFG7fUGGCX6mEkWt e8GyeCFGF/hrTXt6uk3+8UQTJ40NxBoFfA6nsCedHXlpUmen3AmJO2hkSyfXtCvDKhXX V7Utd9MT05v6RZhBsC751K9OSqrpWJcgE50wYipv7XNHeK9+gauVLkAMNnWygCzrrKOJ S4XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072463; x=1726677263; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KZ8pkxfeOzRC4kA801xaO2cK98vlGCE9d9luTt+J2Kg=; b=W2VVLlT7Q1TesRwiJwbrcOVzKdFeGn/BDldprVHND/Av2bAAg9LnJxnJ3urboSEJVs eIp8RmXagbfft797h8VHwjW4TGGgLHNND4sUw/GNkB4MOyZacIw+ImPk5zEONOwphdG0 X/l+SveCEbO0E0t9zUS+juczySwSXr2jMSqDgeisK1uGr/iqya4OqKb14ZoX0itpPAWk UjR24vSpYXhtyxfqzNaRghFwr+XV2eEr+Q1ZDT5HbDEb+WV1ir/46O8l4o1EmZfGtiV+ NhYiU0jsmxR/6Y3SIJmANCh/G4LZp6iZsNnEbLL/52wAv5tYtNKtSx/0pBD3zCGo2/nQ HrZw== X-Forwarded-Encrypted: i=1; AJvYcCUFurCcFCs/8CuT27Rq8Pf8KuZ0y+Xqy/FjXjZiVGGgq3eeOKWvmXKJIkQbgB7KyrAcvaiLjEH/oddBCQ==@vger.kernel.org X-Gm-Message-State: AOJu0YzK3XRVqLyps/v31jAaM7UMoq0iWq+kz64nSXY/9DZLilcdSmZO BPxKP2lZAcuDnWIzcmHeU6AM3vXZb53VnhKwQ8T5psQOZTfKVq0wnpyJCGPq X-Google-Smtp-Source: AGHT+IEG2kHGTzNGAdSjpEaXDSIgfMbjRxkjuMwMgmsa9Moi42gUvNZP3GJN98tnR/Gf0UwGcIEW0Q== X-Received: by 2002:a05:6512:10d5:b0:535:681d:34b0 with SMTP id 2adb3069b0e04-5365880bc04mr17121594e87.47.1726072462342; Wed, 11 Sep 2024 09:34:22 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:21 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v5 1/8] io_uring/cmd: expose iowq to cmds Date: Wed, 11 Sep 2024 17:34:37 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When an io_uring request needs blocking context we offload it to the io_uring's thread pool called io-wq. We can get there off ->uring_cmd by returning -EAGAIN, but there is no straightforward way of doing that from an asynchronous callback. Add a helper that would transfer a command to a blocking context. Note, we do an extra hop via task_work before io_queue_iowq(), that's a limitation of io_uring infra we have that can likely be lifted later if that would ever become a problem. Signed-off-by: Pavel Begunkov --- include/linux/io_uring/cmd.h | 6 ++++++ io_uring/io_uring.c | 11 +++++++++++ io_uring/io_uring.h | 1 + io_uring/uring_cmd.c | 7 +++++++ 4 files changed, 25 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 447fbfd32215..86ceb3383e49 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -48,6 +48,9 @@ void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags); +/* Execute the request from a blocking context */ +void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd); + #else static inline int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) @@ -67,6 +70,9 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags) { } +static inline void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd) +{ +} #endif /* diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 1aca501efaf6..86cf31902841 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -533,6 +533,17 @@ static void io_queue_iowq(struct io_kiocb *req) io_queue_linked_timeout(link); } +static void io_req_queue_iowq_tw(struct io_kiocb *req, struct io_tw_state *ts) +{ + io_queue_iowq(req); +} + +void io_req_queue_iowq(struct io_kiocb *req) +{ + req->io_task_work.func = io_req_queue_iowq_tw; + io_req_task_work_add(req); +} + static __cold void io_queue_deferred(struct io_ring_ctx *ctx) { while (!list_empty(&ctx->defer_list)) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 65078e641390..9d70b2cf7b1e 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -94,6 +94,7 @@ int io_uring_alloc_task_context(struct task_struct *task, int io_ring_add_registered_file(struct io_uring_task *tctx, struct file *file, int start, int end); +void io_req_queue_iowq(struct io_kiocb *req); int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts); int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr); diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 8391c7c7c1ec..39c3c816ec78 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -277,6 +277,13 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, } EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed); +void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd) +{ + struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); + + io_req_queue_iowq(req); +} + static inline int io_uring_cmd_getsockopt(struct socket *sock, struct io_uring_cmd *cmd, unsigned int issue_flags) From patchwork Wed Sep 11 16:34:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800895 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D88E1AB6FE; Wed, 11 Sep 2024 16:34:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072467; cv=none; b=jBhluybBEa4qSX47eul7R3aXoFaktWmhhhS+YO7ZvFx64TmNCTMdRpX4agSHZsSIcQV/BAQsA0t+fOrUlH6HISh/z2cybnzOXJa7Hr6Hks8nGzLoFHHQxAy9iLHr6QuWIBxjE/UlZmRlWIowM1/op0PleJfAgTLsok4K1HlscM8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072467; c=relaxed/simple; bh=xgZK/jw8ZlQXj8cIxR0W7ybGVnlcP1lJNS+9Q/l9a0s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B51+nyUZFg+aO8px768aLfBA0no0hGKQN4XzHTeBRRdYuMXDJMe3jSeBn87HlIqjRq8QksamF4ShsdWQQapPUIJWrwNOFGGuu3p1seL1n+u9Kr2M5wuD58TXQKHenpp20FK3Qbg4afyn139swG9ZDRJSwAo2LfAdUbJ5bTrOnd8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Kvfx9mF3; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Kvfx9mF3" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a8d4093722bso5583766b.0; Wed, 11 Sep 2024 09:34:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072463; x=1726677263; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yyZRXxiAI12SAMIbkwT8bn9l7S8e/TKEaLOeKBVDL1c=; b=Kvfx9mF3sJfmfjZZ+a2nYDQK9YPTkyOvQS5Z39s63fTyatNtNsuWfl43ujvpj0CSjc RAlFe0z7yyOOda2YRPkty7WWvMZbg3TJkHikkljF2lCqPiH4Ssyt/n+kNKvv84W8IfjL RCBpKHUM6QNmKCVuyh4xq/mygh02F21dG6tjB8ApQGgbliZyscTNIzVgxyo0mhBzK/Dl 7+1fsYJ33FFYVjRlmCT8O3GuJkYPj6+aNKCTE56BLiiXUrBgHhRJQCSG6S5VKhRoTyil kxCy01XRGEO77B/occxn/aRZc21jk0Sc4ADETUl0BKo8HffoEVd7EREj9fPjnAnxDaAK UpmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072463; x=1726677263; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yyZRXxiAI12SAMIbkwT8bn9l7S8e/TKEaLOeKBVDL1c=; b=J6hy3G7TTfzHCuebD79n8UWsS8YW5VLszbFKZNeBuwdmiRVHWbBH1yyyQxBHAJunPw wOeXzp4uZr7AdF0v6DkmXRWPf9FlEzMlIFDLT5r1Kv4ydujG0j2MUNW8vS825w/JlHwI CAtG8uiuDS8cStEkTqf2Gex/h1SjHqbQEPU7wIQqKY/0sWQ5Wxzm35z44e0TAJBzI29k bGbzUGsFjP4iUpWjQ+bUNU9r/u8+3FsxQvFVxjt3iYWQEvGx7Vh0aspKhmFzZmdVvsU/ afYjBwkrBFp6vTupXQIBshMkqlADCKNlBTVQdZU+ROlT+JKyWaxDdxw64SkMmCddrpFg Iyew== X-Forwarded-Encrypted: i=1; AJvYcCVAcV3pCWZ8rDYqdFR/vBgfMdW1AclsG5TYNrwUlaBxDXLMLDxZFKIR6fsMoT+4Jl32JIoCUOvlwPqvfw==@vger.kernel.org X-Gm-Message-State: AOJu0YxfEqOIsDvSb1JxU4aZTnEKV4AJqOnlUsi3Zt+I4qYvOQOpTJXA b6y8t9La/VhzdOKI1IHZyveXYFqXj4oYZW3nRJaFmNQuyXYOubQno6ZFedPO X-Google-Smtp-Source: AGHT+IELcXHRTUSdLyLG5jnLYvW4/fqCNo1EAgwvWA0FaGjFDUj4/njuzwhpx/wmLaysANwmyDI4Hw== X-Received: by 2002:a17:907:2da3:b0:a7d:9f92:9107 with SMTP id a640c23a62f3a-a9029690793mr5422666b.58.1726072463487; Wed, 11 Sep 2024 09:34:23 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:22 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v5 2/8] io_uring/cmd: give inline space in request to cmds Date: Wed, 11 Sep 2024 17:34:38 +0100 Message-ID: <7ca779a61ee5e166e535d70df9c7f07b15d8a0ce.1726072086.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Some io_uring commands can use some inline space in io_kiocb. We have 32 bytes in struct io_uring_cmd, expose it. Signed-off-by: Pavel Begunkov --- include/linux/io_uring/cmd.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 86ceb3383e49..c189d36ad55e 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -23,6 +23,15 @@ static inline const void *io_uring_sqe_cmd(const struct io_uring_sqe *sqe) return sqe->cmd; } +static inline void io_uring_cmd_private_sz_check(size_t cmd_sz) +{ + BUILD_BUG_ON(cmd_sz > sizeof_field(struct io_uring_cmd, pdu)); +} +#define io_uring_cmd_to_pdu(cmd, pdu_type) ( \ + io_uring_cmd_private_sz_check(sizeof(pdu_type)), \ + ((pdu_type *)&(cmd)->pdu) \ +) + #if defined(CONFIG_IO_URING) int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); From patchwork Wed Sep 11 16:34:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800897 Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2A471AED3F; Wed, 11 Sep 2024 16:34:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072469; cv=none; b=KmIb+/88tQ10d+ljklVzULApk/7Ujdm+IdoG7SCX+jeWPgc6P1sLMCc4x802SsO15U6WunYQ2ILkhdS0jsjWOZlgXtcwBfCcG0TpbX5zvuzKKVufrpRqWwfLjbXfBnxwEEBG3tZxRpjs23gAY5JHHwWgqhA9sEn/WoacfNx3M+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072469; c=relaxed/simple; bh=JrAOgMM2iAIABQmCU3ZV4QTcl0u9rm5SIWFUspGZWXA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N/Sce4j31eehEsIz+38L2Or5UeYOUt11ZATMd1+I/EgL4wpaa9bkoh3VTXtljDZW89uwCNVfP1n9x9KdnIL1N4OzACVS3zjVGvM9L8JcR9ME9OdoZFR4iKJ2cWh+p3OE4tSM+uK1rnNA0OOcpdtAwmQldhc8pOHTaKzhB0oRzy8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YvzCwbfb; arc=none smtp.client-ip=209.85.218.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YvzCwbfb" Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a8a7596b7dfso192652766b.0; Wed, 11 Sep 2024 09:34:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072466; x=1726677266; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2XjW/1z5M+e5L1HlSBo37JIC76JYeYPBxc+HZavMul8=; b=YvzCwbfbVbEXy1873OZRCktgLD9hwfSRVXTrkYus0CFjMRimUyLMTMtUYS5maimzFH L5tJYz0hgMnWxrfrPcxUd5ZzdVmQ1thcZkflgNaNBNcdsgd1FCFatOBA9egbKxaPAGFw +72u/7tQ1LY4lb/Hc4FNa9A0xMqRHXgrn5mIFknEwfYFh0h1m4zUnn/Hdwtu+9Fb4ZIk 51T+i3NViem8BEJwGWFNuY0WS7mRm4yvImFhIGXhVoPltbycUYtouYZt38b77qegO5bk EGwqO/cIweGpDcJoPGsl2dpo9cR4PJjWzTFKWDcnnfF3UoWTsezRXAShCZUhxjuPtpko BSJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072466; x=1726677266; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2XjW/1z5M+e5L1HlSBo37JIC76JYeYPBxc+HZavMul8=; b=BEfc41FTOfWDwKbawvo0e/zbQcVeGEUWpLh92EhBJ2IxrLfvbRkVqlcQV8P6kFz1s4 HfGC61JT9+G8N69NFdXEyM5CYbHYiXukJa9qCuufG9rzYBwDoE+T+plPZaPYrU8z+Aqb aycvhZjJcYOH2dlEjhKYsOF1rCo/UaCC4ZXb4EcWDNgigCB2Lq1DE+oFuSxcaSIV5AOl yWyTVSVNkI1SNus2TZcdyOMIyLwNs7JJKVlmmcuu0ixDKuanE+nJR9ONMRp7njaknwU+ 721jGL8y4jd1OCXmYzjulH9/uqI9YQRbKwSwWyY50xn0EVRfRLPJdydpDi/dFGtrD3a8 B8ZA== X-Forwarded-Encrypted: i=1; AJvYcCUDAn2FxJMQt6iGXmsj81HqUGElRofMEOK7wcXkRFZPPIIRGNCQV4CsxN1PO5tcjpYVX8Hbid7zFInugA==@vger.kernel.org X-Gm-Message-State: AOJu0Yxgwwg8I+zEQh4FCDXzCHP2cqPoQnDnIbt7AHtZyJNSmwabl9W8 kAJCuX64WT3xGOLWZt+HlMlxNpaEx4cEyfY9NxYq3APpWaJ23iXNe7jLpa1Q X-Google-Smtp-Source: AGHT+IFMn90AnKpQjlSSfbL+WudXYP6lMWX8+dJQnqeRgseHvIS6DUoUvcDEA12mRbA7+KIVujg4Mw== X-Received: by 2002:a17:907:94d4:b0:a8d:141a:87cb with SMTP id a640c23a62f3a-a8ffb2459a6mr490300366b.18.1726072464867; Wed, 11 Sep 2024 09:34:24 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v5 3/8] filemap: introduce filemap_invalidate_pages Date: Wed, 11 Sep 2024 17:34:39 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 kiocb_invalidate_pages() is useful for the write path, however not everything is backed by kiocb and we want to reuse the function for bio based discard implementation. Extract and and reuse a new helper called filemap_invalidate_pages(), which takes a argument indicating whether it should be non-blocking and might return -EAGAIN. Signed-off-by: Pavel Begunkov --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 17 ++++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d9c7edb6422b..e39c3a7ce33c 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -32,6 +32,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, pgoff_t start, pgoff_t end); int kiocb_invalidate_pages(struct kiocb *iocb, size_t count); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); +int filemap_invalidate_pages(struct address_space *mapping, + loff_t pos, loff_t end, bool nowait); int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/filemap.c b/mm/filemap.c index d62150418b91..6843ed4847d4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2712,14 +2712,12 @@ int kiocb_write_and_wait(struct kiocb *iocb, size_t count) } EXPORT_SYMBOL_GPL(kiocb_write_and_wait); -int kiocb_invalidate_pages(struct kiocb *iocb, size_t count) +int filemap_invalidate_pages(struct address_space *mapping, + loff_t pos, loff_t end, bool nowait) { - struct address_space *mapping = iocb->ki_filp->f_mapping; - loff_t pos = iocb->ki_pos; - loff_t end = pos + count - 1; int ret; - if (iocb->ki_flags & IOCB_NOWAIT) { + if (nowait) { /* we could block if there are any pages in the range */ if (filemap_range_has_page(mapping, pos, end)) return -EAGAIN; @@ -2738,6 +2736,15 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count) return invalidate_inode_pages2_range(mapping, pos >> PAGE_SHIFT, end >> PAGE_SHIFT); } + +int kiocb_invalidate_pages(struct kiocb *iocb, size_t count) +{ + struct address_space *mapping = iocb->ki_filp->f_mapping; + + return filemap_invalidate_pages(mapping, iocb->ki_pos, + iocb->ki_pos + count - 1, + iocb->ki_flags & IOCB_NOWAIT); +} EXPORT_SYMBOL_GPL(kiocb_invalidate_pages); /** From patchwork Wed Sep 11 16:34:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800898 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAB651AB6FE; Wed, 11 Sep 2024 16:34:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072469; cv=none; b=cTvxwtwnc+ATG1btuDK4k+ftRx8y71W67+gsbned6iuU8r7yvW75axWL6v3wEJOL67wJqv7RwEQl9bH/JxFBPbHxtNYt2EtTZp0RBLHDsO1fqpypi08TywJVohv9IY0+dOKxWfrGxHpKy3EXaE8pobvrJ3XRExzzhpfNQ6Gl1bs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072469; c=relaxed/simple; bh=NsxbMHzpxEJqp1hRxkl/Mk3cVq+Wxp/7/7/GHuqLwVE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pFPQ0AfirT+EeItxQYc6fGZOpLmih2jRKfSPztYoaiIzi9fMS4R4E2fCA/pQPU39TmhwZanYNyVvYSB7lsUlWa9ZWuL1qCB3ePbGe1aGredl0VlaTASINdbirJhAvBEzP+5BsdJEiAdux9Hy68z0bA/+mWhH46fQrFXYVLthOVE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JDMavKUa; arc=none smtp.client-ip=209.85.218.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JDMavKUa" Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a90188ae58eso3537166b.1; Wed, 11 Sep 2024 09:34:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072466; x=1726677266; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j1ilgt3VEuuXKV+jcKbpBuwPBu7e3oK2wW8bPbjcXIU=; b=JDMavKUa7qJaADOOt8aPhSlM0oUKcrYIwtbOTGRt0ldOuxib3CRYCADlx3fHLu9PoL Rvx3NHrvm/N/mlB2s30gt78ViYahUYrcu1v00Z6WdYHCjQf3974T6LC8030Ev4zL+Iz4 8wdgREWMtRSbnGz6qxeIJVY6owc5Hy1i2sJaDUTABL2LZceGrj+2aGjorIHSTDMSbhHP Bw2ThRQiFYJr6cNL3mnzKyVZJ+lRq5nOTbgdO/P2GBYvobMfauBj+RIIwsWOrplmlW+J 74eQwfgcHgQTLNtIyYinDMq3mriNNPAz68xKszAy74jz08LQ9HNpPeS/T3MfTBccNqCk 5hEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072466; x=1726677266; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j1ilgt3VEuuXKV+jcKbpBuwPBu7e3oK2wW8bPbjcXIU=; b=CXZ9/J+rBD9q7p0igSeI/mHNkakxpvKz/ehWBpxoZpVwTZD31fUdJWBxP6FnnnrKXS +ABeCgPqxZIVby7gGOGOTnTAxg7cppt8O9+fpzXwGo2rnhdspTppvq5tu6XR5WST0BE9 J1EkEh57G+zEFgiSYSxQpDVftQaRPKq/Bji55123Va9gVJOOn7wF9nGsRidEWOC6eoB1 1vH7Dly8/TWHqnVYrRX1+136scCdJb1GT95RsO3YAkoKprMmBo51+gjQKH/EK6Oo92tI /KrdR/trE49lPLqFohGC/fxoWidOhniAdFwXzCPpb3U4fTn5cPkOk77c4/YfSIGRPo9f 49OA== X-Forwarded-Encrypted: i=1; AJvYcCWlekmbnwJFXA760hdqIw3KgzLJHYYsp9G65utFEWOnypB0hCoyEpmvwaeK9BwTfPGnIo3k+iG+FC4iwA==@vger.kernel.org X-Gm-Message-State: AOJu0Yym/V1Rj5EIjNZVHWPpEsCRbaTQLGhS3LFPjFj3kacyC9iJEb2S Iw7wS2PMTn2NIrNHNIEhYtmG2m83KG+PcJl9amSEqkGQuOdDwy4lt4Hdenqf X-Google-Smtp-Source: AGHT+IFotWTb4zxEeL7XJDWd5BKSyUFpJRfVXT5ql2T0pZNPygKiYk4YlGgrsyP58zfDitbg48lEeQ== X-Received: by 2002:a17:907:d3cf:b0:a8d:2d35:3dc6 with SMTP id a640c23a62f3a-a902945f284mr8613966b.26.1726072465802; Wed, 11 Sep 2024 09:34:25 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v5 4/8] block: introduce blk_validate_byte_range() Date: Wed, 11 Sep 2024 17:34:40 +0100 Message-ID: <19a7779323c71e742a2f511e4cf49efcfd68cfd4.1726072086.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation to further changes extract a helper function out of blk_ioctl_discard() that validates if we can do IO against the given range of disk byte addresses. Signed-off-by: Pavel Begunkov Reviewed-by: Christoph Hellwig --- block/ioctl.c | 49 +++++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 18 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index e8e4a4190f18..6d663d6ae036 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -92,38 +92,51 @@ static int compat_blkpg_ioctl(struct block_device *bdev, } #endif +/* + * Check that [start, start + len) is a valid range from the block device's + * perspective, including verifying that it can be correctly translated into + * logical block addresses. + */ +static int blk_validate_byte_range(struct block_device *bdev, + uint64_t start, uint64_t len) +{ + unsigned int bs_mask = bdev_logical_block_size(bdev) - 1; + uint64_t end; + + if ((start | len) & bs_mask) + return -EINVAL; + if (!len) + return -EINVAL; + if (check_add_overflow(start, len, &end) || end > bdev_nr_bytes(bdev)) + return -EINVAL; + + return 0; +} + static int blk_ioctl_discard(struct block_device *bdev, blk_mode_t mode, unsigned long arg) { - unsigned int bs_mask = bdev_logical_block_size(bdev) - 1; - uint64_t range[2], start, len, end; + uint64_t range[2], start, len; struct bio *prev = NULL, *bio; sector_t sector, nr_sects; struct blk_plug plug; int err; - if (!(mode & BLK_OPEN_WRITE)) - return -EBADF; - - if (!bdev_max_discard_sectors(bdev)) - return -EOPNOTSUPP; - if (bdev_read_only(bdev)) - return -EPERM; - if (copy_from_user(range, (void __user *)arg, sizeof(range))) return -EFAULT; - start = range[0]; len = range[1]; - if (!len) - return -EINVAL; - if ((start | len) & bs_mask) - return -EINVAL; + if (!bdev_max_discard_sectors(bdev)) + return -EOPNOTSUPP; - if (check_add_overflow(start, len, &end) || - end > bdev_nr_bytes(bdev)) - return -EINVAL; + if (!(mode & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; filemap_invalidate_lock(bdev->bd_mapping); err = truncate_bdev_range(bdev, mode, start, start + len - 1); From patchwork Wed Sep 11 16:34:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800899 Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69E291B0131; Wed, 11 Sep 2024 16:34:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072471; cv=none; b=BqQdlk+aYQl1capBNjGq12FuYNDoBjr2c9QR0rzZBGQFN1/hw+nyH4dS4rMNzGBFgEC7707DkegQ3vioUF9ARyJ33LWMQqDB+aNbwNHVMfKVcZpVUhUdYLfIlJJ895B273DE2bZuu4nMI/2zfFtrzAE4NiKpA+GW4EwGwqVypp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072471; c=relaxed/simple; bh=ZoYFjq6YenltACEj5sQRlsTFXK59L0GNjOmH0s3i/qs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A3Fy4lBQVQyYdj+G+aiMxvxgAIV5DYP/H1TCXgaxxEMVgM/0yEnTsjGAn8Nh4pbpFc5wjI2i4+eaZN7TSmIXUGqXmdxB8d4VeUCNU4Ur7uO1QOX5KvNgxl/ZFxlhmUIm+3/B+syOoT4lTaRQPVG5/JRm7Vx7AELtaUXTyVxcXC4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nQBJAFaW; arc=none smtp.client-ip=209.85.167.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nQBJAFaW" Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-5365aec6fc1so6767050e87.3; Wed, 11 Sep 2024 09:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072467; x=1726677267; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kF1so/Smn2x5rpzm017teUbERvLKVA5uX9EtHevfvOw=; b=nQBJAFaWJ7jK4KKdEN2YQDLFuOZs6eLZBTHxOhz40EpmCuhraewiq6+PtrGrebniKj 7+PFbSfciLCly9xRJPSt59G1Ck0NBfWvNWy4fY2/qqNSz/VJfWUyWPSRs5vWkj5uCdN/ x/9NKYMlfUNZLcizlDOuaeVLc3DxFDNLFWHZvyIkaLNxfai5qQdpEmcXvtWE1TgJ/xCe 6YSlCZiEA7cXdBiuaCSEtI5di5rH6fmNO8f0OhQUAX3qkAoj7xmr7BEm//6dcqyIE6dg oPze7i32s8pxYDtYKL/HdndfbZNDKlvN/bdcYuXDg+djgbx9iHVvS64Leq9Upieod5U/ MLkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072467; x=1726677267; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kF1so/Smn2x5rpzm017teUbERvLKVA5uX9EtHevfvOw=; b=rAqbXiuqpS0odkVCZhzR1814vbSkWkSYo8Vjk/yKeg9eziAXCydO3ou+NqwLDL+cEk WZVBkhzGQEyeeAQHfgRVdCmo8Ow8n753g3pvgUcRlZlORAgEaO9UCcoT5VSEf2UX01j1 cNenq0c3EWyeHGgVtaIx0tyOF6ci21kKfqz5Hok80azxzvuGh1Yn6JxAeadHzYfhncqq HGCmmEqgWsdQg+Cxs/YY9wB1nSx+K2gJ3UhsrdGoo5MucM1s6tWI0Ws0MYGFTIhCnTo7 9fZLU0hCKvdWIsUPTZJD4EH6kUOmDM+ogz/+WKSJdbLsTNJWGkyOa1trV1LUJMUVjPFj RZTA== X-Forwarded-Encrypted: i=1; AJvYcCVJj2KsSerWBg9pE/HMndU9SCguI50PPYdw4irPYK+Lnk09YgQZIkF3YMJ9hFWV9STp7sEmrmtAiNhvIA==@vger.kernel.org X-Gm-Message-State: AOJu0Yznr5XDC7Ew2oCy8nzun6qM1gWh5ZgQaKB5LNhwQFZXkpKZatNW 42fQ5ER7dfJlcK7vjLNTr2wfoTwBuOGTIVc+xWKsdxfyyhSpq5BdLNrWCXG/ X-Google-Smtp-Source: AGHT+IHLIiqymTGBUF58wtBP7zqzsomLZiPzbl4k1nvQcQaXuT38DwgIAaWI4pNf9MD33TwjU3QnoA== X-Received: by 2002:a05:6512:68b:b0:536:53fc:e8f5 with SMTP id 2adb3069b0e04-5365880a221mr13417594e87.55.1726072467080; Wed, 11 Sep 2024 09:34:27 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:26 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , Conrad Meyer Subject: [PATCH v5 5/8] block: implement async io_uring discard cmd Date: Wed, 11 Sep 2024 17:34:41 +0100 Message-ID: <2b5210443e4fa0257934f73dfafcc18a77cd0e09.1726072086.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring allows implementing custom file specific asynchronous operations via the fops->uring_cmd callback, a.k.a. IORING_OP_URING_CMD requests or just io_uring commands. Use it to add support for async discards. Normally, it first tries to queue up bios in a non-blocking context, and if that fails, we'd retry from a blocking context by returning -EAGAIN to the core io_uring. We always get the result from bios asynchronously by setting a custom bi_end_io callback, at which point we drag the request into the task context to either reissue or complete it and post a completion to the user. Unlike ioctl(BLKDISCARD) with stronger guarantees against races, we only do a best effort attempt to invalidate page cache, and it can race with any writes and reads and leave page cache stale. It's the same kind of races we allow to direct writes. Also, apart from cases where discarding is not allowed at all, e.g. discards are not supported or the file/device is read only, the user should assume that the sector range on disk is not valid anymore, even when an error was returned to the user. Suggested-by: Conrad Meyer Signed-off-by: Pavel Begunkov --- block/blk.h | 1 + block/fops.c | 2 + block/ioctl.c | 112 ++++++++++++++++++++++++++++++++++++ include/uapi/linux/blkdev.h | 14 +++++ 4 files changed, 129 insertions(+) create mode 100644 include/uapi/linux/blkdev.h diff --git a/block/blk.h b/block/blk.h index 32f4e9f630a3..1a1a18d118f7 100644 --- a/block/blk.h +++ b/block/blk.h @@ -605,6 +605,7 @@ blk_mode_t file_to_blk_mode(struct file *file); int truncate_bdev_range(struct block_device *bdev, blk_mode_t mode, loff_t lstart, loff_t lend); long blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); +int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags); long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg); extern const struct address_space_operations def_blk_aops; diff --git a/block/fops.c b/block/fops.c index 9825c1713a49..8154b10b5abf 100644 --- a/block/fops.c +++ b/block/fops.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "blk.h" static inline struct inode *bdev_file_inode(struct file *file) @@ -873,6 +874,7 @@ const struct file_operations def_blk_fops = { .splice_read = filemap_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .uring_cmd = blkdev_uring_cmd, .fop_flags = FOP_BUFFER_RASYNC, }; diff --git a/block/ioctl.c b/block/ioctl.c index 6d663d6ae036..007f6399de66 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -11,6 +11,9 @@ #include #include #include +#include +#include +#include #include "blk.h" static int blkpg_do_ioctl(struct block_device *bdev, @@ -747,3 +750,112 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) return ret; } #endif + +struct blk_iou_cmd { + int res; + bool nowait; +}; + +static void blk_cmd_complete(struct io_uring_cmd *cmd, unsigned int issue_flags) +{ + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + + if (bic->res == -EAGAIN && bic->nowait) + io_uring_cmd_issue_blocking(cmd); + else + io_uring_cmd_done(cmd, bic->res, 0, issue_flags); +} + +static void bio_cmd_bio_end_io(struct bio *bio) +{ + struct io_uring_cmd *cmd = bio->bi_private; + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + + if (unlikely(bio->bi_status) && !bic->res) + bic->res = blk_status_to_errno(bio->bi_status); + + io_uring_cmd_do_in_task_lazy(cmd, blk_cmd_complete); + bio_put(bio); +} + +static int blkdev_cmd_discard(struct io_uring_cmd *cmd, + struct block_device *bdev, + uint64_t start, uint64_t len, bool nowait) +{ + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + gfp_t gfp = nowait ? GFP_NOWAIT : GFP_KERNEL; + sector_t sector = start >> SECTOR_SHIFT; + sector_t nr_sects = len >> SECTOR_SHIFT; + struct bio *prev = NULL, *bio; + int err; + + if (!bdev_max_discard_sectors(bdev)) + return -EOPNOTSUPP; + if (!(file_to_blk_mode(cmd->file) & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; + + err = filemap_invalidate_pages(bdev->bd_mapping, start, + start + len - 1, nowait); + if (err) + return err; + + while (true) { + bio = blk_alloc_discard_bio(bdev, §or, &nr_sects, gfp); + if (!bio) + break; + if (nowait) { + /* + * Don't allow multi-bio non-blocking submissions as + * subsequent bios may fail but we won't get a direct + * indication of that. Normally, the caller should + * retry from a blocking context. + */ + if (unlikely(nr_sects)) { + bio_put(bio); + return -EAGAIN; + } + bio->bi_opf |= REQ_NOWAIT; + } + + prev = bio_chain_and_submit(prev, bio); + } + if (unlikely(!prev)) + return -EAGAIN; + if (unlikely(nr_sects)) + bic->res = -EAGAIN; + + prev->bi_private = cmd; + prev->bi_end_io = bio_cmd_bio_end_io; + submit_bio(prev); + return -EIOCBQUEUED; +} + +int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) +{ + struct block_device *bdev = I_BDEV(cmd->file->f_mapping->host); + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + const struct io_uring_sqe *sqe = cmd->sqe; + u32 cmd_op = cmd->cmd_op; + uint64_t start, len; + + if (unlikely(sqe->ioprio || sqe->__pad1 || sqe->len || + sqe->rw_flags || sqe->file_index)) + return -EINVAL; + + bic->res = 0; + bic->nowait = issue_flags & IO_URING_F_NONBLOCK; + + start = READ_ONCE(sqe->addr); + len = READ_ONCE(sqe->addr3); + + switch (cmd_op) { + case BLOCK_URING_CMD_DISCARD: + return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); + } + return -EINVAL; +} diff --git a/include/uapi/linux/blkdev.h b/include/uapi/linux/blkdev.h new file mode 100644 index 000000000000..66373cd1a83a --- /dev/null +++ b/include/uapi/linux/blkdev.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI_LINUX_BLKDEV_H +#define _UAPI_LINUX_BLKDEV_H + +#include +#include + +/* + * io_uring block file commands, see IORING_OP_URING_CMD. + * It's a different number space from ioctl(), reuse the block's code 0x12. + */ +#define BLOCK_URING_CMD_DISCARD _IO(0x12, 0) + +#endif From patchwork Wed Sep 11 16:34:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800900 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01F581AED3F; Wed, 11 Sep 2024 16:34:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072471; cv=none; b=cmXOllqFuPDWsFvmtkMduiMmRN/Egkutm8clUTl+8/Qr6MpQ6tkGMSaKn9Q0lepUc8vDuCRc+Za0Io/7DG/ZEPxyzufznD5e786ZLTDVInvLnZQF7QWUYXYBNzHHoPMBZ4Z9Fu2BuSvbkq9KdtitrlS4/uHQIOaouzU51T/Cmmg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072471; c=relaxed/simple; bh=ikFHeLGNtER3kiAGY6dEyQFelOp7bVS4cCMwIaMUdFo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pMslUKfxkLCFr/XMAPAE/o4bIUR3k6mfepBu9OwCpEmi8wqPdORMqlPFpA0H7OKtQvOzqZIPXy+cYSAoH/qZmKO3vrrd87+tszv/mUDds3ETTtpmKHX6ClXH4e8x3eIXJGs6UXGSHlSKqT2D7BpsrI5x34NuqBNM47a7s6JMKXI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cBFxrCIo; arc=none smtp.client-ip=209.85.218.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cBFxrCIo" Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a8a789c4fc5so190251366b.0; Wed, 11 Sep 2024 09:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072468; x=1726677268; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=scYTie9WkhCZlLX/43ma4VJqtQTMJpI9jsSYXTTakWw=; b=cBFxrCIofe3HyW4VhFWV9uoo9qFFieOHGlbCVvE8bAiXh7+lme8TFYx2I+xpOFTycv Ois1phsPEJx1Wa4BYtnAC9smTSbaMuDXPR+/Ql7emVkGGbEGEdbxw530WxJeKBnp3s/8 zCT5Yo0BpLR9OQqRFnJPydcM8ZV29PKa5lCaHLbUVh2nNfOa/XVkblYPwxlqdHB3A2nB W62mbC7NNh9ey5qqSy0dC2A7HUiCjuhgFLl6J2BOSUibF66AdpZOy/kcFpC/QUH6GsYv T+mfzp+GjHRVVYnCYWEnCOcpSaGI8+2fMNd7NRz+dZcYcdKyqDVMV2z6MgZSnY4NW2Um QlUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072468; x=1726677268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=scYTie9WkhCZlLX/43ma4VJqtQTMJpI9jsSYXTTakWw=; b=uP+CqU54FyVw4FC3rpFCbnBSYgmKeqHN5KNi0dOWV0KfmGcPTKxw67pM/bI+wlx3Se be9dVNvRVLSO+7ByR+9Yv10XdG/dHB5H72q/G6iMsbKJrYpcanv5XnZekTwZLRfk9lR+ PPTU39UbOvOgh5b0rF9fGLqE9H2piwq8XeitD8HMUcoazwLfDtooppZrsI/0dEs8+5t/ 5H5Hte+faiJxgahQ7lFsRO+2e071RKMFNsuSqmkLgl0Uv78l/3p6anpK0Mqok+0W13Pd ZeEOD7FT8Xw4/L9+j+u2zBL3m+epfaJ7OvpqEhFuZJuHKHu1szAH+0Q6BjQ+CHr2Zkns oWnQ== X-Forwarded-Encrypted: i=1; AJvYcCUADBB/qCgLb1Nz1jxQpDzer5YZ/u5fPuH7DzTuV9KhzLHoa/v2wWqXllUH3VpcyyqxM1QjArENMVic4Q==@vger.kernel.org X-Gm-Message-State: AOJu0Yy9gPbXkAfG7gAWi5scalggcWfvsSqZjrVvP/44bFm2uwjd47I3 W8ntlQjimp6C4GlkzmrbOMNvvCdwp19RRKRAAGmQFnkGpo/m6DwF6Z1wpnNL X-Google-Smtp-Source: AGHT+IFtkK/R+pcDbDqSnBe+yOdtrkxtJbf96YiBq1nB7U6U1178eNxdDg00TSuprdAdoC34frzF6g== X-Received: by 2002:a17:907:944b:b0:a8a:87d5:2f49 with SMTP id a640c23a62f3a-a8ffae3a20cmr485726666b.28.1726072468161; Wed, 11 Sep 2024 09:34:28 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:27 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , Conrad Meyer Subject: [PATCH v5 6/8] block: implement write zeroes io_uring cmd Date: Wed, 11 Sep 2024 17:34:42 +0100 Message-ID: <8e7975e44504d8371d716167face2bc8e248f7a4.1726072086.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a second io_uring cmd for block layer implementing asynchronous write zeroes. It reuses helpers we've added for async discards, and inherits the code structure as well as all considerations in regards to page cache races. It has to be supported by underlying hardware to be used, otherwise the request will fail. A fallback version is implemented separately in a later patch. Suggested-by: Conrad Meyer Signed-off-by: Pavel Begunkov Nacked-by: Christoph Hellwig --- block/ioctl.c | 66 +++++++++++++++++++++++++++++++++++++ include/uapi/linux/blkdev.h | 1 + 2 files changed, 67 insertions(+) diff --git a/block/ioctl.c b/block/ioctl.c index 007f6399de66..f8f6d6a71ff0 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -778,6 +778,69 @@ static void bio_cmd_bio_end_io(struct bio *bio) bio_put(bio); } +static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, + struct block_device *bdev, + uint64_t start, uint64_t len, bool nowait) +{ + struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + sector_t bs_mask = (bdev_logical_block_size(bdev) >> SECTOR_SHIFT) - 1; + sector_t limit = bdev_write_zeroes_sectors(bdev); + sector_t sector = start >> SECTOR_SHIFT; + sector_t nr_sects = len >> SECTOR_SHIFT; + struct bio *prev = NULL, *bio; + gfp_t gfp = nowait ? GFP_NOWAIT : GFP_KERNEL; + int err; + + if (!(file_to_blk_mode(cmd->file) & BLK_OPEN_WRITE)) + return -EBADF; + if (bdev_read_only(bdev)) + return -EPERM; + err = blk_validate_byte_range(bdev, start, len); + if (err) + return err; + + if (!limit) + return -EOPNOTSUPP; + /* + * Don't allow multi-bio non-blocking submissions as subsequent bios + * may fail but we won't get a direct indication of that. Normally, + * the caller should retry from a blocking context. + */ + if (nowait && nr_sects > limit) + return -EAGAIN; + + err = filemap_invalidate_pages(bdev->bd_mapping, start, + start + len - 1, nowait); + if (err) + return err; + + limit = min(limit, (UINT_MAX >> SECTOR_SHIFT) & ~bs_mask); + while (nr_sects) { + sector_t bio_sects = min(nr_sects, limit); + + bio = bio_alloc(bdev, 0, REQ_OP_WRITE_ZEROES|REQ_NOUNMAP, gfp); + if (!bio) + break; + if (nowait) + bio->bi_opf |= REQ_NOWAIT; + bio->bi_iter.bi_sector = sector; + bio->bi_iter.bi_size = bio_sects << SECTOR_SHIFT; + sector += bio_sects; + nr_sects -= bio_sects; + + prev = bio_chain_and_submit(prev, bio); + } + if (unlikely(!prev)) + return -EAGAIN; + if (unlikely(nr_sects)) + bic->res = -EAGAIN; + + prev->bi_private = cmd; + prev->bi_end_io = bio_cmd_bio_end_io; + submit_bio(prev); + return -EIOCBQUEUED; +} + static int blkdev_cmd_discard(struct io_uring_cmd *cmd, struct block_device *bdev, uint64_t start, uint64_t len, bool nowait) @@ -856,6 +919,9 @@ int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) switch (cmd_op) { case BLOCK_URING_CMD_DISCARD: return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); + case BLOCK_URING_CMD_WRITE_ZEROES: + return blkdev_cmd_write_zeroes(cmd, bdev, start, len, + bic->nowait); } return -EINVAL; } diff --git a/include/uapi/linux/blkdev.h b/include/uapi/linux/blkdev.h index 66373cd1a83a..b4664139a82a 100644 --- a/include/uapi/linux/blkdev.h +++ b/include/uapi/linux/blkdev.h @@ -10,5 +10,6 @@ * It's a different number space from ioctl(), reuse the block's code 0x12. */ #define BLOCK_URING_CMD_DISCARD _IO(0x12, 0) +#define BLOCK_URING_CMD_WRITE_ZEROES _IO(0x12, 1) #endif From patchwork Wed Sep 11 16:34:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800902 Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CCBB1A3AB8; Wed, 11 Sep 2024 16:34:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072474; cv=none; b=sxdgVoYN5mZ4dqv9k+DADIlNrhdkAxjm0m3WB2CXksTMrUXQ+6jDgxbjMghUH+Qg1+VAuA6XjO/bkA5AMayk0tLbSCDbNxbmmcelQUk1uNsQQAgdbazHHxINxJL5hj0gHLI7x97r7/cVVm2cpGbB/IBvRMOU+vLugn61LJRNb4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072474; c=relaxed/simple; bh=sSEr2oV7JZ8OWOfM36VBUS5CdQbzuCCnTxq3sxwf8Kw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O3IIr8WEqfuvRs5Wr8EdB4Td2Pa+HbBLIT5NQYSf/zWfIr22vJfygTQ9gv4/NmZqarqdu2HlVvdoWSiua4ChdYPXdyNT8viBqQwzuR0dM0XiEnpCk9YSb3pd3b2pMKRcpAyyjqaQA8kx1xBeZv43HjrHwMsUbIRu7LeqeRseyz4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TJyR4zEj; arc=none smtp.client-ip=209.85.208.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TJyR4zEj" Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-5c26815e174so7509492a12.0; Wed, 11 Sep 2024 09:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072469; x=1726677269; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Jv/xRgK93eDswhXa+H3bNlv587Pzbd+1FrWwfpAcjww=; b=TJyR4zEjQWywhOCIuO/HN7pRet4zeDHUI8BfrFcd/84vJ/84EGkeOtF5pZIk9QsbsR rftggtp3rsrba4fm7q45HqjA6F+rdaYwbA13BVTMxXWiBJmkvpLUim+qcA9jJFf8s3b/ 8JFGE9iIJweKIoN4DSA/B0tqyVxQEo/5cEy456cKYPP7krCRUT+XhVXeoPyCv084A23Y NDvE04c0SdvVkHNAjfBJ2rzMIq2A3YezheiFIy1q6+NaW6zjSFONHA+LtqDqGVATte/o sVaJgJ2sXt42WocF+zsLM1HomFmApVhA7z1OFQr+5BItv9WGl8FnbYsFa9d9NLlNirY2 tFog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072469; x=1726677269; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jv/xRgK93eDswhXa+H3bNlv587Pzbd+1FrWwfpAcjww=; b=owNOCgPNUOsoiiyVvezI2LG7fOGlDt9mxLeiZyx2ljg0KUe9trUEI/wy3L0G7ja7bh hiFZgqytugjsIPmAGg+9chxVxWdkBtg310XnrOEPGOcjinqQBmHzTzL9HQkecANCuT1U qyT82sa6o0K39XaI/sltJkrwcRKPoSoLKoNEOkFpWAjSFBO7RLrFdayptXJg+r5BI58S tnFawOArR8QeZ7LpnnEJ+RwuU94+1iXwY1r3xBsAeI4HY1SqUCyt1r+dpFsIStqYlWdD PUVaW4WqcR1q41Oav34nwvUrB5sx51QFPzLhyxUcSBrf5ucbJa2wLmBWdOyOAFrxu6JV ntKg== X-Forwarded-Encrypted: i=1; AJvYcCXni89A6+lhFg1pxMcm6rZ74id2ET/aafyOJ2xduzocq1wZVGjPUjlC0KIOC1+pHh119a5aWVyDXdlzIQ==@vger.kernel.org X-Gm-Message-State: AOJu0YzyKVu37reQiaxNM+BNGnHENgEpRT24kFXPtf41zEx8g0jfGugv 8LkXUQsK/7M1UnVnGt0UprdqHrSNHrWmU9B2aPbDCj7/HMlaczKYATjerO0K X-Google-Smtp-Source: AGHT+IGuiTLA2mNX0YCzkTRNyEFx9mbWso8US7cNHDcH1lN98BWCM89d0c663QhibBxvnr/nZH5TzQ== X-Received: by 2002:a17:907:f186:b0:a86:f960:411d with SMTP id a640c23a62f3a-a9029409aedmr9838766b.2.1726072469422; Wed, 11 Sep 2024 09:34:29 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:28 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v5 7/8] block: add nowait flag for __blkdev_issue_zero_pages Date: Wed, 11 Sep 2024 17:34:43 +0100 Message-ID: <0a4e17bf70d5fa035329664b25a780bf4f53e38f.1726072086.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To reuse __blkdev_issue_zero_pages() in the following patch, we need to make it work with non-blocking requests. Add a new nowait flag we can pass inside. Return errors if something went wrong, and check bio_alloc() for failures, which wasn't supposed to happen before because of what gfp flags the callers are passing. Note that there might be a bio passed back even when the function returned an error. To limit the scope of the patch, don't add return code handling to callers, that can be deferred to a follow up. Signed-off-by: Pavel Begunkov --- block/blk-lib.c | 22 ++++++++++++++++++---- include/linux/bio.h | 4 ++++ include/linux/blkdev.h | 1 + 3 files changed, 23 insertions(+), 4 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index 83eb7761c2bf..40bb59f583ee 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -192,20 +192,32 @@ static unsigned int __blkdev_sectors_to_bio_pages(sector_t nr_sects) return min(pages, (sector_t)BIO_MAX_VECS); } -static void __blkdev_issue_zero_pages(struct block_device *bdev, +int blkdev_issue_zero_pages_bio(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned int flags) { + blk_opf_t opf = REQ_OP_WRITE; + + if (flags & BLKDEV_ZERO_PAGES_NOWAIT) { + sector_t max_bio_sectors = BIO_MAX_VECS << PAGE_SECTORS_SHIFT; + + if (nr_sects > max_bio_sectors) + return -EAGAIN; + opf |= REQ_NOWAIT; + } + while (nr_sects) { unsigned int nr_vecs = __blkdev_sectors_to_bio_pages(nr_sects); struct bio *bio; bio = bio_alloc(bdev, nr_vecs, REQ_OP_WRITE, gfp_mask); + if (!bio) + return -ENOMEM; bio->bi_iter.bi_sector = sector; if ((flags & BLKDEV_ZERO_KILLABLE) && fatal_signal_pending(current)) - break; + return -EINTR; do { unsigned int len, added; @@ -222,6 +234,8 @@ static void __blkdev_issue_zero_pages(struct block_device *bdev, *biop = bio_chain_and_submit(*biop, bio); cond_resched(); } + + return 0; } static int blkdev_issue_zero_pages(struct block_device *bdev, sector_t sector, @@ -235,7 +249,7 @@ static int blkdev_issue_zero_pages(struct block_device *bdev, sector_t sector, return -EOPNOTSUPP; blk_start_plug(&plug); - __blkdev_issue_zero_pages(bdev, sector, nr_sects, gfp, &bio, flags); + blkdev_issue_zero_pages_bio(bdev, sector, nr_sects, gfp, &bio, flags); if (bio) { if ((flags & BLKDEV_ZERO_KILLABLE) && fatal_signal_pending(current)) { @@ -285,7 +299,7 @@ int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, } else { if (flags & BLKDEV_ZERO_NOFALLBACK) return -EOPNOTSUPP; - __blkdev_issue_zero_pages(bdev, sector, nr_sects, gfp_mask, + blkdev_issue_zero_pages_bio(bdev, sector, nr_sects, gfp_mask, biop, flags); } return 0; diff --git a/include/linux/bio.h b/include/linux/bio.h index faceadb040f9..e4e495a4a95b 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -684,4 +684,8 @@ struct bio *bio_chain_and_submit(struct bio *prev, struct bio *new); struct bio *blk_alloc_discard_bio(struct block_device *bdev, sector_t *sector, sector_t *nr_sects, gfp_t gfp_mask); +int blkdev_issue_zero_pages_bio(struct block_device *bdev, + sector_t sector, sector_t nr_sects, gfp_t gfp_mask, + struct bio **biop, unsigned int flags); + #endif /* __LINUX_BIO_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 643c9020a35a..bf1aa951fda2 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1098,6 +1098,7 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ #define BLKDEV_ZERO_KILLABLE (1 << 2) /* interruptible by fatal signals */ +#define BLKDEV_ZERO_PAGES_NOWAIT (1 << 3) /* non-blocking submission */ extern int __blkdev_issue_zeroout(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, From patchwork Wed Sep 11 16:34:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13800901 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BEFD1AED3F; Wed, 11 Sep 2024 16:34:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072474; cv=none; b=irtkP1vmZZRm8PnDLO5DAIlgHXlxE3dk14tQOwWdeNx7A64e+fYa2sHHeHlTx5juoztsBtDgmJHnsBBJZXW/uNkTHciMP3uIVEGQNTbRrDVJNKEIZg69rLBZwOKLx1CBpWn1nmO5/BTicjkK6AkfgwyI+4MO7YOGAIJcRdExlCw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1726072474; c=relaxed/simple; bh=47FgsBjGylI50GriBm4QB0MtYvxzC2qf9DBpWjJOkco=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P7jJ7HZPLk+P1aaHuicK5GtAVjVvDgwCucr+RS14hqnbCkT22dyQOqIyWva9KxSliLpiyRqnpKvREuHUnuwpffFp3XA95LtN+vwJtFJtA7t5Zi54quvj7HT92trIptKFQiqSh013u2nsk1Qg3G8Gb/vATmmzq++1fw342ay4q10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=doF9fpPl; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="doF9fpPl" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a90188ae58eso3549366b.1; Wed, 11 Sep 2024 09:34:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726072470; x=1726677270; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tt1Sdp5HcYzfSkBqCxePJrZcv+DNb0SwUWNG181PEJo=; b=doF9fpPl6rDmTUVmE9TvTW8qnbFTiSAt6zz66lQQrYWEHFx0NW5lmBe8uo6GUiZgX2 T7+nACbZnDwlJMPWzEwrbs2SIwpPe/hFhSb+cXqZQJaYP4Pnjv91CCrzr9+aw+WrY9Tg xLVUbwu/mVdqwuxBKlk+kKWwmpwjq+wU82dRZ3hUiEH67rc6iUbQETXHDoeNrMeDhgK1 rTvvo79lh1C613ZMRkDfvwwmLW9dJNJ6xI3lgzsYqed2O/5hL2ZyJh3Wr9ecyrakQ5db J2zsmwHJBIW/bz3z0Qt7Kg+ybHpIIyQv8kM0wmiFU0Yh23b5Aboeb5jMml6PqNKbV31S BFXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726072470; x=1726677270; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tt1Sdp5HcYzfSkBqCxePJrZcv+DNb0SwUWNG181PEJo=; b=aZiDFNLaOBYkfQrvrXYSo7qvdTkrCA5EY9AYg+tr0SNN0k2Sb8Vv/NudpwXg/1X9tf n4lx4GhF65aCERa39XngyO31QM98qbptJhT1sqs29684ec3JlpXzt+OKvK5EZ4Qy2Scn 3COlwAI9FdE47C2SxLNg+s+XlKLIHZnDor+zZCAwG9EM9DFaJcPBd5UbEriOC90cy/6C ldd7wN8YIGIcu27lrwGe/mKnSCBuoZvdoTSR+dvWTERf+s0hYXs8a1nFW95u7qK2cJ7w EFQiihxjUfWTEo9/J4t9POIgCDNOE25pTZRXEOQbAsTp5RBMmo5m30r/FgsWUkWwdpEt TB1Q== X-Forwarded-Encrypted: i=1; AJvYcCUD6iSR3Qo/JwKD2kJ6vP0KzX2WbZp7rpk3D5tmjjSFSmtqf9nLZT9OtDE5s+RKtqZSu+YKGLkgyyIEsQ==@vger.kernel.org X-Gm-Message-State: AOJu0Yx0MC5OevPaMomMUY4iwT56Qc9p8ru11ls9AVM1KFm3HN4Ua2/9 XiyTiHm9PDLQ8pSmX3pXGkTBHkug7HcKmY3HZWhxwD/66UjIvFKv2s7BIYVt X-Google-Smtp-Source: AGHT+IGzBI9+dMWJvPlTYbOfk11QQ+FrGskd3fYscDMuABz9+Q22m/SZtfuiKraKUWCJKvHI3YqGqw== X-Received: by 2002:a17:907:d2ce:b0:a8d:3f6a:99ce with SMTP id a640c23a62f3a-a902961a769mr5471566b.49.1726072470444; Wed, 11 Sep 2024 09:34:30 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8d25c72ed3sm631820866b.135.2024.09.11.09.34.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Sep 2024 09:34:29 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-block@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig Subject: [PATCH v5 8/8] block: implement write zero pages cmd Date: Wed, 11 Sep 2024 17:34:44 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a command that writes the zero page to the drive. Apart from passing the zero page instead of actual data it uses the normal write path and doesn't do any further acceleration. Subsequently, it doesn't requires any special hardware support, and for example can be used as a fallback option for BLOCK_URING_CMD_WRITE_ZEROES. Signed-off-by: Pavel Begunkov --- block/ioctl.c | 21 ++++++++++++++++++--- include/uapi/linux/blkdev.h | 1 + 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index f8f6d6a71ff0..08e0d2d52b06 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -780,7 +780,8 @@ static void bio_cmd_bio_end_io(struct bio *bio) static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, struct block_device *bdev, - uint64_t start, uint64_t len, bool nowait) + uint64_t start, uint64_t len, + bool nowait, bool zero_pages) { struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); sector_t bs_mask = (bdev_logical_block_size(bdev) >> SECTOR_SHIFT) - 1; @@ -799,6 +800,17 @@ static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, if (err) return err; + if (zero_pages) { + err = blkdev_issue_zero_pages_bio(bdev, sector, nr_sects, + gfp, &prev, + BLKDEV_ZERO_PAGES_NOWAIT); + if (!prev) + return -EAGAIN; + if (err) + bic->res = err; + goto out_submit; + } + if (!limit) return -EOPNOTSUPP; /* @@ -834,7 +846,7 @@ static int blkdev_cmd_write_zeroes(struct io_uring_cmd *cmd, return -EAGAIN; if (unlikely(nr_sects)) bic->res = -EAGAIN; - +out_submit: prev->bi_private = cmd; prev->bi_end_io = bio_cmd_bio_end_io; submit_bio(prev); @@ -921,7 +933,10 @@ int blkdev_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) return blkdev_cmd_discard(cmd, bdev, start, len, bic->nowait); case BLOCK_URING_CMD_WRITE_ZEROES: return blkdev_cmd_write_zeroes(cmd, bdev, start, len, - bic->nowait); + bic->nowait, false); + case BLOCK_URING_CMD_WRITE_ZERO_PAGE: + return blkdev_cmd_write_zeroes(cmd, bdev, start, len, + bic->nowait, true); } return -EINVAL; } diff --git a/include/uapi/linux/blkdev.h b/include/uapi/linux/blkdev.h index b4664139a82a..880b292d2d0d 100644 --- a/include/uapi/linux/blkdev.h +++ b/include/uapi/linux/blkdev.h @@ -11,5 +11,6 @@ */ #define BLOCK_URING_CMD_DISCARD _IO(0x12, 0) #define BLOCK_URING_CMD_WRITE_ZEROES _IO(0x12, 1) +#define BLOCK_URING_CMD_WRITE_ZERO_PAGE _IO(0x12, 2) #endif