From patchwork Mon Mar 18 00:41:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594897 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B7A918029; Mon, 18 Mar 2024 00:43:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722621; cv=none; b=VBXHkd67zyd3I7kKjFoD0oN3gEI1mXb2GgPuFuk4arwvXS/xZgAfAan+mkF+7mw6HAJpNN/yV5SoleZaWlZXnbl037HiAdvVzDz9emzO0d7a/deRG+H8S5TUjTkl42AAbd9Bq7JTQeyyiD3Mw18sUDtP5zo+UMnG03QIroESBrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722621; c=relaxed/simple; bh=/izrqIB3O3T0N3fMc9YK/lxjvDLgUtp/Le9kuZyzI+Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mmaVwHCCFY9VXqRmXuwNSf9PwYNPm3Hb34tAr6oPtkXizAb9YO6NH4fyLT1QDr8oNPch8JIlQSsC2ckys4GlkzE8r1Ueo6gCSOpayzXPuQoeff7DSiA+TkXTPelT5nprhn11C0qp+5s1ahq1Dzac4O3tM8R+PoziiVsIZH4L+tU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eVBTRMXf; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eVBTRMXf" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-56845954ffeso5009741a12.2; Sun, 17 Mar 2024 17:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722617; x=1711327417; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+iTfQQgM7iNJagOFnCh9386l4X2lTE2ddUvAFovvm+E=; b=eVBTRMXfE7/A4ViS+Vfs7it2vg0IdP0GPY7cfq876BivZ9983y2YbxIbtvO/Rp7Qxl Pnn3R/POTNHG3WMW7+zRjTVMVYeucNrZp4yxm3U6MDUBPeEoMh2WG3N4Ouy5ojFA6bw5 KRdPjgp3pE5/tVRjvwo/OD7BlE5Q9czQd1R5HDHLH2upAgB6fSgsUYhquz0Qh3xlclfB Dskqr17u9JN4RQwprW6kIfOwT+bAtn6WJI2gcbpApJHUov6UZRzlEn5RTJe+zqnBmSXN f3eua8PTVxq2NoFsw57Vb1p+1a8SYndWuC2nQVxzKvmWcZZajkCuGuFb1jxCXzYtctig 9VvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722617; x=1711327417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+iTfQQgM7iNJagOFnCh9386l4X2lTE2ddUvAFovvm+E=; b=qRS5ZDQvBSTNfIyJLYFvP+vfshH+oMQM6XMqPwZ8uZrJbrio6Go1EANgJ1jw51UX/s NLbv1DOW6Sr+8BxDSG1mxS13/6YKm4C5dc+3cuQPcfcx4K8ra8w267gUgqqSaG/xQbAz UX45go0rEDZ7XTreuP9jUMcQfxa4ys75NgMdG/tycIiRBuJEwh/pSTwcUutKDKn2Ky28 Jg6l4RXzTO74wikaOZGp9j9/MLheL0A4Zmxzo76KXZ6RpOu3vMSA7j4v7viiUPaMFdnJ D0zpIkRAb5gzfEF0WvSSj7jyAYGXakZA1jp1AUwhgbW1QdX4wfUo+1fESle05gxO2/LD Ajcg== X-Gm-Message-State: AOJu0YxZpm03DEpCUVI6EAXmEJneE1Ia7+WkhOGWrWVXrHlWc1HPnP/j 9ZGgFjxVh/85076bK4lFA/RiP8xXYrVT/6EZunbPAg93EYDtzJA2Z2OzJGhq X-Google-Smtp-Source: AGHT+IHkthDaEhoJicRBACBhl7XjesZXCE++CiBA6ndX6onTbaeEYp16qrkP/9z1K0kg+1JYjkV9zw== X-Received: by 2002:a05:6402:1f0f:b0:568:fb58:bc4b with SMTP id b15-20020a0564021f0f00b00568fb58bc4bmr1216006edb.2.1710722617366; Sun, 17 Mar 2024 17:43:37 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 01/14] io_uring/cmd: kill one issue_flags to tw conversion Date: Mon, 18 Mar 2024 00:41:46 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring cmd converts struct io_tw_state to issue_flags and later back to io_tw_state, it's awfully ill-fated, not to mention that intermediate issue_flags state is not correct. Get rid of the last conversion, drag through tw everything that came with IO_URING_F_UNLOCKED, and replace io_req_complete_defer() with a direct call to io_req_complete_defer(), at least for the time being. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/7f0d5ddfb5335d038bfd8db50656a1d69daed37f.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/uring_cmd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 42f63adfa54a..f197e8c22965 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -100,11 +100,11 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); + } else if (!(issue_flags & IO_URING_F_UNLOCKED)) { + io_req_complete_defer(req); } else { - struct io_tw_state ts = { - .locked = !(issue_flags & IO_URING_F_UNLOCKED), - }; - io_req_task_complete(req, &ts); + req->io_task_work.func = io_req_task_complete; + io_req_task_work_add(req); } } EXPORT_SYMBOL_GPL(io_uring_cmd_done); From patchwork Mon Mar 18 00:41:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594898 Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com [209.85.208.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E6491CFAD; Mon, 18 Mar 2024 00:43:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722623; cv=none; b=eNLGS/FL4gTNrJpi/MVmbT/LSGeLsJAK587jcMvCSoQ+kw+pME3RHdIlNhJ/IqLrnPBK9YmhGPcqPbu1o7jZWRXc4JNv7jj1xDIHEugp20vZviKxPoJMk4L4YM5odgWhMvAwhw0pdvVDSRCx9CLGJL+05JWyIdyVZ6/jSJIrR/Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722623; c=relaxed/simple; bh=YMGi35BAzF26VICJT9FSWmE/3j00VQwimSvy4GbKuR4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rYN06dMn/ultWFqx+C3f82LFNOvCuwoGZj4whZHmXkIXOGWin+LRcbO+TVUTGNiEtjsvTHEiOSOdb88LitSbbYKfHaRWBxXT7Qzyr0tT+AG5AqNYxONqilXYq/vVcM/V/gptLGEFSgYyTsBRjARJDlMTvOtNSJ95xUWE+DwVshc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=kr+kJSgH; arc=none smtp.client-ip=209.85.208.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kr+kJSgH" Received: by mail-lj1-f173.google.com with SMTP id 38308e7fff4ca-2d2509c66daso55018531fa.3; Sun, 17 Mar 2024 17:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722619; x=1711327419; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H9WedqTxnnip3tXkbNwAUX++IyECvsw3O+o4OwI+ASU=; b=kr+kJSgHo+XbggDqYo6DTxIUSYDGfOdIkzV6rzO9kzFnxWb1YMcvyvvInOixCOuczI VTucclpIgEw+uMzA09MH71udph002N50l2bQ7+o3228pOCq4iL3fTxR5KG+9I8V9D+gd GIu86wkPck2BWTuPjMnUoymVZLzOW1mCrnIwlykEVu7V5fwwCIxu9kj9ACUG8UrS6lQ7 C2gCUcm09TWQa474m/v3jkv/wFrXFljKKdUOi7xxtVyD8cF6tcNtI21aPoMMJYTA32H4 0G70LAj1EmsZCTh6NLr1hC1EIVdVlBtURV4dzRF3ZCA6x50KNuIQtaf0JFk+wLnHnJzg KATw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722619; x=1711327419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H9WedqTxnnip3tXkbNwAUX++IyECvsw3O+o4OwI+ASU=; b=MlaOO21HFFgT6+fq13q0KpqUdJncgcnZbSzN6f3NSrnjgU2yQOvawrkBuaizQ/vpnR FkU5ghx5z0mZJ6UtJ7IuXMjAnlB16VtDUAIOz9O/AO/Z4JPIRMwk5ZihAId+HcCFVzYR H5ITG4rRKXCPViNuxHa6mTHJzawTInZO2VOwG6TfLTlW/JALwAmzFogsGm7YzkiEJ77+ uSVNtYAulACwHVrSXNqSLlja5pb9uLlJj1NtviY4SbxFIZ64OFjmi/dAH3UB9X78QXKc VHT4607IChGFDCr26OLD2C/31i4EU0zPOOnoccHwKOJleTG7aq3EqyCZXhps4CD3rYMf 9K4w== X-Gm-Message-State: AOJu0YygU72KkPFjRyz8qyTRBKjg5eOdiln2TK4GRdo9PJMzMFEYLSu2 pZbepmRcE7VS2FKUiKqdYiiqp6G0NgccHrM3FMmjAaThExwuIzwKj3gh9n7G X-Google-Smtp-Source: AGHT+IHWpDHvih4EBQ8dEG8gLQiAomXc95wFNR0A/ArxIHisTUp/e/imKmv+M9ZS9ryIuhL/SHxEKA== X-Received: by 2002:a2e:9396:0:b0:2d3:ffa0:8782 with SMTP id g22-20020a2e9396000000b002d3ffa08782mr6736034ljh.43.1710722618482; Sun, 17 Mar 2024 17:43:38 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 02/14] io_uring/cmd: fix tw <-> issue_flags conversion Date: Mon, 18 Mar 2024 00:41:47 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 !IO_URING_F_UNLOCKED does not translate to availability of the deferred completion infra, IO_URING_F_COMPLETE_DEFER does, that what we should pass and look for to use io_req_complete_defer() and other variants. Luckily, it's not a real problem as two wrongs actually made it right, at least as far as io_uring_cmd_work() goes. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/eb08e72e837106963bc7bc7dccfd93d646cc7f36.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/uring_cmd.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index f197e8c22965..ec38a8d4836d 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -56,7 +56,11 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; + unsigned issue_flags = IO_URING_F_UNLOCKED; + + /* locked task_work executor checks the deffered list completion */ + if (ts->locked) + issue_flags = IO_URING_F_COMPLETE_DEFER; ioucmd->task_work_cb(ioucmd, issue_flags); } @@ -100,7 +104,9 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); - } else if (!(issue_flags & IO_URING_F_UNLOCKED)) { + } else if (issue_flags & IO_URING_F_COMPLETE_DEFER) { + if (WARN_ON_ONCE(issue_flags & IO_URING_F_UNLOCKED)) + return; io_req_complete_defer(req); } else { req->io_task_work.func = io_req_task_complete; From patchwork Mon Mar 18 00:41:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594899 Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20C061DA58; Mon, 18 Mar 2024 00:43:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722623; cv=none; b=DJLYl7sSPjvcLpHCeNuP6afnVdz/IeXZrB2ikrX8LeRjrpySVYx21bckpuVa40CSzH7fNV2SkFgflFkk1qAHksNJOEhEfAQ5HDVfQnVeB529/xEYXKbp3nysr9c8Ta7hYSXXjYCFIYeKJOvVKVsnVRJeVe3H+kcb/U3a8VoZLyo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722623; c=relaxed/simple; bh=T/VCekeEwmZ+62RASLtLVFssRRVNurSPBNec8raRGrA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QylSmGAyfqZZEOX47CXeh5lvXrehmqtnH/yLeHyiBddMDYN2iG5+JI8veQGvUoQk63DDCmkE6b0KfuOS/Xjr+dumhEAqqkMcymQVn3EIZ7MY5MG4LDiM3VOXn9iyfGBmFFzt1XSN+vZ/4rMt9SEFZNOCoEb3MlxO8+aTsJ1f8dE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZeYZIBo/; arc=none smtp.client-ip=209.85.208.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZeYZIBo/" Received: by mail-lj1-f169.google.com with SMTP id 38308e7fff4ca-2d49f7e5c2cso15010451fa.2; Sun, 17 Mar 2024 17:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722619; x=1711327419; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s2m+aoIuQvYdyvrDDwOXPH4Abav6/EJ7hVeNVD0XZuQ=; b=ZeYZIBo/E7Co25iZrfj9fLLF4S5CqmA8Fz79zDaeCBPinFO/SNGjYNvJkj6dPQ8ioH ZvcFQMkiLXTaKmmHK6+owsMWKLclaRcevulEWuziDjjEe6m5FWspytr/KjCOrROd7Yt+ dmILbUniUJR5vZjThq5fVGd6l7gF/22srDOUlQ1cQZ0uofiFZe1Hq0ub3NCjjoL6Sbjd jJB20S4arxKSPZCjuMZxVy/MUBkWAwmv7q5ayzmNWotP6WYNeXvmNXRfJ7ojkvqbAT+w LeOMpnFmUbiJNYXBLdjJSIrk2AOEAnFPgV09I6PY6MCZS1Ja22uwmR2jJ17mWSNSjfqF XcfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722619; x=1711327419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s2m+aoIuQvYdyvrDDwOXPH4Abav6/EJ7hVeNVD0XZuQ=; b=lXq7+Qro/dBAazoKxDtgzQvgPrOgAJ+uzplaG9L8DIiP7zAe4Zzg7k5n2zDbAWcAD2 +HWEOFk8kg69AryY1/ceAvvqrPGEOHsq/dizj5lq9dXCVh9VnleLJfjtiNkyk5CjNAsg d15vWPabWhYFoioptv/hWNJ55v0vZjQnkQG1hEF5noOOHxu70I5MPC6yNAgXpVK8EcbS /T15GpbPek4kFzXeJWoQIydM8gflQRHH/Mmte+2H5A2fMgia2FLX0CuldYTO3B89HZuX Oq7YSvkzViiaba2S9AXAExwF9cWDdYn0xwR1vviLyj2zyYtmC/BA4CB3ujqtJkJyD/i2 y+zg== X-Gm-Message-State: AOJu0Yydem/MwEgkDlA1q1lJ0foC3MYV/C+u4UyvwDZbxNZdh/ze0KQG ud5GSafw4jxQ9mAF6DKRFatAdOlqV9m839/f78wcAgHKkQasq2xNOogzVRR4 X-Google-Smtp-Source: AGHT+IGehzGNRYp3rih/rwWEnYfCQPZojaX2X86xL3RX2J5HAleJS8jrkmATZdPItFfNMTV9B3mbSg== X-Received: by 2002:a2e:9001:0:b0:2d4:7455:89f4 with SMTP id h1-20020a2e9001000000b002d4745589f4mr5780291ljg.2.1710722619406; Sun, 17 Mar 2024 17:43:39 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 03/14] io_uring/cmd: make io_uring_cmd_done irq safe Date: Mon, 18 Mar 2024 00:41:48 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring_cmd_done() is called from the irq context and is considered to be irq safe, however that's not the case if the driver requires cancellations because io_uring_cmd_del_cancelable() could try to take the uring_lock mutex. Clean up the confusion, by deferring cancellation handling to locked task_work if we came into io_uring_cmd_done() from iowq or other IO_URING_F_UNLOCKED path. Signed-off-by: Pavel Begunkov --- io_uring/uring_cmd.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index ec38a8d4836d..9590081feb2d 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -14,19 +14,18 @@ #include "rsrc.h" #include "uring_cmd.h" -static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd) { struct io_kiocb *req = cmd_to_io_kiocb(cmd); struct io_ring_ctx *ctx = req->ctx; + lockdep_assert_held(&ctx->uring_lock); + if (!(cmd->flags & IORING_URING_CMD_CANCELABLE)) return; cmd->flags &= ~IORING_URING_CMD_CANCELABLE; - io_ring_submit_lock(ctx, issue_flags); hlist_del(&req->hash_node); - io_ring_submit_unlock(ctx, issue_flags); } /* @@ -44,6 +43,9 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, struct io_kiocb *req = cmd_to_io_kiocb(cmd); struct io_ring_ctx *ctx = req->ctx; + if (WARN_ON_ONCE(ctx->flags & IORING_SETUP_IOPOLL)) + return; + if (!(cmd->flags & IORING_URING_CMD_CANCELABLE)) { cmd->flags |= IORING_URING_CMD_CANCELABLE; io_ring_submit_lock(ctx, issue_flags); @@ -84,6 +86,15 @@ static inline void io_req_set_cqe32_extra(struct io_kiocb *req, req->big_cqe.extra2 = extra2; } +static void io_req_cmd_task_complete(struct io_kiocb *req, struct io_tw_state *ts) +{ + struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + + io_tw_lock(req->ctx, ts); + io_uring_cmd_del_cancelable(ioucmd); + io_req_task_complete(req, ts); +} + /* * Called by consumers of io_uring_cmd, if they originally returned * -EIOCBQUEUED upon receiving the command. @@ -93,8 +104,6 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, { struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); - io_uring_cmd_del_cancelable(ioucmd, issue_flags); - if (ret < 0) req_set_fail(req); @@ -107,9 +116,10 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, } else if (issue_flags & IO_URING_F_COMPLETE_DEFER) { if (WARN_ON_ONCE(issue_flags & IO_URING_F_UNLOCKED)) return; + io_uring_cmd_del_cancelable(ioucmd); io_req_complete_defer(req); } else { - req->io_task_work.func = io_req_task_complete; + req->io_task_work.func = io_req_cmd_task_complete; io_req_task_work_add(req); } } From patchwork Mon Mar 18 00:41:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594900 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 816191DFC7; Mon, 18 Mar 2024 00:43:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722625; cv=none; b=ehkhc1b2V4G7WK7m07OJl47/cqkEwroPxiCVDwVCgv+W0gSNp4/aiIBdd5utBZqhK1XAHv5cVU5lSpmLejgRtOmHIQ3Vs/QGlG5VwdU9dxfl6h0HkyUbnFet7txG0Sn8BhFY6EmplTdGlOgyORwnd56MpXFeyIW5FPZw29PJOzE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722625; c=relaxed/simple; bh=hRFIhnTfLnaAyAR/FJHl3b6p2rBAKes40e/Ty6a3718=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nLextfg463TfjxfBe6Ior+5dBusA5R0JvNBWbLtzXXkbZYwF/olSXo4ILcA54TR6Iw0VpwUIhrObc8m/4NP4iOnXgVtq/JUzncCeCzVgjBCsfSWyuPieldU6scV7EhfcTGtrnZmObH5/Fnj+nxC9VO8YTdnmr1a+XKtgWXQ3MIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Owyo1AXb; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Owyo1AXb" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5684ea117a3so5431370a12.0; Sun, 17 Mar 2024 17:43:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722621; x=1711327421; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9xW49xRGwOEZ3X1kEj3TcgEavVPmzEUMLGOfhWA4lGU=; b=Owyo1AXbFMS8d65Dplt4WBQtL9zqxCUoIw2FCrF3/jffqxvH7tAQkG/MYyAKREd7Qw fz977Aepxnh0tnwz8dq3gHr5iFwcj0PGwRoBogsIF2IBmS+VE74YtgOfaB2f9i4r7YDJ yXH5pQtHxrSDU2xrt9GkUvBTpq/gbSP54coMXFuqKZPsuzPQXBCfnJSV9NF0AHFKENa5 3X/Sls0PALRAwoXtS8GT8GT5fPZTaSGehYfhw7XGEo1s+zcOTmTXCHCjfkcAaEZAN0uG KgBgUXWal5tw/+OJqh7fGdfNeTjYxUY322HaJH5K66Myus3VFamosHNWMKYhWrR0kr/1 mFRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722621; x=1711327421; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9xW49xRGwOEZ3X1kEj3TcgEavVPmzEUMLGOfhWA4lGU=; b=jioV/DEROohbAF5dgjbwmF8VzN8ej/9cE11UNMLjUpulOOw1MpeYincxCU/r2sLdlF 88TDrpAKe54e/D5m5pCfa9ZLouurGiFMpAnbjIC4zjYhk0/vl1NOzzrfkg7KVmsz7WCv lRs+bvT3EH6tceWHmXjrPw42AUgumo2GvxX2Sca6Tw1dX8sLG8InOQAaCJDsYf0aHGM2 pjpPuObrE6zb8RDfNLqK+KflVVaLWY7Er+rh+JZoOST6Y7aIik04jMsofaybYaDc3X4o lDA6+RQfFlj85+aEz/eZt+l/+1mA1gOXOlL/LXv3/W1cDAvrLlcxyFa2x6erKLf1jH+H JWVg== X-Gm-Message-State: AOJu0YyMZXFa4T+YCqVEw5r4vZ4rHDHEgoAoAuZ6Fo0hv0PFmaMtkJu6 P8rxbPffsJxXi2YEnuZXn4k6puA8GYG4GNarttXpzjxDi6YWg1vojxb9yQuX X-Google-Smtp-Source: AGHT+IEWDJFtwqWqwAJ+4gPipnYVYFDYgB4VPf7Q8gdWx8xHrgFnSfnvg8C6m3q1mbqFCrx+o2OCrg== X-Received: by 2002:a05:6402:3222:b0:568:b38c:e2c1 with SMTP id g34-20020a056402322200b00568b38ce2c1mr5608670eda.35.1710722620964; Sun, 17 Mar 2024 17:43:40 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 04/14] io_uring/cmd: introduce io_uring_cmd_complete Date: Mon, 18 Mar 2024 00:41:49 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring_cmd_complete() does exactly what io_uring_cmd_done() does, that is completing the request, but doesn't ask for issue_flags argument. We have a couple of users hardcoding some random issue_flags values in drivers, which they absolutely should not do. This function will be used to get rid of them. Also, add comments warning users that they're only allowed to pass issue_flags that were given from io_uring. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/bb7e81aa31f9c878780d46e379d106124a7aa102.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- include/linux/io_uring/cmd.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index e453a997c060..9cbe986eab7d 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -26,12 +26,25 @@ static inline const void *io_uring_sqe_cmd(const struct io_uring_sqe *sqe) #if defined(CONFIG_IO_URING) int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); + +/* + * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd + * and a corresponding io_uring request. + * + * Note: the caller must never invent the @issue_flags mask, it's only allowed + * to pass what has been provided by the core io_uring code. + */ void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret, ssize_t res2, unsigned issue_flags); + void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned), unsigned flags); +/* + * The caller must never invent the @issue_flags mask, it's only allowed + * to pass what has been provided by the core io_uring code. + */ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags); @@ -56,6 +69,17 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, } #endif +/* + * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd + * and a corresponding io_uring request. Similar to io_uring_cmd_done() but + * without issue_flags argument. + */ +static inline void io_uring_cmd_complete(struct io_uring_cmd *ioucmd, + ssize_t ret, ssize_t res2) +{ + io_uring_cmd_done(ioucmd, ret, res2, IO_URING_F_UNLOCKED); +} + /* users must follow the IOU_F_TWQ_LAZY_WAKE semantics */ static inline void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned)) From patchwork Mon Mar 18 00:41:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594901 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0C451E53A; Mon, 18 Mar 2024 00:43:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722626; cv=none; b=quqO78mJOKvFqfmI2yGJbIee9W89zi7tTeOiqR661VQ3FGOMUr5rWeFG9VjQqcVraP9YgvVFiwZU/TWlWW0nmH8gIpO+jVY2dtiQNp+reLUKm5yjtM+uh4wsdD+RVez9FNzAUwUIYimI4L49IvSmzVFA39pwXopOpzI/uTMr5/U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722626; c=relaxed/simple; bh=2YNAhpdedsrTS00Mv1/nJs8615L2dx3VtYzU37e96po=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K5M3Szc9Ss1DSS1oHuW5sVmOIjnA0/ZYoXqhhGgFS9/FSBkO4Jds+ik/4Adi383PAtY3oZAqERt8/Ptf5YRLxpuN8VY9iTQ35nHLTP/7+3gKXhrnd25xczjf/muo0Q8vrMSQQoJE84HrGAuKHbDm034LslQZ3f+Th/KMpnG3sTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XhonV++Z; arc=none smtp.client-ip=209.85.208.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XhonV++Z" Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2d29aad15a5so46489631fa.3; Sun, 17 Mar 2024 17:43:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722622; x=1711327422; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZxmPTulNs2Gq+0rppneuvhvFDS8ypZJZX543PVINW9c=; b=XhonV++ZhgTn/ygtIcq2YVo1+zv1ID+Q5i/Vvl1MesNuJZXaVqCqWZ+RDeX/o8/y2o +6qpRtHj03qvhat7uyyMVo45orJihr4HbNzrYDJW+asqDs7HEH/f1IpxNDiMxwz5vjkL C1WQy2bCxQ9QJpi/yjtR+HFBHkh4Z6MWo1a9NK9kAVW19ESdE/Z2A5sLnS8v2dSguSiR MzyJRSBAmExwAPQKtyi2eoq7cZyCn/VX0VYjrtrpIykVqCEkZ8OnyVuh/VIbhR/7A0nb g3eqYpgjireJFl8Wp9KEfpazKuPLh7H2HIwopLb9WJnJrVwtaoAGmvTCPXyR6Yp1d1Es GoDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722622; x=1711327422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZxmPTulNs2Gq+0rppneuvhvFDS8ypZJZX543PVINW9c=; b=pOyw6KOPH/G0uy2nWpLKewU02AIw2DLxIItS3zbLn7plEfJR3gDWX0NLFZljqbx++E eztv0NYW3+JyM7f00TuIWJI1eL87fCGQ5BClCgPck7l1Zcuhszptspujpne+Pmbi57Y6 s74OprO+96UwaRwNPheyUtI6kZHT8gWWCudwcPLGYpCaACCnb77CC5AltnziHworbGN5 iiTY+yh2MyUD48D1skrdEiqjix3vb80N4DZi07OOOhbal7XL5hfsE/V17rcbpUDbLBfa jucBzOIuZ6Lnli05Hlrs0V+TQwVVlftZMLiQZ6Mx+Y0HDMQFA9mtz6GfEuBen+mT3I9k PRZQ== X-Gm-Message-State: AOJu0YyEVmtVHqfO9akLukaaqvWgeO8Ch0a+9oYfF+D8tlNGFO47fcz5 dXRLs2Zh5F1KIPuR2FssuG2Tm3oqYkaDMKu+VLgaWTBLiSkM8cgnwG3ggcAG X-Google-Smtp-Source: AGHT+IFuo2sTulBUQJ39TA92S8vpjPrTmR57rDtiouFuE9yKzW+XpT/vJMmnLBmFwhw1GyTi6zudkQ== X-Received: by 2002:a2e:7a02:0:b0:2d2:d3de:44fd with SMTP id v2-20020a2e7a02000000b002d2d3de44fdmr4959213ljc.29.1710722622565; Sun, 17 Mar 2024 17:43:42 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 05/14] ublk: don't hard code IO_URING_F_UNLOCKED Date: Mon, 18 Mar 2024 00:41:50 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 uring_cmd implementations should not try to guess issue_flags, just use a newly added io_uring_cmd_complete(). We're loosing an optimisation in the cancellation path in ublk_uring_cmd_cancel_fn(), but the assumption is that we don't care that much about it. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/2f7bc9fbc98b11412d10b8fd88e58e35614e3147.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- drivers/block/ublk_drv.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index bea3d5cf8a83..97dceecadab2 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1417,8 +1417,7 @@ static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq) return true; } -static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io, - unsigned int issue_flags) +static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io) { bool done; @@ -1432,15 +1431,14 @@ static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io, spin_unlock(&ubq->cancel_lock); if (!done) - io_uring_cmd_done(io->cmd, UBLK_IO_RES_ABORT, 0, issue_flags); + io_uring_cmd_complete(io->cmd, UBLK_IO_RES_ABORT, 0); } /* * The ublk char device won't be closed when calling cancel fn, so both * ublk device and queue are guaranteed to be live */ -static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd) { struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd); struct ublk_queue *ubq = pdu->ubq; @@ -1464,7 +1462,7 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd, io = &ubq->ios[pdu->tag]; WARN_ON_ONCE(io->cmd != cmd); - ublk_cancel_cmd(ubq, io, issue_flags); + ublk_cancel_cmd(ubq, io); if (need_schedule) { if (ublk_can_use_recovery(ub)) @@ -1484,7 +1482,7 @@ static void ublk_cancel_queue(struct ublk_queue *ubq) int i; for (i = 0; i < ubq->q_depth; i++) - ublk_cancel_cmd(ubq, &ubq->ios[i], IO_URING_F_UNLOCKED); + ublk_cancel_cmd(ubq, &ubq->ios[i]); } /* Cancel all pending commands, must be called after del_gendisk() returns */ @@ -1777,7 +1775,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, return -EIOCBQUEUED; out: - io_uring_cmd_done(cmd, ret, 0, issue_flags); + io_uring_cmd_complete(cmd, ret, 0); pr_devel("%s: complete: cmd op %d, tag %d ret %x io_flags %x\n", __func__, cmd_op, tag, ret, io->flags); return -EIOCBQUEUED; @@ -1842,7 +1840,7 @@ static void ublk_ch_uring_cmd_cb(struct io_uring_cmd *cmd, static int ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) { if (unlikely(issue_flags & IO_URING_F_CANCEL)) { - ublk_uring_cmd_cancel_fn(cmd, issue_flags); + ublk_uring_cmd_cancel_fn(cmd); return 0; } @@ -2930,7 +2928,7 @@ static int ublk_ctrl_uring_cmd(struct io_uring_cmd *cmd, if (ub) ublk_put_device(ub); out: - io_uring_cmd_done(cmd, ret, 0, issue_flags); + io_uring_cmd_complete(cmd, ret, 0); pr_devel("%s: cmd done ret %d cmd_op %x, dev id %d qid %d\n", __func__, ret, cmd->cmd_op, header->dev_id, header->queue_id); return -EIOCBQUEUED; From patchwork Mon Mar 18 00:41:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594902 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20A261E879; Mon, 18 Mar 2024 00:43:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722627; cv=none; b=Kul9MXyhGETSOEt+m5IWVrgur8bbXj/79l3VebzxEUJXFSFnN4UcAe3Gox8zzTwEMvecthItjHN7lTFl9zaPUwVfai4lzWcEI3CXr8G/3Zlu8LZuObNRKGnKLmD0dT4aegmBO7ImreKIV0zzrRGUs/sGaf4n2Qflz/XmZDBWnP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722627; c=relaxed/simple; bh=UbiTyPTYYZh3urZ8adZCwTIZvRvXSzAJm3mJtdK9Ehw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mZLZOgf7jDc7lzr9lDT32uLl90kTcPYTh6ktMSgi9q5IFjURHSOHqSQrH3eo2917WbfaH3enP4HnuQ3Q/2D8StcspUas9DJ8GTkzo84whcWvyqnKZdli4cBhjtafyb5afsiHwOGSquHWXI1PreuNdvWOGrMc7++hYebqKppPLT4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fPssFLjh; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fPssFLjh" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a466a1f9ea0so279713566b.1; Sun, 17 Mar 2024 17:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722624; x=1711327424; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XhKP8/ntiNtXJNW1YgeAv+tpjW8wxFfpi5StwH3GfS8=; b=fPssFLjhrjvtejrDB6PklYx5rF5MDROuCnPXWShncm6E69ZE0ZWzCkROANJGHve2ec n3m4QqO4ZXbMDd+EzsPsZWNhvloAb8a/+D/xhG661cihKCMfhzlGUcal8ZpfLBv2gHdq VzP/L0OHA1uzkn2HOGF3B4Hrr7OUcDp9WJytbn0moehFyM1+f66Do2EzwSjpQciUBbjC sDbaDF2ew7XKMAAX/ZBL4EVp/45+UvEuWBGqmOa6lehE1WO4rB1X9XmxpDNAbLNS4a2O ovAm6r5kUcOXP0fKhT8AtpFHRzfxiaWsFBPLWVGvDCDQT0XpmN7d5iAJwmsIgjwv9Jko NESQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722624; x=1711327424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XhKP8/ntiNtXJNW1YgeAv+tpjW8wxFfpi5StwH3GfS8=; b=QOFn8D9kOEwsZRTErSzJ32MpQfgP4CN6B9RJfE/iDGUfcyLQ8LSCIxhcrbF/tqCiU2 rI0rKpEGqL7HlcrEACbXzEFMQkN9/ms0VW34QA3/n8a504a/K23BcZO/rl2fbAVBKzzt MxoXSLl5oVxiDblRY9vYCBPuB+RwVXCXvdxXc3VRl7Qkm+Sgq/CdlIhLtmfkuiPcl/1B ZHeDmercYsSHz//94wV9qnLFRzMzPKNFn/MyW7fiC89MiWN1RM0FDdcd09r7RXODIOuR hcwFcqdO85BaS7BUfdRGFyare9QuccAjgJP+72huiKgptPr2F7HKk4fz+b6PkDXsrUI/ a52g== X-Gm-Message-State: AOJu0Ywbl00AO27Yr8mCQ8ApHsjjrBK/IvXudQK3tr+YLbF90HUs/g8B a16B9b4H3EgNauFroRN+93fvwe87Nlp7VVxFklBQ4lMtUWmjg7JsM3+MkwCh X-Google-Smtp-Source: AGHT+IHdQQEJxd/Ppa+IK/WPyTkY66l1gkV86eRmXnUotAPJ4/TROqh6EkJrdUfLI7aFyKnuvu3HbA== X-Received: by 2002:a05:6402:3786:b0:566:ab2b:e1ce with SMTP id et6-20020a056402378600b00566ab2be1cemr3741482edb.18.1710722623917; Sun, 17 Mar 2024 17:43:43 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 06/14] nvme/io_uring: don't hard code IO_URING_F_UNLOCKED Date: Mon, 18 Mar 2024 00:41:51 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 uring_cmd implementations should not try to guess issue_flags, use a freshly added helper io_uring_cmd_complete() instead. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/c661cc48f3dd4a09ace5f9d93f5d498cbf3de583.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe Reviewed-by: Kanchan Joshi --- drivers/nvme/host/ioctl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 3dfd5ae99ae0..1a7b5af42dbc 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -426,10 +426,13 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, * For iopoll, complete it directly. * Otherwise, move the completion to task work. */ - if (blk_rq_is_poll(req)) - nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); - else + if (blk_rq_is_poll(req)) { + if (pdu->bio) + blk_rq_unmap_user(pdu->bio); + io_uring_cmd_complete(ioucmd, pdu->status, pdu->result); + } else { io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb); + } return RQ_END_IO_FREE; } From patchwork Mon Mar 18 00:41:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594903 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74AC81EA7E; Mon, 18 Mar 2024 00:43:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722629; cv=none; b=nq0+J+xLUO1g6mfcfLCZuZYsn9Hw6I3thM+Rtj9QQ0L3SjdNNHcz4e1oOWegxLU6nkdG0WVHR5kIbbUCG/pY3n6OpwoYJaHbrmxrnwT3/CFaeoK9+BW/x4H1Ttj6nSRcFtq/ORZxmjK265h9/TFrC74+hOM3+h0l2/illjw84vs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722629; c=relaxed/simple; bh=JHMPfBFEnuALZqDa0moFr296ertczTzbv2NG2KcqXD0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mq9wELDQZV8myDOX8rcdPmgiYJTz1hwxSmVGssiUQwgmQH5z1e5yWuE1BVjVhpjbDaiBZby8wq1mbEV4ciHo30C0QMuGU0W+GHc/Aw0qQdnVUyjPpB6Ajp/uFg2oBWsiDCLngqgCYDJaI0A5cvkPiBw8eUBCxgyP4dqZI/GIFdU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ES5CCecQ; arc=none smtp.client-ip=209.85.218.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ES5CCecQ" Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-a465ddc2c09so276140666b.2; Sun, 17 Mar 2024 17:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722625; x=1711327425; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mfQZGoJZoDIJgsr2a2vsleYuSq2cLitr4Li2kyTmpxg=; b=ES5CCecQlCu7bL7GMrweirTp8DsqfuGP+/ttH2NiXsVvftFpVwhBzxwCtDzIkoO2nP nGPet2UmA/wMZa+9pvueMsROQUjTGbRHtFpBmhHinV9T4qTJdbFIrNKOt9KMRpKiMZvY rxJd1k3pacVDtq0TM/yJd+XmXZDh1QaPB6+hyS9pOAh+XlvToeoQt/HXgqu41frSB40e OujjvogrTkrVQ3LE1qYl437M1/WIM8sZ4eqX974NuLdAzKyoJPJC4N6IE6zqo5Qq+I2t VknN+wCspgaVgD2MxzpZW1ZCxLT8m0tVUzxNz5UWRkQcjdQ/R139V6HI7kaELG6nHoGL BDHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722625; x=1711327425; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mfQZGoJZoDIJgsr2a2vsleYuSq2cLitr4Li2kyTmpxg=; b=o8Ib4J56RAsT+6jM165imRJANi0b8ZK/bu9qt/fmYLTU1cs4o6JRLljkHefT07SZ43 Mt6MapPf2aPlnbrc9UpF1GMLGCOOZkdf10X3TuecQbpghR6cC5Wf/U+4iFmfRrbK/wf5 ZzfootxP9HWa9AtFeCsBdkWSQ4h1xKT5pAVIaOExxO6Znpw8H08e2KY+D4L4OJVQciA+ 3/5XWJXESZ5U1AQQUsRjVmwwveNhyeilsroPB4esJFr7Rd/bdHDiXq6begC9X5gE3neU 4u51X3mbjFYLkP7x8wjF8qxpuLlB8HtJLrRUg/SbKOZGQgX+NK189mQzK4DbzaVSjoa0 v+nQ== X-Gm-Message-State: AOJu0YwBC5IBchhk+7/f+m2kljrxgqOtcUcwpNJe8Tt1o0HzwkAynT3i 6hC/MjSaC9Y/wlJ+Cu1edrLr7tlvd4EzcHIAvCKxfJd1uAfk6YZOmMmYlLaU X-Google-Smtp-Source: AGHT+IF2I7pmyz8EqfxAZqX3ZeYVV5Gb7FAbI8a2E0XYGlwzOZH5yK5Pa/YSA6PwOxR/MoQUZxW9cw== X-Received: by 2002:a05:6402:1f08:b0:565:f7c7:f23c with SMTP id b8-20020a0564021f0800b00565f7c7f23cmr3765621edb.3.1710722625177; Sun, 17 Mar 2024 17:43:45 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 07/14] io_uring/rw: avoid punting to io-wq directly Date: Mon, 18 Mar 2024 00:41:52 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 kiocb_done() should care to specifically redirecting requests to io-wq. Remove the hopping to tw to then queue an io-wq, return -EAGAIN and let the core code io_uring handle offloading. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/595021a813bebf0256e80b95bcc594377096e08b.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 8 ++++---- io_uring/io_uring.h | 1 - io_uring/rw.c | 8 +------- 3 files changed, 5 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5d4b448fdc50..9d7bbdea6db5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -498,7 +498,7 @@ static void io_prep_async_link(struct io_kiocb *req) } } -void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use) +static void io_queue_iowq(struct io_kiocb *req) { struct io_kiocb *link = io_prep_linked_timeout(req); struct io_uring_task *tctx = req->task->io_uring; @@ -1505,7 +1505,7 @@ void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts) if (unlikely(req->task->flags & PF_EXITING)) io_req_defer_failed(req, -EFAULT); else if (req->flags & REQ_F_FORCE_ASYNC) - io_queue_iowq(req, ts); + io_queue_iowq(req); else io_queue_sqe(req); } @@ -2088,7 +2088,7 @@ static void io_queue_async(struct io_kiocb *req, int ret) break; case IO_APOLL_ABORTED: io_kbuf_recycle(req, 0); - io_queue_iowq(req, NULL); + io_queue_iowq(req); break; case IO_APOLL_OK: break; @@ -2135,7 +2135,7 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) if (unlikely(req->ctx->drain_active)) io_drain_req(req); else - io_queue_iowq(req, NULL); + io_queue_iowq(req); } } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 6426ee382276..472ba5692ba8 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -79,7 +79,6 @@ struct file *io_file_get_fixed(struct io_kiocb *req, int fd, void __io_req_task_work_add(struct io_kiocb *req, unsigned flags); bool io_alloc_async_data(struct io_kiocb *req); void io_req_task_queue(struct io_kiocb *req); -void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use); void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts); void io_req_task_queue_fail(struct io_kiocb *req, int ret); void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/rw.c b/io_uring/rw.c index 0585ebcc9773..576934dbf833 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -187,12 +187,6 @@ static inline loff_t *io_kiocb_update_pos(struct io_kiocb *req) return NULL; } -static void io_req_task_queue_reissue(struct io_kiocb *req) -{ - req->io_task_work.func = io_queue_iowq; - io_req_task_work_add(req); -} - #ifdef CONFIG_BLOCK static bool io_resubmit_prep(struct io_kiocb *req) { @@ -405,7 +399,7 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, if (req->flags & REQ_F_REISSUE) { req->flags &= ~REQ_F_REISSUE; if (io_resubmit_prep(req)) - io_req_task_queue_reissue(req); + return -EAGAIN; else io_req_task_queue_fail(req, final_ret); } From patchwork Mon Mar 18 00:41:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594904 Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24B3B1F959; Mon, 18 Mar 2024 00:43:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722630; cv=none; b=mTBi4iHyJ89gemrF6R2bRo2G2+9/lWEC921hBnxBmEoh6wIlqfHOdnfuA8e730Hzj7uPmHP/icwhASPeFX9XbzfFA0CVAxTzvRu656YiIP0NBV7nha4pOSEJ/DVL2FcMq33EtFHwfzwjiXLwEoDilswY0uyjWZj34lMVA/HEcNI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722630; c=relaxed/simple; bh=I4FA13dkgmmuNG4n0r1xczqangu1VhnefSMMGZSB7hg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pBDIaU8YOgw6b2Xn1d2k/blRI74TU/67P/pudAEFfCLl+wOQ+0jb759EEAqgrFoSaWy9wtfzNZiM6LconqB8w8PJo/ZD5sPaGZFlTfx0YLJWpavKNtX7N7TNQdmIC+JXpHfDSobJ8a17rCg73/hi7ajgIyD+73rWWSLfZiQEO3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=I9gBQCxB; arc=none smtp.client-ip=209.85.208.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="I9gBQCxB" Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2d4698f4936so54382991fa.1; Sun, 17 Mar 2024 17:43:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722626; x=1711327426; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qQestWxBXECy7oXqcNOqOE6lZ4//y3lbdvEiKFWaNno=; b=I9gBQCxB295irkeWaHHJi5hQxSNST32BtBxBSmI5BvpFdiaxcVQJ9hmEthYEdH16cm 4vFRtLI2K5esvwjyOtWAxLdJ322AYSyEX28BosYd76PDoeadi4jgiBp7fL5/qd03Lef8 1bWzbw+6gC9ueMBFzYH7R5Ia/m38OLuXDEj9loPCCgrUBKM1qjaV+cg9NAQdFS2WfBeo 0EvQGtc5qTaczZTBky13+ure5p6NMBM8JVb45lRdBMC144R/DleUHEB3wkRAz7+6tTw1 IVaRjvs2JpUa6hZhQT2D+/CNTP4tVEWovcA9DpSFpg5uT6ddMA1jUSa7SyjA/tXf+uwp P6HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722626; x=1711327426; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qQestWxBXECy7oXqcNOqOE6lZ4//y3lbdvEiKFWaNno=; b=nc44i/G8aBvzI4DykDHzxBGe+OpGmEDf7XhSi2cGbNDPI+5IoThzGjpgqbhhoizfDT oVkt3aT/7l9UyBVNrnN6xqe+j+N5OqEPdwET8dZ0WBRDAp0ZA2mrfwkdAsApUp9U7HvG SaYt0Kg2vFR/FvT2iiFqPgAd/oLnITgH7oq6SrKTy0AaLdXtwnH3+g7z3d4Gr5b5QD1K loF1uJ/iXQ++BrZ0VHTjiA/ySTXkYkgVEjgYEA3qJWHm4HnRsXyxSd/Ts8s2Km4kbQhe TLjLWtumR1sFxB0R8bv+TJbERjEj2diD3NsUu91BPXYZniJxJOINqYD7okkf2NyYkOWO mF2w== X-Gm-Message-State: AOJu0Yy6Z4FmqatAy8B6Q5mqj7tmsG/zaMO9F5F08M3Wd0vZMXIbxiSe Y4M0fLdqPrtArUwFc0Cei7wmW2X8GngoqCdvenloA2C8XUkZIkXI1NnNhPaC X-Google-Smtp-Source: AGHT+IGMuz5hDsj9rZkOXR1giw06fXqQq6D5mJ5PkG3GWCbYHwpVFQHzT+cIJeEEpVyP+7ywIRvaDg== X-Received: by 2002:a2e:95c8:0:b0:2d4:7285:b997 with SMTP id y8-20020a2e95c8000000b002d47285b997mr5968301ljh.27.1710722626576; Sun, 17 Mar 2024 17:43:46 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 08/14] io_uring: force tw ctx locking Date: Mon, 18 Mar 2024 00:41:53 +0000 Message-ID: <49af41a9127c06a8133a371b59390e771b39bae4.1710720150.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We can run normal task_work without locking the ctx, however we try to lock anyway and most handlers prefer or require it locked. It might have been interesting to multi-submitter ring with high contention completing async read/write requests via task_work, however that will still need to go through io_req_complete_post() and potentially take the lock for rsrc node putting or some other case. In other words, it's hard to care about it, so alawys force the locking. The case described would also because of various io_uring caches. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/1f7f31f4075e766343055ff0d07482992038d467.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9d7bbdea6db5..3184d57f9a35 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1191,8 +1191,9 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - /* if not contended, grab and improve batching */ - ts.locked = mutex_trylock(&ctx->uring_lock); + + ts.locked = true; + mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } INDIRECT_CALL_2(req->io_task_work.func, @@ -1453,11 +1454,9 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, if (io_run_local_work_continue(ctx, ret, min_events)) goto again; - if (ts->locked) { - io_submit_flush_completions(ctx); - if (io_run_local_work_continue(ctx, ret, min_events)) - goto again; - } + io_submit_flush_completions(ctx); + if (io_run_local_work_continue(ctx, ret, min_events)) + goto again; trace_io_uring_local_work_run(ctx, ret, loops); return ret; @@ -1481,14 +1480,12 @@ static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = {}; + struct io_tw_state ts = { .locked = true }; int ret; - ts.locked = mutex_trylock(&ctx->uring_lock); + mutex_lock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &ts, min_events); - if (ts.locked) - mutex_unlock(&ctx->uring_lock); - + mutex_unlock(&ctx->uring_lock); return ret; } From patchwork Mon Mar 18 00:41:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594905 Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4B0D208CB; Mon, 18 Mar 2024 00:43:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722632; cv=none; b=AsfgGWcnslLTROv6VVBEuzAdIQAgsvhMgfKbo+9nrOtXgvcMRcNi4MSCjosF2fB2LtAulQz37lS70pxGP9jfFFGxlfr2cRztzm5i+wFKQBztbZp4ha3xCHObWY37aYr0lxr3+82Mwjxw0Vgv7hD6LRbxaXfyEhU0uGrbqxWeCeY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722632; c=relaxed/simple; bh=02HujhgwnfkOWVmPn2Yk089NvVni5bJATFLrT2ChMok=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EltMs4Y4DeD3vq6HbF50VWCNz9FRgoj3M6L1sdwavND2be8ndmlft54RDBjTZnh0Perq0OwtWFSfIlK3GwEN63dvPHkelBuWFIumBZl4+kI3SHUNRvTOryECHBElhdeIC6fL93Ni1gNTqbaiRBKS/P1MdvUgjR1gkUGIsm2Rm20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jGjAyyMb; arc=none smtp.client-ip=209.85.167.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jGjAyyMb" Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-513d717269fso3581460e87.0; Sun, 17 Mar 2024 17:43:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722628; x=1711327428; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M3C7yKMELUaSacTHIbXhejLYdgP7SmusGhf9nsXaFx0=; b=jGjAyyMb0rwL2qUpdhOSJm7kaWu7pbu07DQbc6DLpGKW7TL0tgZP6DSevk3NHMfbOV nMw8KGtakvz1iNveMQKU9VFyGpd7fW0sivH2yftesdFakHrxyGID12+hv+0pKuyrCMIn MGS8JhMEstmTrWRhaqjJ4Mz16mV/excymJi0sCjFLJ1P4q7v2z53ORFE28mqRQFPfByD 7TZNGbj8N9aZR2TJ1V8rxrdTntji/d5rfB3zURm7iySBNi7AXgB0tQV6KHWBiYzEmibp Lu9Omsd2YSvvEhH9jvlFgz00mS4u6o3rX0R97vG0o6/Jp84Vjv0qmPybJmPw47ibHbgz bl0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722628; x=1711327428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M3C7yKMELUaSacTHIbXhejLYdgP7SmusGhf9nsXaFx0=; b=vwD83lHZLrBi/LTd7vyCRdQThfIJKihBWGi2Fn60fyn0MOWX2UfEN7NbeYI+KKnY0Q xgOneyJa0V8aoWrFK57cPQvCqrhFAtAwFLNAckDnRn6I54POg+yCTepo1nnEAsoH5Wau rBDZd1yj6KnjLCsnobvZhlspIMBHE+T3qG2kjwMoQ90QVOGSZiBCzdlcNIm4Xhnv7YtA tJEv9XJYzPwwjU2nkihaWQ0M/+lgxu2c1w8JGz2fpjFeByPRfgaAO0KGtu+LIlJ6EUUQ qAe48FM/s+c7p1cxjbKl/yYwdSEpYzmLQOEFY+RRDIl6kB/VuDD18yskxHoqnXAFeSHC 9BJg== X-Gm-Message-State: AOJu0YxdFuZjvfJ4lP+LBLW5qm8ablUeRtslrlLxYAurnsIj5sAcJGhp RB7lZou25rLhYnGHnUwjMj7DQNdVSCthcQcpUukTBahbUD7Dv1cYLDq+pTEi X-Google-Smtp-Source: AGHT+IFVGBdvHlJG+ynJ29UPahkl1CoWKwLZkZRRF9dkpQWMsQ3YuUcnZlUQNjMrAdEHKlKpKdUdRA== X-Received: by 2002:ac2:5f92:0:b0:513:cf4e:941b with SMTP id r18-20020ac25f92000000b00513cf4e941bmr6461889lfe.19.1710722628068; Sun, 17 Mar 2024 17:43:48 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 09/14] io_uring: remove struct io_tw_state::locked Date: Mon, 18 Mar 2024 00:41:54 +0000 Message-ID: <1609600918e10b0b8038a6571f5b84254b12d221.1710720150.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 ctx is always locked for task_work now, so get rid of struct io_tw_state::locked. Note I'm stopping one step before removing io_tw_state altogether, which is not empty, because it still serves the purpose of indicating which function is a tw callback and forcing users not to invoke them carelessly out of a wrong context. The removal can always be done later. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/04482ca7bfea9eb47ba483c590aa46c83315a6ea.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 2 -- io_uring/io_uring.c | 31 ++++++++----------------------- io_uring/io_uring.h | 5 +---- io_uring/poll.c | 2 +- io_uring/rw.c | 6 ++---- io_uring/timeout.c | 8 ++------ io_uring/uring_cmd.c | 8 ++------ io_uring/waitid.c | 2 +- 8 files changed, 17 insertions(+), 47 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e24893625085..5a2afbc93887 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -439,8 +439,6 @@ struct io_ring_ctx { }; struct io_tw_state { - /* ->uring_lock is taken, callbacks can use io_tw_lock to lock it */ - bool locked; }; enum { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3184d57f9a35..ef089f6367ea 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -252,14 +252,12 @@ static __cold void io_fallback_req_func(struct work_struct *work) fallback_work.work); struct llist_node *node = llist_del_all(&ctx->fallback_llist); struct io_kiocb *req, *tmp; - struct io_tw_state ts = { .locked = true, }; + struct io_tw_state ts = {}; percpu_ref_get(&ctx->refs); mutex_lock(&ctx->uring_lock); llist_for_each_entry_safe(req, tmp, node, io_task_work.node) req->io_task_work.func(req, &ts); - if (WARN_ON_ONCE(!ts.locked)) - return; io_submit_flush_completions(ctx); mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); @@ -1163,11 +1161,9 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) return; if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_andnot(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); - if (ts->locked) { - io_submit_flush_completions(ctx); - mutex_unlock(&ctx->uring_lock); - ts->locked = false; - } + + io_submit_flush_completions(ctx); + mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); } @@ -1191,8 +1187,6 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - - ts.locked = true; mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } @@ -1465,22 +1459,16 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = { .locked = true, }; - int ret; + struct io_tw_state ts = {}; if (llist_empty(&ctx->work_llist)) return 0; - - ret = __io_run_local_work(ctx, &ts, min_events); - /* shouldn't happen! */ - if (WARN_ON_ONCE(!ts.locked)) - mutex_lock(&ctx->uring_lock); - return ret; + return __io_run_local_work(ctx, &ts, min_events); } static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = { .locked = true }; + struct io_tw_state ts = {}; int ret; mutex_lock(&ctx->uring_lock); @@ -1708,10 +1696,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts) { - if (ts->locked) - io_req_complete_defer(req); - else - io_req_complete_post(req, IO_URING_F_UNLOCKED); + io_req_complete_defer(req); } /* diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 472ba5692ba8..6cad3ef3408b 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -344,10 +344,7 @@ static inline bool io_task_work_pending(struct io_ring_ctx *ctx) static inline void io_tw_lock(struct io_ring_ctx *ctx, struct io_tw_state *ts) { - if (!ts->locked) { - mutex_lock(&ctx->uring_lock); - ts->locked = true; - } + lockdep_assert_held(&ctx->uring_lock); } /* diff --git a/io_uring/poll.c b/io_uring/poll.c index 6db1dcb2c797..8901dd118e50 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -322,7 +322,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_fill_cqe_req_aux(req, ts->locked, mask, + if (!io_fill_cqe_req_aux(req, true, mask, IORING_CQE_F_MORE)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; diff --git a/io_uring/rw.c b/io_uring/rw.c index 576934dbf833..c7f9246ff508 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -305,11 +305,9 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts) io_req_io_end(req); - if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; + if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) + req->cqe.flags |= io_put_kbuf(req, 0); - req->cqe.flags |= io_put_kbuf(req, issue_flags); - } io_req_task_complete(req, ts); } diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 7fd7dbb211d6..0a48e6acd0b2 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -72,10 +72,7 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; if (!io_timeout_finish(timeout, data)) { - bool filled; - filled = io_fill_cqe_req_aux(req, ts->locked, -ETIME, - IORING_CQE_F_MORE); - if (filled) { + if (io_fill_cqe_req_aux(req, true, -ETIME, IORING_CQE_F_MORE)) { /* re-arm timer */ spin_lock_irq(&ctx->timeout_lock); list_add(&timeout->list, ctx->timeout_list.prev); @@ -301,7 +298,6 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *ts) { - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_kiocb *prev = timeout->prev; int ret = -ENOENT; @@ -313,7 +309,7 @@ static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *t .data = prev->cqe.user_data, }; - ret = io_try_cancel(req->task->io_uring, &cd, issue_flags); + ret = io_try_cancel(req->task->io_uring, &cd, 0); } io_req_set_res(req, ret ?: -ETIME, 0); io_req_task_complete(req, ts); diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 9590081feb2d..77c29f6883d3 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -58,13 +58,9 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned issue_flags = IO_URING_F_UNLOCKED; - /* locked task_work executor checks the deffered list completion */ - if (ts->locked) - issue_flags = IO_URING_F_COMPLETE_DEFER; - - ioucmd->task_work_cb(ioucmd, issue_flags); + /* task_work executor checks the deffered list completion */ + ioucmd->task_work_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); } void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, diff --git a/io_uring/waitid.c b/io_uring/waitid.c index 77d340666cb9..6362ec20abc0 100644 --- a/io_uring/waitid.c +++ b/io_uring/waitid.c @@ -118,7 +118,7 @@ static int io_waitid_finish(struct io_kiocb *req, int ret) static void io_waitid_complete(struct io_kiocb *req, int ret) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); - struct io_tw_state ts = { .locked = true }; + struct io_tw_state ts = {}; /* anyone completing better be holding a reference */ WARN_ON_ONCE(!(atomic_read(&iw->refs) & IO_WAITID_REF_MASK)); From patchwork Mon Mar 18 00:41:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594906 Received: from mail-ed1-f47.google.com (mail-ed1-f47.google.com [209.85.208.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D16A820DD0; Mon, 18 Mar 2024 00:43:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722633; cv=none; b=APMqoIba7xKpdTDlnrjiYVAMQVjPh2123v+OvUr3/lTNVcFp024T/HBVAKJdrJhkzd3cYiPx31xwete5MbEaDIcseban4OTb0nM3hhl2E5OQjPkCsPzpPx93BiFe2k83Cdr+SpoXpEy+ops6q6mC9Q6PsqiUyguhfBAafr7gkvw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722633; c=relaxed/simple; bh=NM6gv/o9UF/VHTQExPt8kQq8ePNaVU2X1jg4KWfWkLA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a5zhOP/xoFfUQWleTtIDWaztTeIA8inFcZGhQn7y/BaZQdtyh7Sy44Y/k7+3lrQiXkUkyssg6aX7NhxRRH8KHRcEKHaVagQq8PwPjnjbDE/uRBJWoZAKgs7yG4V7LLwjqEPIfWpT3XpNfbCQmJUCY+MXolO5w8urTeSA5JKWBQw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JB/84Zok; arc=none smtp.client-ip=209.85.208.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JB/84Zok" Received: by mail-ed1-f47.google.com with SMTP id 4fb4d7f45d1cf-56840d872aeso3407861a12.0; Sun, 17 Mar 2024 17:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722630; x=1711327430; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H6AXZxC8m1pgrrNg/yyM/YZEKwCJeLeCZY1gTFaZL+4=; b=JB/84Zokz628Th5Z05nk3UiXFARy0zZw69ugYNpImWBk43mld06OwlWijsFkrE9Xf1 EzbJTjho/GQmEvz/dhC0pkUE8hdMLmZ5ccQQBI69fHkIAh7TznWZvhgOcEkDNB1DhnTu HdtU6tIeNJMFsu8F96FATHzMnPUwNqfu7stk1DXpT7b0h67nlW415tg5pRvMJ9iG3no/ ZFR56s49UUefn+9856pFIBFDQJltT3GLgD2cK+oIB4Vn4HUAzPCfq5jGYHcq0Zu9WoD2 8ixp1uZ9T3/dPbide8pxPSpaHsEpEKQ8HqEFNBJksY8CXaW3Hll624NCab1R+evY8bfJ rVYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722630; x=1711327430; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H6AXZxC8m1pgrrNg/yyM/YZEKwCJeLeCZY1gTFaZL+4=; b=uBUQL41Mu0XwmWxMaRjrPQeKioLAr3ZYfNNxjZPAWIW4y/Q8Oa7p4QPxGYCUU+A1Jd uBaRuVOaooWehrVFM84OShCAeoyJea64xrhYpcFa43NZ39vVyDCZ3vxhR2zcAfk7VGFi 4lYwayX3t651iRsjWeugMvqDkaahJa4klZ9ScwAg3FisxQGiIGJ/xTYn9QoUnF7mvYyA Fh4VY6j7PWlu/k4f6/5DVvRLmn4KnAP6DQ4sa24SnOb7BdzboIhGK3Dbe+aJyhstNFal tLaiz26gwiHx2wMmkNM0MlUbcn1/TNYNqkg5+Wj2Q/+1VDRN3CjGIw6U9JiPLLCY9ng5 0Kww== X-Gm-Message-State: AOJu0YzCfvKyu2GrxFlQx6knX23tAdbwsRwdpzNY86zWucDec8W0+j65 d415kkyogdTSMRiDeJyjFfjO1XyOlabjgeTi1zRUIdO19KCQPdx2EdPqZ7gy X-Google-Smtp-Source: AGHT+IEvjoOsStRKdKyACweecEAcBNvOD2LAaL8zDYHFV6kdWGoSyS2vgteQuv11dwX/exxHXR9RKA== X-Received: by 2002:a05:6402:5d0:b0:565:6e34:da30 with SMTP id n16-20020a05640205d000b005656e34da30mr5331650edx.21.1710722629665; Sun, 17 Mar 2024 17:43:49 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 10/14] io_uring: refactor io_fill_cqe_req_aux Date: Mon, 18 Mar 2024 00:41:55 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The restriction on multishot execution context disallowing io-wq is driven by rules of io_fill_cqe_req_aux(), it should only be called in the master task context, either from the syscall path or in task_work. Since task_work now always takes the ctx lock implying IO_URING_F_COMPLETE_DEFER, we can just assume that the function is always called with its defer argument set to true. Kill the argument. Also rename the function for more consistency as "fill" in CQE related functions was usually meant for raw interfaces only copying data into the CQ without any locking, waking the user and other accounting "post" functions take care of. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/777fb7fbd2a3ba526a876fc422288c5f65283f12.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 16 +++------------- io_uring/io_uring.h | 2 +- io_uring/net.c | 6 ++---- io_uring/poll.c | 3 +-- io_uring/rw.c | 4 +--- io_uring/timeout.c | 2 +- 6 files changed, 9 insertions(+), 24 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ef089f6367ea..30542dda1473 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -913,40 +913,30 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) state->cqes_count = 0; } -static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { bool filled; io_cq_lock(ctx); filled = io_fill_cqe_aux(ctx, user_data, res, cflags); - if (!filled && allow_overflow) + if (!filled) filled = io_cqring_event_overflow(ctx, user_data, res, cflags, 0, 0); io_cq_unlock_post(ctx); return filled; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) -{ - return __io_post_aux_cqe(ctx, user_data, res, cflags, true); -} - /* * A helper for multishot requests posting additional CQEs. * Should only be used from a task_work including IO_URING_F_MULTISHOT. */ -bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags) +bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) { struct io_ring_ctx *ctx = req->ctx; u64 user_data = req->cqe.user_data; struct io_uring_cqe *cqe; lockdep_assert(!io_wq_current_is_worker()); - - if (!defer) - return __io_post_aux_cqe(ctx, user_data, res, cflags, false); - lockdep_assert_held(&ctx->uring_lock); if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 6cad3ef3408b..4bc96470e591 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -67,7 +67,7 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); -bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags); +bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); diff --git a/io_uring/net.c b/io_uring/net.c index 1e7665ff6ef7..ed798e185bbf 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -706,8 +706,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, * receive from this socket. */ if ((req->flags & REQ_F_APOLL_MULTISHOT) && !mshot_finished && - io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - *ret, cflags | IORING_CQE_F_MORE)) { + io_req_post_cqe(req, *ret, cflags | IORING_CQE_F_MORE)) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; @@ -1428,8 +1427,7 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (ret < 0) return ret; - if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - ret, IORING_CQE_F_MORE)) + if (io_req_post_cqe(req, ret, IORING_CQE_F_MORE)) goto retry; io_req_set_res(req, ret, 0); diff --git a/io_uring/poll.c b/io_uring/poll.c index 8901dd118e50..5d55bbf1de15 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -322,8 +322,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_fill_cqe_req_aux(req, true, mask, - IORING_CQE_F_MORE)) { + if (!io_req_post_cqe(req, mask, IORING_CQE_F_MORE)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; } diff --git a/io_uring/rw.c b/io_uring/rw.c index c7f9246ff508..35216e8adc29 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -955,9 +955,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) cflags = io_put_kbuf(req, issue_flags); rw->len = 0; /* similarly to above, reset len to 0 */ - if (io_fill_cqe_req_aux(req, - issue_flags & IO_URING_F_COMPLETE_DEFER, - ret, cflags | IORING_CQE_F_MORE)) { + if (io_req_post_cqe(req, ret, cflags | IORING_CQE_F_MORE)) { if (issue_flags & IO_URING_F_MULTISHOT) { /* * Force retry, as we might have more data to diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 0a48e6acd0b2..3458ca550b83 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -72,7 +72,7 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; if (!io_timeout_finish(timeout, data)) { - if (io_fill_cqe_req_aux(req, true, -ETIME, IORING_CQE_F_MORE)) { + if (io_req_post_cqe(req, -ETIME, IORING_CQE_F_MORE)) { /* re-arm timer */ spin_lock_irq(&ctx->timeout_lock); list_add(&timeout->list, ctx->timeout_list.prev); From patchwork Mon Mar 18 00:41:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594907 Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B13D520DE8; Mon, 18 Mar 2024 00:43:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722634; cv=none; b=Lo3VR9k6/hNfe+56ZKH1T8uLF5APAb6FXSS6nAaN1hpWCSoQ39IQ3egVTURRtwu2eguu1+8O7+PeIm13iB+BqVwrz+yeM2LtKx7e2xjblxZa3inNUODhsRUymKoQF03gLiLSxcwTeMRIL2DL4NEJmfFU6n8nwKaZNXgE55GgHIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722634; c=relaxed/simple; bh=x2/cLve01Gl9KonIriyqaOu836GCLMtAir5I+UFsNBo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B/rulofAz4zc0eM/O79y4HgNHAgW/0KdDzAE4JIhjTSAWZjSu+8sryhb411/PzVxiHgKLTwZSduNVcaT3ClBZM3L9m9tpObmzbLihympmHEh6LOky58EqSQazuRfADb6YYji5MbVC17ApHOxsvW6s6mLDAOdeCXESqU4x9OfEG8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mNEr+XtC; arc=none smtp.client-ip=209.85.208.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mNEr+XtC" Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-568a9c331a3so3279012a12.0; Sun, 17 Mar 2024 17:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722631; x=1711327431; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ENeNI8erwyYqlcnu32QiKd1Gq82426YelFfnxEPhMrM=; b=mNEr+XtCbaVpe9AjQn1sy8ysI31xFZLuhUg1op5/gH+dlHhUH2RCaHlmjmlYrFZWlz dkkr4dBahP1JQ528TYdax1O9LntUJ089NioyzvUW8+23wFPlo/IlSghRHDnRbXDuAQG4 KISyoH9GPF8CgEXalFz/9YhtFwLZ8cHT3i+ghFoCR7snlEiPovWA4Em25RiHsi/684YS PcVvlRutzbhScCDC/b2sVamXdl2y0jSvd1SETHObN29ZehdLPwvHVVnZbdZ+yszOXyKe FK5AlrcbVVNudaZxjELLpPMY1AhfqeEXhU2b1RIxOn7ccdtRjwlUH2/5Haf7zl+1Int2 y7mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722631; x=1711327431; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ENeNI8erwyYqlcnu32QiKd1Gq82426YelFfnxEPhMrM=; b=k8d0b0SrZh/LZGGo/UbKSwXcCGKr0wYrbIb+1I1qJG0AYMqQEuoi5P8TE0OsXb/jQQ TFrJGSQMl7I4QR100IT7IwUbLKynDDpI/opmFnxALxpuMaep86Ov4kdT70zppvKSRCDk m7rcdCZtVVOw7u8k4cACdo36U2CyAmDbIY61V5HPn+wGPquo4rvgk858ZVLenJEbU1Ei zEO0pjcotPDWHcpZr5N8qQwg+WlfTpNRU/f/mBVBoq4uejCDTQUGMhuFK/88R1iLP5H2 KLPm4LQ15OZxgUZK8CE/f1ABcXeq4lMID0HsaaVU9A8KPvYaa/WAPw0Hy1Bk57QljW/k rG1w== X-Gm-Message-State: AOJu0Yz4yH5Yh6CjU4/3Ultze69b4SOsF9DBDG9CpmcyU9Gy2Tnskbw+ +bS87uy0oslyo9jPVK5DOfkjH+Zr/14qwRT1XJxMZdhnxhlpJ07f26BuvBvH X-Google-Smtp-Source: AGHT+IFifW6UGgAX+3YlBzNel2SfEM9hqh6EhYll2yzJzkFWJGRgB72veFvMsHA/6T1lOATLjZsZeQ== X-Received: by 2002:a05:6402:3648:b0:568:1b94:fb88 with SMTP id em8-20020a056402364800b005681b94fb88mr8843363edb.23.1710722631021; Sun, 17 Mar 2024 17:43:51 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 11/14] io_uring: get rid of intermediate aux cqe caches Date: Mon, 18 Mar 2024 00:41:56 +0000 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_post_aux_cqe(), which is used for multishot requests, delays completions by putting CQEs into a temporary array for the purpose completion lock/flush batching. DEFER_TASKRUN doesn't need any locking, so for it we can put completions directly into the CQ and defer post completion handling with a flag. That leaves !DEFER_TASKRUN, which is not that interesting / hot for multishot requests, so have conditional locking with deferred flush for them. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/0eb3f55722540a11b036d3c90771220eb082d65e.1710514702.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +- io_uring/io_uring.c | 64 ++++++++-------------------------- 2 files changed, 15 insertions(+), 52 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5a2afbc93887..ea7e5488b3be 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -205,6 +205,7 @@ struct io_submit_state { bool plug_started; bool need_plug; + bool cq_flush; unsigned short submit_nr; unsigned int cqes_count; struct blk_plug plug; @@ -342,8 +343,6 @@ struct io_ring_ctx { unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; - struct io_uring_cqe completion_cqes[16]; - spinlock_t completion_lock; /* IRQ completion list, under ->completion_lock */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 30542dda1473..8c485bcb5cb7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -176,7 +176,7 @@ static struct ctl_table kernel_io_uring_disabled_table[] = { static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { if (!wq_list_empty(&ctx->submit_state.compl_reqs) || - ctx->submit_state.cqes_count) + ctx->submit_state.cq_flush) __io_submit_flush_completions(ctx); } @@ -636,6 +636,12 @@ static inline void __io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } +static inline void __io_cq_unlock(struct io_ring_ctx *ctx) +{ + if (!ctx->lockless_cq) + spin_unlock(&ctx->completion_lock); +} + static inline void io_cq_lock(struct io_ring_ctx *ctx) __acquires(ctx->completion_lock) { @@ -888,31 +894,6 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, return false; } -static void __io_flush_post_cqes(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) -{ - struct io_submit_state *state = &ctx->submit_state; - unsigned int i; - - lockdep_assert_held(&ctx->uring_lock); - for (i = 0; i < state->cqes_count; i++) { - struct io_uring_cqe *cqe = &ctx->completion_cqes[i]; - - if (!io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags)) { - if (ctx->lockless_cq) { - spin_lock(&ctx->completion_lock); - io_cqring_event_overflow(ctx, cqe->user_data, - cqe->res, cqe->flags, 0, 0); - spin_unlock(&ctx->completion_lock); - } else { - io_cqring_event_overflow(ctx, cqe->user_data, - cqe->res, cqe->flags, 0, 0); - } - } - } - state->cqes_count = 0; -} - bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { bool filled; @@ -933,31 +914,16 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) { struct io_ring_ctx *ctx = req->ctx; - u64 user_data = req->cqe.user_data; - struct io_uring_cqe *cqe; + bool posted; lockdep_assert(!io_wq_current_is_worker()); lockdep_assert_held(&ctx->uring_lock); - if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { - __io_cq_lock(ctx); - __io_flush_post_cqes(ctx); - /* no need to flush - flush is deferred */ - __io_cq_unlock_post(ctx); - } - - /* For defered completions this is not as strict as it is otherwise, - * however it's main job is to prevent unbounded posted completions, - * and in that it works just as well. - */ - if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) - return false; - - cqe = &ctx->completion_cqes[ctx->submit_state.cqes_count++]; - cqe->user_data = user_data; - cqe->res = res; - cqe->flags = cflags; - return true; + __io_cq_lock(ctx); + posted = io_fill_cqe_aux(ctx, req->cqe.user_data, res, cflags); + ctx->submit_state.cq_flush = true; + __io_cq_unlock_post(ctx); + return posted; } static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1551,9 +1517,6 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_wq_work_node *node; __io_cq_lock(ctx); - /* must come first to preserve CQE ordering in failure cases */ - if (state->cqes_count) - __io_flush_post_cqes(ctx); __wq_list_for_each(node, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1575,6 +1538,7 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) io_free_batch_list(ctx, state->compl_reqs.first); INIT_WQ_LIST(&state->compl_reqs); } + ctx->submit_state.cq_flush = false; } static unsigned io_cqring_events(struct io_ring_ctx *ctx) From patchwork Mon Mar 18 00:41:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594908 Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CDF221A19; Mon, 18 Mar 2024 00:43:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722635; cv=none; b=licibyaJ/61Hwy4UyUuRIvXEtuHZL9vHGTMSMAcqat7+T6QK6UuFqFv+DeokERqSMYeow2KW5MSKIJMCcbnq03SiUeXFCUsa5vZ6/CvFvZD5yI3+Kz0hCn+cIx+Msv+mMXjy9PWarbqRAxVpXwDqDNN1q/tiFc+AbzmTbopG00I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722635; c=relaxed/simple; bh=i0Hj4ZJyn+qslQjM0g9NmEqa1lCv6CqW2NFU4mBV/4I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eqs3HgtR3BAxHMeCsH5CQdOB2tBevaf78EeWnjoBnnQvKEAiJ1+P9MWfzBnJurDYZk9Rl+osZTilTsWf81XtcLn3IW/8HpjUl8l/liaDCicH32mA7tAx/45hPyRQOi9+O4oo8TuYyeJwV0K7XBLUuSAdkR8VwWTNju4ZxAEBpL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OVO5+WEc; arc=none smtp.client-ip=209.85.167.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OVO5+WEc" Received: by mail-lf1-f48.google.com with SMTP id 2adb3069b0e04-513da1c1f26so3759614e87.3; Sun, 17 Mar 2024 17:43:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722632; x=1711327432; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=glxNnNtmfNsWk//vdNi4gUCF0HOiRABAhUOOJS4lOws=; b=OVO5+WEciHioOAkUvPwgo5dLB7Nrz7t9DHHU7KQ/CvvFECH3W8FjKYx6CGcIxkQsZw QWJhR0iHVMalTjiA7QTGHZIaytkEhAsp/YUr53abzJvEAuK9VLBgdI0n/PvtdhzWhV1i QBWLqGu3765Pq7rGC7YEoKgHoQpZ25gc6RYZVPCrtf22ZMXrUnl9jaOmXpsU69ZU9PcZ U377JanAHBuhGxaxmcZkG/nd07QlLMc/KYV6gyTptAMhfGqd/M1yZ64BmngcQQYI1b82 oL4Ap5fzdlvGNMiWQE4zLgkzJoC5lV+5fKdAfCdyViFPQYAw6V8j3j/4is0lcfEYGOHE FtcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722632; x=1711327432; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=glxNnNtmfNsWk//vdNi4gUCF0HOiRABAhUOOJS4lOws=; b=Fu6oQsx4khmBnF9lrRWfRUTLa72e2gPMDQLmbEIzyMOrHdrOR08MnFLrLXzTUzoDLs +CVbyPhI6ZP08pegU6ksyGB/KQD74xHbTSRWFhCL4xMEnwaabB8Im7k5w4BOu1cK3V1y GSxJeJJwdGwpuGXh51cy8MQIaNp7fysahi0eu82DCn3KJhu4jbPBz3jgZgbiWQifoml8 cTWkSfHwFc7pZbp/ct1E2ewrmwXhJTxi2IeLdFUjwGU2QRO45ESFE7J+K1LBHVLNuDmm XOzY6juTvFQnV7KOjqljOUQ1Lz1pg/9UW7fLywTKTFeyF+M40Y1n7WbMApGe2lIYyIrJ iSzw== X-Gm-Message-State: AOJu0Yxwz/L/KgK++8P49yN4fdYem8c3fFxz7HWKABWkxvt6b/2XWOiP aR6TuS3o/BhjUB1fEDwEMlT3PDya+7UWgzTTR++CwmFLMsQ8SE2aTj8Yxn14 X-Google-Smtp-Source: AGHT+IFQNKagcRwQtrYAYAucEd2H5Vhhb3L1d+K2JiCynErIw6OO3RwKIguNGLzwBVCIcgY7tkGwwQ== X-Received: by 2002:a05:6512:3087:b0:513:cfa0:b688 with SMTP id z7-20020a056512308700b00513cfa0b688mr8584863lfd.60.1710722631943; Sun, 17 Mar 2024 17:43:51 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:51 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 12/14] io_uring: remove current check from complete_post Date: Mon, 18 Mar 2024 00:41:57 +0000 Message-ID: <104a91a59f1c3a1af938ce7c1614f4a3e659bef3.1710720150.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 task_work execution is now always locked, and we shouldn't get into io_req_complete_post() from them. That means that complete_post() is always called out of the original task context and we don't even need to check current. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/c6a57b44418fe12d76656f0a1be8c982f5151e20.1710538932.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8c485bcb5cb7..0b89fab65bdc 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -978,7 +978,7 @@ void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; - if (ctx->task_complete && ctx->submitter_task != current) { + if (ctx->task_complete) { req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || From patchwork Mon Mar 18 00:41:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594909 Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F0392209D; Mon, 18 Mar 2024 00:43:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722636; cv=none; b=CXxiGCP4tdbk1O+GAb0StcTDKRYMy6yDqDMuLUswCyx8kWJvbixweCQcsz5pIXRrJm88YQYKmK+Bo6I/PMLqOde0djiBb/tN4nCDEPUotRTSeNHkbwt6WZNwMQ9Dbg6UI0JkKDc1/dEZihjk+1DghLuDBvCABQOVpYcyS1Xb+i4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722636; c=relaxed/simple; bh=tspyoNcqRcJMnQG7roFhmaTi2ZGh3rmyISZ/OP7BKQI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LzsdfbJo6Ku53/ATBFnZgmBo/U2XW9/tkCSL9M7viKpDPWg10lClXSUr05eXb8dTN8sGIWdhQo91NzsWCgBlWWjFuf8JgOhdhkpYaqxIT6iB40DCdzzHfoPSWY35HVtXoU7BCubwaB1NRM0y7VxajqScOkj8RHjugn6ojMZ8fOI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XZP7rEbg; arc=none smtp.client-ip=209.85.208.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XZP7rEbg" Received: by mail-lj1-f181.google.com with SMTP id 38308e7fff4ca-2d21cdbc85bso53744641fa.2; Sun, 17 Mar 2024 17:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722633; x=1711327433; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w8EA2C18hEB59l01Sz2plpqRO9/27NRPOgErU8VDH/I=; b=XZP7rEbgpJR4QiVm0p5IM2EloosHCAMl8fl6QUrtJEdsnWUPb9Gk5Ie+KUqXI5197l C79qfeKaIAkJTiOoNeun34f+VkGBIugaAYlL0yJQqI+ZO/4+ra2LoaIfeDt44rNjk9Be YUL8Gkd7nIPxwieJ4uEaQtvXlVS+Zwjngpq61ktlGr20FJxYXKhOHQw01c1DFfDwE7gA R72i5kGOfb7VkU85eIccdf6utixrsXTUMy0L+SlUCyzdc0xHty8pXq17Td1Vcd8/CGeY 6C4m5JAR5QVW2jXcdBpZXgqEMvzGT13RxEp6vUOyERXT0+MPUcMMvCO+DIEeCEG0tu7n lTDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722633; x=1711327433; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w8EA2C18hEB59l01Sz2plpqRO9/27NRPOgErU8VDH/I=; b=Ta56gSlnd1jVnO8IVLxgMzV2asaDSsHomXd5el6daTFNUfgIuEFUt93br5kwKpSP6M f3l8F+Fbs/BvoxNAD7vLXXCvuFhHbUpI0cQOz5BTdXZpm2lv7yQivO2pWVHus1ysXX4T aAVoBwtkcumBOJ4NFZq2TwY9xtzkE2PoOhX6LRXfjE6S77DybDVs9MAZjllWSW+UvZ6r olHWYtsFM3+FJVJSzcQ1rSYAsXxz6h99MutuIql445z39H96xCDG8xyFwCUwxRwHcixG bJpvMwNKGmPXo6+EOeukVHga6a+jR1EyrSCMv6/tK0/55VKxCvZOtSR6OatgjQe//Pun 3Erw== X-Gm-Message-State: AOJu0Yw+3wmSKUMQJLeRurvOovbPLBXD10/jvRhDQheMpC7gd5x/ZyM0 vZZAlodRBc3raTHLiMIArP7FOKOmLnVgHWUDlmjtN3a18I6Kh3Sz15M8bZWf X-Google-Smtp-Source: AGHT+IFBPp+zzvTY1fqaWU4tJrYl2jx4TD5WAnIVfE6WyWEShOnJG10VdmeG7HEY09Q+uOcZ0RqnIQ== X-Received: by 2002:a2e:981a:0:b0:2d4:5850:28e8 with SMTP id a26-20020a2e981a000000b002d4585028e8mr5759509ljj.26.1710722633344; Sun, 17 Mar 2024 17:43:53 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:52 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 13/14] io_uring: refactor io_req_complete_post() Date: Mon, 18 Mar 2024 00:41:58 +0000 Message-ID: <5ac955f317b2e57bec3d3ced95e59a1b08cd953d.1710720150.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make io_req_complete_post() to push all IORING_SETUP_IOPOLL requests to task_work, it's much cleaner and should normally happen. We couldn't do it before because there was a possibility of looping in complete_post() -> tw -> complete_post() -> ... Also, unexport the function and inline __io_req_complete_post(). Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/f0d46b81e799e36d85d4daf12e2696c022bf88fb.1710538932.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 29 +++++++++++------------------ io_uring/io_uring.h | 1 - 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0b89fab65bdc..68c69b553b17 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -926,11 +926,21 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) return posted; } -static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) +static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *rsrc_node = NULL; + /* + * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires + * the submitter task context, IOPOLL protects with uring_lock. + */ + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) { + req->io_task_work.func = io_req_task_complete; + io_req_task_work_add(req); + return; + } + io_cq_lock(ctx); if (!(req->flags & REQ_F_CQE_SKIP)) { if (!io_fill_cqe_req(ctx, req)) @@ -974,23 +984,6 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) } } -void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) -{ - struct io_ring_ctx *ctx = req->ctx; - - if (ctx->task_complete) { - req->io_task_work.func = io_req_task_complete; - io_req_task_work_add(req); - } else if (!(issue_flags & IO_URING_F_UNLOCKED) || - !(ctx->flags & IORING_SETUP_IOPOLL)) { - __io_req_complete_post(req, issue_flags); - } else { - mutex_lock(&ctx->uring_lock); - __io_req_complete_post(req, issue_flags & ~IO_URING_F_UNLOCKED); - mutex_unlock(&ctx->uring_lock); - } -} - void io_req_defer_failed(struct io_kiocb *req, s32 res) __must_hold(&ctx->uring_lock) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 4bc96470e591..db6cab40bbbf 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -65,7 +65,6 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); void io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); -void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); From patchwork Mon Mar 18 00:41:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13594910 Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A24C0224DD; Mon, 18 Mar 2024 00:43:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722638; cv=none; b=LN7BntJrjC0gg2jknYWAKXoWqsrqD5kVUo8YFJUnmvmtgIAbOi3RJOnuoHfOLP1M8EeUOHK8/XqSCsnJtSgaD2rm+iLJB2Ay5pPBcNN615YuU7Zmqw5xqIGPAjTq67do0NICvCeOF8eKIJqUysnRfmBfMVRBsdEBcXreFzvR58s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710722638; c=relaxed/simple; bh=bNt8jz/8WnJqhRG/3PrzblUGUleUniIqAeDbF11GcI4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gHMRda3towtltBwjUY+eLVyHkgUXlWILfWfJRYfRQxys5721AD6xprg/ioa1m9u/srqGIiHGXllrckkuqh06aCn2iD/7Qa4QS/5CvXojs3zRaISNkh3GjZACyfebLNsI12zhMb+VyaV8QHpf7WrfDIOI6O0+pcpM5RzGXEB4Opw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=k7QK03Tc; arc=none smtp.client-ip=209.85.167.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k7QK03Tc" Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-513cfc93f4eso4275165e87.3; Sun, 17 Mar 2024 17:43:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710722634; x=1711327434; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aUxb5daalxdf9Y1mQ3RaNl2uAIpS4+M356xPG6h8qG4=; b=k7QK03TcAcWyWTwRNttqO8L3IYARVLkp0IrgOBZg7aB0ETZkDWGmiyGTtNM15REsI5 wPUER/VqCM32rjQIs98rYNiR76mN8xTwYRFeaKVTD9soiERiFF1xQ7qA6uD5lJ9UiXit X0b9GUvnTC7xWaanarhniLmuMAQ2svYVUxff1ynbFabZHeChcyWxhqESMxmrHUiUQxiz nnJCPFM8wBg0tLmHQIrPq7NBCHqmWKGvtU2oSRwJrQByRjX74pH7UTcHUOYpE0Lv3hl8 OjmDnDItQ4JaI7iaHte8kQCXso/dNZqk6HmqMmgOfQZyOpQFh5TU4Wj7UGns2ELy0erZ us3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710722634; x=1711327434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aUxb5daalxdf9Y1mQ3RaNl2uAIpS4+M356xPG6h8qG4=; b=rkMn3LHuqSQiSIo43H+SDkiIJj3RaDABGh8eEoce2bUeVUsuuNQJp5ZMn005KY/h1C w5hApOcu+9g8loA774S6LPfUk9S9OWcniKv67Q6RF8pDyPEfVm/bgO9TBY6TIzkVj0zQ Y+awRtztv5e9ZqGNt988dqAgQnIptPCKzRgU57z85a0AidjTGGH5FgFjMyjovSQKusEP rP1v81oDjPYA0GFhb2RqeRU+ziNT7ey8m3XNHAQ9V8BcCw6Z4NY7XPlKxvAJrAoEXULN fIEcxKijcwGh0xCnjqhnloFChXwT/0vAHmpLRdRHtYSE474W+yjZyusOKjakJqqrXnqt OtXg== X-Gm-Message-State: AOJu0YzZ+MWe8fTdhydKTQQh9/Md8eduGmMju91nrDWVPUGbLY4C+nTj BEwE/tXyhkB7kV9A4BZo+Gr9nQ8TosvsM8ZAGrqxCvKKdEWUXOw1REII3gwT X-Google-Smtp-Source: AGHT+IGBHmLq2pEncWA7qUyFEC5gqFOgsLJpmA1AXXRQO9yGBSPKq71rUTUykihkLHm69UpjhU6iPg== X-Received: by 2002:a05:6512:3447:b0:513:9ae1:4708 with SMTP id j7-20020a056512344700b005139ae14708mr7097539lfr.46.1710722634364; Sun, 17 Mar 2024 17:43:54 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id p13-20020a05640243cd00b00568d55c1bbasm888465edc.73.2024.03.17.17.43.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 17 Mar 2024 17:43:53 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v2 14/14] io_uring: clean up io_lockdep_assert_cq_locked Date: Mon, 18 Mar 2024 00:41:59 +0000 Message-ID: <317cb75cd09247efec3628d3cf1a77847cbfeec1.1710720150.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move CONFIG_PROVE_LOCKING checks inside of io_lockdep_assert_cq_locked() and kill the else branch. Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/3c7296e943992cf64daa70d0fdfe0d3c87a37c6f.1710538932.git.asml.silence@gmail.com Signed-off-by: Jens Axboe --- io_uring/io_uring.h | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index db6cab40bbbf..85f4c8c1e846 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -119,9 +119,9 @@ enum { void io_eventfd_ops(struct rcu_head *rcu); void io_activate_pollwq(struct io_ring_ctx *ctx); -#if defined(CONFIG_PROVE_LOCKING) static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) { +#if defined(CONFIG_PROVE_LOCKING) lockdep_assert(in_task()); if (ctx->flags & IORING_SETUP_IOPOLL) { @@ -140,12 +140,8 @@ static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) else lockdep_assert(current == ctx->submitter_task); } -} -#else -static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx) -{ -} #endif +} static inline void io_req_task_work_add(struct io_kiocb *req) {