From patchwork Fri Mar 15 15:29:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593650 Received: from mail-wr1-f45.google.com (mail-wr1-f45.google.com [209.85.221.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FC1F3D396; Fri, 15 Mar 2024 15:31:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516674; cv=none; b=pYCidgnUvJO5YQS/5PrhZfgc65d74HnjCLJHIIjgyN4SWg/VAhSvSG6yF/YY+9ousyXltsYSptkUQc6GQF2/xUIUoD1a2OrJxKp34rgzfAsyvTckOOdykvhIp66hlxm6hjDSkOyL0SMnQTfIyynBTAlPgTliFAhhB4cidZ/5ELQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516674; c=relaxed/simple; bh=oftQEKh345/gTaOfwh9tZhbjwTcfq5CN3JVoPDNxuMA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s0duRztsmOlx41F0JKe13uI/LMMLh+lovdXxl+tr2xF5/kRp+fObJthfC9BMdF5tPzsM5hXpxXJeTfOiGkXrLwNGhvykn1l86+poccSnKugCLQOdfg+Bd7Pyala+4U0TllyR84zNW82soWspBDqqYbxvbrz/nXrKzVbidHW0zrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q+K/5dvd; arc=none smtp.client-ip=209.85.221.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q+K/5dvd" Received: by mail-wr1-f45.google.com with SMTP id ffacd0b85a97d-33e1d327595so1463936f8f.2; Fri, 15 Mar 2024 08:31:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516670; x=1711121470; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0Lql2IHeg3l+zlV9/JRjb/JI8yw+EcE5DCy9istQ0cI=; b=Q+K/5dvdwd7+rlbeJldswHUq2CYOfb9Nt5LJZvk4OAcgq64hRTaPWPvcdA9F1xO+JK cA1OTM6TJC7ZgajgLtslBcXgn7gMyd4wgdv5bqcVzvlyxqowW0K8XqbBBK44s8or3zLg P/tuV80xHDFKQIfMEHhFgQlYH3yHkIDlbdRl5E+E08oWhdVgOW18Sr7qmZUj8IB8g/qA gfGo5k5+MGGCgXTL9cDROIN1jbgogojr8ABfLohsVtKHEXRv4zuLA61UfnKvTC9Uxkgc hYUfuAQbECdsvgeKWvkwDH/uLlwWmPrubV7/B7eBQmwOh2Dnv7sXhAw1T9goZZ78YUFh 0qlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516670; x=1711121470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0Lql2IHeg3l+zlV9/JRjb/JI8yw+EcE5DCy9istQ0cI=; b=n/9gwrfWm1PcHDWs331mNslK9mDsa+jUNygl/ArsNteTptFVAjoh+twxKEy6diaOjC lXutHX/2eO0Bues7qAvvK3DikT95myDF2YLmdhjPVJUY+iB35vtuQ6Otp1Oecv2XYhle klXDmuybCyRme4UrgRt9IEO1RDLA2pUdlAxcvDW1BaqH+AYMBMajuRwotF8mHyDvPFur 2NCGnifQb838EDn/mA7hOFhw+jeljVFBJEk3mbb6/o6k1AnKQW9Zj0AW4egTSbCqLCNZ MNIglZuzpm8Ed4W6p2wfvAU17QOrDz4U+s33TSHmcmmLqdO0uekeIJxG8ll7DggPsnY4 chjA== X-Gm-Message-State: AOJu0YxxHamDz3FpZt81PSs0UId1gskCB9VSEvEq1DyDVytsw0Xlm2JT y9UGYpJyEY6GogikNFxBTDxMhZbW1ToOXZgAjCjs3cvF58KxOA+2ZTrWfLdV X-Google-Smtp-Source: AGHT+IFNyaqBSE9MwjIzQU87Q7dsGeCH05fauXIAogeW3Q0L/1Cvh4D/iCk2oyVNfpZmuHVL64dxpw== X-Received: by 2002:adf:f60f:0:b0:33d:2180:30e8 with SMTP id t15-20020adff60f000000b0033d218030e8mr2839231wrp.25.1710516670298; Fri, 15 Mar 2024 08:31:10 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:09 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 01/11] io_uring: fix poll_remove stalled req completion Date: Fri, 15 Mar 2024 15:29:51 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Taking the ctx lock is not enough to use the deferred request completion infrastructure, it'll get queued into the list but no one would expect it there, so it will sit there until next io_submit_flush_completions(). It's hard to care about the cancellation path, so complete it via tw. Fixes: ef7dfac51d8ed ("io_uring/poll: serialize poll linked timer start with poll removal") Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 5f779139cae1..6db1dcb2c797 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -996,7 +996,6 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) struct io_hash_bucket *bucket; struct io_kiocb *preq; int ret2, ret = 0; - struct io_tw_state ts = { .locked = true }; io_ring_submit_lock(ctx, issue_flags); preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table, &bucket); @@ -1045,7 +1044,8 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) req_set_fail(preq); io_req_set_res(preq, -ECANCELED, 0); - io_req_task_complete(preq, &ts); + preq->io_task_work.func = io_req_task_complete; + io_req_task_work_add(preq); out: io_ring_submit_unlock(ctx, issue_flags); if (ret < 0) { From patchwork Fri Mar 15 15:29:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593651 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF2E11946C; Fri, 15 Mar 2024 15:31:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516675; cv=none; b=g7PGGuKUZhJekexpn2P0Naj/hfZXYoYEjbFwV/YReE0PzjdbNcbSCGhP5lvwx10+PBUg/uRwswYRkYXdV+Btwwco1qUjdJWWRybz8zJ1Wnpw2lkNDCTqgySGNWeizw6e8SyyQ+Mva4EqzGX/pVcfZNcOOxfuuB19qlSs5QqCeXA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516675; c=relaxed/simple; bh=cC9SsXnrQzN+jeTDnCngPu75SO4LrcOOTxH+MZXD7fQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UlcjCAByahmRVELjHPWc0QIjJcLv92OlhWAquSAr2DqGdjRgVnDhW0B2iM8nm01seQNT9TC0IeaAxgBTMYDETRkJx5NQt0/BI20Sq973AoOsTeI3fc0q3q52PPPsub/YGNHmJRbVCU+c50f0HF2RtILQ178td7gJjrLFZ52W+ww= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iD+lYkd1; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iD+lYkd1" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-33ec8aac149so1013829f8f.1; Fri, 15 Mar 2024 08:31:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516672; x=1711121472; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8S+2Qr+5j+oXYUgnnJL29zNsQhxWk60nTT2RQa3fYOM=; b=iD+lYkd163gUJ/7v6TqwOzfp2AccRYIfmbBvMqwhUKI+vfV8RXRMqIeEIRtCAJHfED J9mAfwwFthx6B9XUaq2Vz7ezt52uUIS3B3Zxuv2/iWUOMQJ4S3/qBOtanXuYKTiqXcbX vGf8VAgy2rdtHXkhlZIiv25gLFrz4Tc21YGxC6EzKEyqKhbPdcSnrGWOHbUHjnTgQYHJ zU8dts8hv12WHHEMGmG+sYDjmY2eYQ3jjl+oigWG8E9VyMGqyJTeai12l7wiFWRqsL1Z RsjHwjbWbngz0rI3OHgNm693yGxYlEWp8+4Th5Fs6zNriWDezaPwUV2+hq+3kDprtCmt T7PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516672; x=1711121472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8S+2Qr+5j+oXYUgnnJL29zNsQhxWk60nTT2RQa3fYOM=; b=l8WoRl7ybB/cdnbDwM1WWsnvQRVdn6hnhxzabdK+yZcAdpBuHw2NjKc8+UdtJAGd7j HeEEZR9jakggzgqueJ7VKC5Xf7uAf5S0Wf/x9H3Q4Md7M8ShhULXaKssH9bmNsVJtS6K xxkR/DngJSbzSvuLEPt193wScl3uK/cSNIaGyZasURje4Q4jak+1yrmHH7Dght8fkbJz 3G+1Aau//I0xlbgNjSdpKbCQKhJLVT9LeZ8GVsd9jin4MYvn5pVRNakloTWoit0vPMUq VeL85ky7Y0gRQWWOICEaHcwOQJ3En3d3K/E4XpklMH3kOWOdXB1d3w4MTeXQSPEBOYTr CRbw== X-Gm-Message-State: AOJu0Yylg4SP9YyG8qKAGQGjftqWIwcAUJUE9UUsoc5SXOcrYaZxIDV5 nCWFGdzigGYAAdx8s47Rs/d5mFrbZn4EF1IW8+VdjBPQab+1yTbUrd6t84TZ X-Google-Smtp-Source: AGHT+IFlDr1gsoKLBk+XeXsyhmacreCOtvMrxLLkiZMVEw9qddNM8m4PWMIAu1+gC9gJTcfUqS4e7Q== X-Received: by 2002:adf:fe49:0:b0:33e:a0b1:e783 with SMTP id m9-20020adffe49000000b0033ea0b1e783mr3824574wrs.22.1710516671919; Fri, 15 Mar 2024 08:31:11 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 02/11] io_uring/cmd: kill one issue_flags to tw conversion Date: Fri, 15 Mar 2024 15:29:52 +0000 Message-ID: <7f0d5ddfb5335d038bfd8db50656a1d69daed37f.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring cmd converts struct io_tw_state to issue_flags and later back to io_tw_state, it's awfully ill-fated, not to mention that intermediate issue_flags state is not correct. Get rid of the last conversion, drag through tw everything that came with IO_URING_F_UNLOCKED, and replace io_req_complete_defer() with a direct call to io_req_complete_defer(), at least for the time being. Signed-off-by: Pavel Begunkov --- io_uring/uring_cmd.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 42f63adfa54a..f197e8c22965 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -100,11 +100,11 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); + } else if (!(issue_flags & IO_URING_F_UNLOCKED)) { + io_req_complete_defer(req); } else { - struct io_tw_state ts = { - .locked = !(issue_flags & IO_URING_F_UNLOCKED), - }; - io_req_task_complete(req, &ts); + req->io_task_work.func = io_req_task_complete; + io_req_task_work_add(req); } } EXPORT_SYMBOL_GPL(io_uring_cmd_done); From patchwork Fri Mar 15 15:29:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593652 Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FA2F3D396; Fri, 15 Mar 2024 15:31:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516676; cv=none; b=VEiHtWr/x0O6orC8XoSEljZUAfl6kremwkLqlLrJ2xYmJd+s6xBjkh7F48wqDWaxkWiw6OXY294IQ4G09BgcHyT4o7vv//062Irz0BHPsRi+97uRkHzZOR+rM1cJPJxpeifIeA1XjWOfaTuSaQauVElbt/0luIehBUIW2VsUCOc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516676; c=relaxed/simple; bh=kb6QtRhUVQyZegxltTMy3UbJXD0nOnBpiAusxJ5t7YQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EpDmhWVqEghr7FXDKl+neBQ7LhhCgp1a0di9IndLzr+0m6q7EMgiX1O9lreCamIjKXkvcl8CJuQe+QFVtE8ekB3qMBNVapQPjrvjYRLqa2El+9oE5DsfFIOiEf5h78bg++G66kcn218vt91BsxH2SaFtDi18e/yfhy2eexYOW5o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ePnGu9aH; arc=none smtp.client-ip=209.85.221.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ePnGu9aH" Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-33ed4cf02ebso383567f8f.0; Fri, 15 Mar 2024 08:31:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516673; x=1711121473; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=S+TFOq4id2xuuzz0BwmgK+/9Jeh0/lQJu3EsML3ViLg=; b=ePnGu9aHzlait8MC3LNlQ4KI3pBpbEBFf/kBAlAYdA+3sdx8jlM1USvFJ7/YhnTeXS qJmo3+cUb6fD2F66Q3ZBYG5yJNFaTl8dmcgF3JjlrdswQ74KSQhmzfoE2CmNUJ6dJ/eC WUL6YLXDa9Gnr7rzPuutVNZVZ3V4Nsu9c63qNuBpjYK8oPwAK51wD7iU7Wyt6FbhCxsf gZ8skwJWCoZDZbR1rXwkFbgSsJczLiCUubqx+4yegkPkJA+uKmRw/QdXKoJjeh9fw1zr S24DbHZl4HBOR63LVnzZG0fdG5YwOii58h0z5LRehKlSKAUuSkwoQqJCQ/l+vU3X4B1+ ehfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516673; x=1711121473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S+TFOq4id2xuuzz0BwmgK+/9Jeh0/lQJu3EsML3ViLg=; b=ppE0oOivHq6gSgt/6Uw8yB4zg5EJ6S18QsVL9Uyy4FPkqGAs2NIWKR9SZbDutr18k0 6UkeRAdUSNNYVOhvr6sGHp/xAzaZbjNR6VITxCPoyu1NjIuz2XMEFJsvbSUExXJSxUMD wJtMhSgvNuU8GwhVcDWHguldeEzOLY8YyDLfdAz7mvh1QWd787XTbx93m+q1f/R3MFvs kCxou6wuIdUFMS7ahTX0nk9huTLaNKqcEjSmd+3VUND4DoWcLelwKHmL8Q0dQjumeTtL 5LUTwyK8pig7m3BugYLrWv3cjwn626KFK4Dr/vkfZ574Gw1CtYqGf94/GkC6yqND5ZrJ Xv7w== X-Gm-Message-State: AOJu0Yx4rtGzUl5JaoNhVSXqNhQvpMg/OqXvl5VrJ2+RbquYCgT72xYI whJWNpyMamQtKTeumVJYaliU6IZ4hg3b6+RP9oziimx7NWk5cGMuzJm6leb+ X-Google-Smtp-Source: AGHT+IHfgKJWxAwalu66ZuDaaVQMQ9FuFZrNyC4CC9upS2iDPZbQx0FPq4m/CrVeKjTSmbsYXsIDSw== X-Received: by 2002:adf:ec11:0:b0:33e:1a98:46e2 with SMTP id x17-20020adfec11000000b0033e1a9846e2mr3706177wrn.28.1710516672915; Fri, 15 Mar 2024 08:31:12 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 03/11] io_uring/cmd: fix tw <-> issue_flags conversion Date: Fri, 15 Mar 2024 15:29:53 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 !IO_URING_F_UNLOCKED does not translate to availability of the deferred completion infra, IO_URING_F_COMPLETE_DEFER does, that what we should pass and look for to use io_req_complete_defer() and other variants. Luckily, it's not a real problem as two wrongs actually made it right, at least as far as io_uring_cmd_work() goes. Signed-off-by: Pavel Begunkov --- io_uring/uring_cmd.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index f197e8c22965..ec38a8d4836d 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -56,7 +56,11 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; + unsigned issue_flags = IO_URING_F_UNLOCKED; + + /* locked task_work executor checks the deffered list completion */ + if (ts->locked) + issue_flags = IO_URING_F_COMPLETE_DEFER; ioucmd->task_work_cb(ioucmd, issue_flags); } @@ -100,7 +104,9 @@ void io_uring_cmd_done(struct io_uring_cmd *ioucmd, ssize_t ret, ssize_t res2, if (req->ctx->flags & IORING_SETUP_IOPOLL) { /* order with io_iopoll_req_issued() checking ->iopoll_complete */ smp_store_release(&req->iopoll_completed, 1); - } else if (!(issue_flags & IO_URING_F_UNLOCKED)) { + } else if (issue_flags & IO_URING_F_COMPLETE_DEFER) { + if (WARN_ON_ONCE(issue_flags & IO_URING_F_UNLOCKED)) + return; io_req_complete_defer(req); } else { req->io_task_work.func = io_req_task_complete; From patchwork Fri Mar 15 15:29:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593653 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FDCF1946C; Fri, 15 Mar 2024 15:31:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516677; cv=none; b=vDTvMbOB/olAacp2rIFeD+uCuvfvlVelSkkv386iN8f8HPjSnwr6j0OoeSKkC2lzMJZxjKflaZRyG3Cik7NF+ie9MML3pJvW+7lqZWtkAkarktm2MXxSHde8TBo6PNyL0sypcKK9FbxccvNHtkdPWFTzXdTlEVpMronooeKq61U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516677; c=relaxed/simple; bh=zAf6c7/XJ1nhbxWUTxcMM5i8EgkOlmQJlR+NVc+j6Jg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cIATBQhxKn/Y2bsOa4WSfwINKVbwn7gubRzaBc3svl6lyyHv8zim7iTZ/GwfDKaDSdEhcBiGziOrJ0Bsj4iwLsVx6LAKtiMglpVBGY8Ex4PAk0YOKvzFiipVxKRTk7/PaSdlfyeb9FegneD0G9WKhZljNAbCUdVngGwV0MkpU2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Hu1df6+u; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Hu1df6+u" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-33e966f56c7so2310990f8f.0; Fri, 15 Mar 2024 08:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516674; x=1711121474; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+UMgC0mUDzTo76vypApcjS0mpEbmkLe5hnsQfuh8pPk=; b=Hu1df6+uh1dVgZeaotiD7pgDiSvTOv/LspDdtMMsdCCBC2uPNF4S5vJRzKnwCjgS4o PXAPyIv4BL2Ka3wt0Vx66S883FGfW6Wzslf4V/7MgJ6aY5ItjBC4N33OmlgjCFC0GIcK 8ppTCJTaFTK4iSd0ziAfOP8VNGxKvm5b1cIFSM8dmn3zv4lQWzNlCaKSMWv684SqwigG i3mqVLGeBb7nQDN1jZD4IjlBXVEgJM4eUGxIukWEyA9yHHuQxhyGGPkQv5Qq7/5cLcGD lGQ6dPX4gdse9FOfjFKwU2V+k+ovHQ5R0CTQ3kzKML50NotWrhw+JYczasy2kCSdZLOd DUZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516674; x=1711121474; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+UMgC0mUDzTo76vypApcjS0mpEbmkLe5hnsQfuh8pPk=; b=BfAtBa4NOgv7s9BY+lD+9hSYTrno+5vhKxMCsRDokmhqiB/hfYu2NABtsaDFETjVeO VRecq9jIUuB0rpucPzqUowvOCokCSTTB2jw5xO1ogjyuEOP/68l6ZCcK5sCBDe5+UbKs nDx57k+8vc0aazzF/x3KljTusNvcUWd8yjWbGXG6zZQFhcIlGzqY7UBkzFnxVQuJQ7vj dAIGQA3kwvxctScS1OMyPEYvw9OM5bE0OAYV8ToMsNsH2KUpBb1zBIG6qDPTB87cySTG ZnMMPMje+92q8Zmif+ZsUBibhcuo6EDKhX5wNG/JFhUBcDU/xl6NFi8M93FSUybmmKfj F2MQ== X-Gm-Message-State: AOJu0Yxbm2MwjDiEqrC0sgUiu2L2gySFieOMXy4FwUhvPbJLdM4vDwSg wpVdLfSqfsY9jmfOVy5HkY5Lnx6FzbFig6YfL18UM/ZWIn54U1s2Fly7m5Cw X-Google-Smtp-Source: AGHT+IGQ07XO0QxIj+Ccxxec6obPCN59IwaNJeiNvnUv2xQot8JuiSz3IToMGeDq4rxk75UnyrXZFQ== X-Received: by 2002:a05:6000:1103:b0:33d:73de:cd95 with SMTP id z3-20020a056000110300b0033d73decd95mr2988774wrw.17.1710516674298; Fri, 15 Mar 2024 08:31:14 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:13 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 04/11] io_uring/cmd: introduce io_uring_cmd_complete Date: Fri, 15 Mar 2024 15:29:54 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring_cmd_complete() does exactly what io_uring_cmd_done() does, that is completing the request, but doesn't ask for issue_flags argument. We have a couple of users hardcoding some random issue_flags values in drivers, which they absolutely should not do. This function will be used to get rid of them. Also, add comments warning users that they're only allowed to pass issue_flags that were given from io_uring. Signed-off-by: Pavel Begunkov --- include/linux/io_uring/cmd.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index e453a997c060..9cbe986eab7d 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -26,12 +26,25 @@ static inline const void *io_uring_sqe_cmd(const struct io_uring_sqe *sqe) #if defined(CONFIG_IO_URING) int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); + +/* + * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd + * and a corresponding io_uring request. + * + * Note: the caller must never invent the @issue_flags mask, it's only allowed + * to pass what has been provided by the core io_uring code. + */ void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret, ssize_t res2, unsigned issue_flags); + void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned), unsigned flags); +/* + * The caller must never invent the @issue_flags mask, it's only allowed + * to pass what has been provided by the core io_uring code. + */ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags); @@ -56,6 +69,17 @@ static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, } #endif +/* + * Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd + * and a corresponding io_uring request. Similar to io_uring_cmd_done() but + * without issue_flags argument. + */ +static inline void io_uring_cmd_complete(struct io_uring_cmd *ioucmd, + ssize_t ret, ssize_t res2) +{ + io_uring_cmd_done(ioucmd, ret, res2, IO_URING_F_UNLOCKED); +} + /* users must follow the IOU_F_TWQ_LAZY_WAKE semantics */ static inline void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned)) From patchwork Fri Mar 15 15:29:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593654 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5A6513F9FB; Fri, 15 Mar 2024 15:31:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516679; cv=none; b=jEjH+0BenJpw3fOBvgKDnEMzGC4sL4HxwfP4AgM+7MHVsVk78KOknyj27HYTEfxWjooagE3Zd6cg6pI1n5o4QP6xAV/PRZzT4Vs1fgv+Y7Oc9L1S57446d5TnZIr+1eg6W6n8OWQfrAJf2pC3FI3qcsPoGNmY5scIlkGxhZhZdA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516679; c=relaxed/simple; bh=cgqvBTb7ybCJdaMD4fjwY2yF0dr8IFrOj9E5bPJqda8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PoXj+ur2A1Tpm3jCeHxleOgsTXEwNaLzv4BTg29ZZClQz5HKyUnK943nTYWZHxxsgsz1y4ohKRVwMS/vtOuPWg7adZ0WIEiXYCUYyktWVj3JR+RhEA4Je07729nPGjofv8MzY6IFJ9ed2KYLmeffW4Q8ZervKRgK9gPszoAfsMM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GtdAT4Re; arc=none smtp.client-ip=209.85.221.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GtdAT4Re" Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-33ecbe78c3fso932548f8f.3; Fri, 15 Mar 2024 08:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516675; x=1711121475; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vCYhmaYi+jcvj1JXX6M48ExoNcATA43GN1CGszIjmcE=; b=GtdAT4ReFlaILOu2czTwZNWaKMRXmBbb6gAO2sXxCPUlpd966W/fmhHHw+xMRabG3V JfGZGRbTrTwdKANf12JZ3V3/eiF3Qx6vRG5vZ26vHJOAnxfsiuBIVmm9NpPZ29pee/lH KZq1zeUkypaXM/rwQanvZhh93Nd0pII7AzCIrAXdxsxi3yca7So/WXKLAYMbsTIIrO5p VETI5R5VkegAhNrZfJigVH7ODFh5fTNZwl2htIdOhHNGS68o31o401LxXHSdtlKNVu8S b4zD5q+Eo+jaolfkCR5gQw8NbI8WyW5GsHeu8ANDFong/KlTUkFxXjuuKoQOHBk6Myh1 BQnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516675; x=1711121475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vCYhmaYi+jcvj1JXX6M48ExoNcATA43GN1CGszIjmcE=; b=odUeoZifTY9vhezWKCpiDCeKSFSY5bafvHL/Z2R8rfUq5Hx1hCMVddXb1Eht0MvEDM TDXvMzdoqqFmNtV8BZ2YpaE8qN8A0OMbX7eNwpZMghcP+/IOgHJ9iKJEZBGFEupb1QJw iMCdqlQA83Kd9/GIKcuc+XqEfRsY8Q3PhxPMrLB7yjLlmg74rlt+UoLWL1qi9FGcRWG3 Uqv8497+qMe4kmx+k82iGRXILKSYs8UXnfyf7VOmoZdbBfyDge4q/pxJJfSqLoYldkdn QVBpcqzrftntP6Lhm0RxtCJhc6YhLdqvMMUzH4pCIxq7ilTB5x11l+3gR7N7wDSZANtq O72w== X-Gm-Message-State: AOJu0Yzro+WPMWAGi1fLNQpwDOMoDdSNE1BAJXEeWDqw+//FATO3quew n1NtfYx49m11gnM0SCLjKmDdS7ybMCjZy9LFDahEm4s8cB2Tjm1fTZiQe2U/ X-Google-Smtp-Source: AGHT+IHIvgy3lJFevAdRRoCyzphrATiOEMWCxY3BUCP4kdlBZy9QkawZBLa/prD5rMkXWJ3V4A7VVA== X-Received: by 2002:a5d:51c3:0:b0:33e:7f65:bb7b with SMTP id n3-20020a5d51c3000000b0033e7f65bb7bmr3847356wrv.5.1710516675305; Fri, 15 Mar 2024 08:31:15 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:14 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 05/11] ublk: don't hard code IO_URING_F_UNLOCKED Date: Fri, 15 Mar 2024 15:29:55 +0000 Message-ID: <2f7bc9fbc98b11412d10b8fd88e58e35614e3147.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 uring_cmd implementations should not try to guess issue_flags, just use a newly added io_uring_cmd_complete(). We're loosing an optimisation in the cancellation path in ublk_uring_cmd_cancel_fn(), but the assumption is that we don't care that much about it. Signed-off-by: Pavel Begunkov --- drivers/block/ublk_drv.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index bea3d5cf8a83..97dceecadab2 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1417,8 +1417,7 @@ static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq) return true; } -static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io, - unsigned int issue_flags) +static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io) { bool done; @@ -1432,15 +1431,14 @@ static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io, spin_unlock(&ubq->cancel_lock); if (!done) - io_uring_cmd_done(io->cmd, UBLK_IO_RES_ABORT, 0, issue_flags); + io_uring_cmd_complete(io->cmd, UBLK_IO_RES_ABORT, 0); } /* * The ublk char device won't be closed when calling cancel fn, so both * ublk device and queue are guaranteed to be live */ -static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd) { struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd); struct ublk_queue *ubq = pdu->ubq; @@ -1464,7 +1462,7 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd, io = &ubq->ios[pdu->tag]; WARN_ON_ONCE(io->cmd != cmd); - ublk_cancel_cmd(ubq, io, issue_flags); + ublk_cancel_cmd(ubq, io); if (need_schedule) { if (ublk_can_use_recovery(ub)) @@ -1484,7 +1482,7 @@ static void ublk_cancel_queue(struct ublk_queue *ubq) int i; for (i = 0; i < ubq->q_depth; i++) - ublk_cancel_cmd(ubq, &ubq->ios[i], IO_URING_F_UNLOCKED); + ublk_cancel_cmd(ubq, &ubq->ios[i]); } /* Cancel all pending commands, must be called after del_gendisk() returns */ @@ -1777,7 +1775,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd, return -EIOCBQUEUED; out: - io_uring_cmd_done(cmd, ret, 0, issue_flags); + io_uring_cmd_complete(cmd, ret, 0); pr_devel("%s: complete: cmd op %d, tag %d ret %x io_flags %x\n", __func__, cmd_op, tag, ret, io->flags); return -EIOCBQUEUED; @@ -1842,7 +1840,7 @@ static void ublk_ch_uring_cmd_cb(struct io_uring_cmd *cmd, static int ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) { if (unlikely(issue_flags & IO_URING_F_CANCEL)) { - ublk_uring_cmd_cancel_fn(cmd, issue_flags); + ublk_uring_cmd_cancel_fn(cmd); return 0; } @@ -2930,7 +2928,7 @@ static int ublk_ctrl_uring_cmd(struct io_uring_cmd *cmd, if (ub) ublk_put_device(ub); out: - io_uring_cmd_done(cmd, ret, 0, issue_flags); + io_uring_cmd_complete(cmd, ret, 0); pr_devel("%s: cmd done ret %d cmd_op %x, dev id %d qid %d\n", __func__, ret, cmd->cmd_op, header->dev_id, header->queue_id); return -EIOCBQUEUED; From patchwork Fri Mar 15 15:29:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593655 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C6A93FB8B; Fri, 15 Mar 2024 15:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516680; cv=none; b=Y1olo9I1GtVK0cOaLIGAEiZv+uWuO8jiT2ItbO7oz+kkyq5elQv9pyzBgiDUnjUJcTEI2jD63KhzP57eCVwpU4uKiwK9WOhiH57hyZejaZcZ5dA9a9BVLnJi6h6miYKYJvbov5iLC5v61ev1etXpTpqAev9LNKI/hyrG993GxS4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516680; c=relaxed/simple; bh=m132cIT+AJFfgzdRRIrfI6J6BFJk9LRkNt+dDeqcOLY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HnUs36+PYo75bwcdgH+63iUWrhd/H+RasnbVM9PPtY8hyapXNY/bkpgaO6JdDqYvDyviqcbwntjlVnxgim2IO3N2nDOC6o2YJW2Xu8iz6a0r2iFzDKipDG7dSrJNoqYFbQM6LODbjo2yGfQmE0mTdv6cCjAbMm0Sy+k6aCx3DUc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lNdJ87nq; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lNdJ87nq" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-33e94c12f33so2010869f8f.3; Fri, 15 Mar 2024 08:31:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516676; x=1711121476; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VXqszJfOgUziV1la4Q1uPQvBAycH5jxAJpYRpwbwMxw=; b=lNdJ87nqqAFN7pnBUIO9s9hfztp/5cDmb5zUzae9/pMwIEeeNTg+HLZk86xNvsrL5J A4WqXNQ0DjPBVK17f6MjD5pEjmlYowGHqvRGGdtpagq3ux0pswjxZbiHi9Kkc7je4CKD MxozrGxVgEN4sLqndseB/Rj7VHybwXKRCM1QIHn42BtUGOWU5NMrbEe6w/HPVWz3Y+I8 YUCI7PoBH28WG2348BNoscdwAS0GGAAM1hfsF1FMKkqVjE43aYIN5vOQAedth6mKyJc7 yKxrN2ufWK3uLY+DxwBIAX7GU5d5L0IPmV01p0PwfexFqfZV6bRkCAg+5ME62lvOES8v R/Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516676; x=1711121476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VXqszJfOgUziV1la4Q1uPQvBAycH5jxAJpYRpwbwMxw=; b=WAq6VteD4o34NwxERjhtT3OBc3B0StrmjNoeRSvd78awuJsJF1tEep+pZvqlcEytbv tPGSJ057yaXbc5cX2Orj11HIPio4fl56rOM+S7G295mov5BWgRztRWDFwzq5j+986l1E UuoAVdAuEmYQjfU5PmNFvfaAHCVSSTVenZeFudgF7xq4m28zeF88CfRf8m7DtDoWMIk7 A2q4dKChBEDWFFrXFuk/fdzsn1c6/+7EmSg+lkl9z3U0UeJGm+5cmn9Kx0h2CNk2lhPg Cd6p69NMa31k0sRfLlHSUuvB0HOUebOgzogefHU3Q+RhiqqlCb9CYuntCI6ZKtzElgN3 uzeg== X-Gm-Message-State: AOJu0Yy8zYWGJjZEIGwPPDc6VtYBcEJ0lj+/iM1OuIMKI6idGCrXwn4D W5/87ZiwGn88DRTrdDsoFWZBUaKJ6HQ9vvYkDH14B0c5h/Mtw3V4kB7psJQF X-Google-Smtp-Source: AGHT+IFl30KyTVkxfCFaaFzBLjCH1fgF0JuRtqETr40GsyTlhNlAXQCR+lE6/d6/8kbeCgFCxY0vgA== X-Received: by 2002:a5d:6384:0:b0:33e:c99a:ce5 with SMTP id p4-20020a5d6384000000b0033ec99a0ce5mr3586174wru.70.1710516676608; Fri, 15 Mar 2024 08:31:16 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:15 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 06/11] nvme/io_uring: don't hard code IO_URING_F_UNLOCKED Date: Fri, 15 Mar 2024 15:29:56 +0000 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 uring_cmd implementations should not try to guess issue_flags, use a freshly added helper io_uring_cmd_complete() instead. Signed-off-by: Pavel Begunkov --- drivers/nvme/host/ioctl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 3dfd5ae99ae0..1a7b5af42dbc 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -426,10 +426,13 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, * For iopoll, complete it directly. * Otherwise, move the completion to task work. */ - if (blk_rq_is_poll(req)) - nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); - else + if (blk_rq_is_poll(req)) { + if (pdu->bio) + blk_rq_unmap_user(pdu->bio); + io_uring_cmd_complete(ioucmd, pdu->status, pdu->result); + } else { io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb); + } return RQ_END_IO_FREE; } From patchwork Fri Mar 15 15:29:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593656 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62D5D405F2; Fri, 15 Mar 2024 15:31:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516681; cv=none; b=LzMUVAsG258kU3+53ShL+Nyov4q7KU8z5jK9aqZwRJxFkV9V1HDLQ6Rpy43OFHL6kEjqKx/3087NidM1lBHzQ8KRTHh2B87QL6sjh0xRl6HmWoRJuKJ3tjXjFJWH2vBGbpZC/hB4MQa24PD5PgByy5d4UaSQsLrSfJnI/PWu6nA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516681; c=relaxed/simple; bh=qr+OEoI4upU9Yi8B9mRLbLGrCTyXCm30+PolMuN56HQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UYN8vUlJ1Vt0vly6853eNfDSvgvps+Du4DWwyJH7ZzlQHPRQGSM153ZE34x6z4zJjzxKu1BhwcsQAmeCQihsfiRYLOc0hk2t1mOMtV4irPPOqr66G3pjE1QkDucYXgYsW3wCCoTmgpmvCT+VG8myn+5LrI2aNt53hjj2XBqipck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hv/mggmY; arc=none smtp.client-ip=209.85.221.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hv/mggmY" Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-33ec8aa8b6bso1345666f8f.3; Fri, 15 Mar 2024 08:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516677; x=1711121477; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eGBFnnncz6Ltll0EYPFfWplud6SFUjJrE80gGTXdd+Q=; b=hv/mggmYETy9XB0Wom4kneyLWrZu55233qAWY7Zi+2jP5y6zMCa4sRaZeCpcUDitKS Nv5hDH/330HyZenV5adRCht0OfstNlrSHLjLT2obc25JyqQn7QKQExL0EhNYBhYPBmpv JJ+5XgioEj+oY8i+EAqhMSON5BY+mYsncloMsVdehJtyJ1rZz+gcnl/Dl1Zc+s2ytfeQ XNjuqirPzwpCIwbIgs8OTZJOj2thMj1fVoqKDLMTJKtmUqUCUbNzLeONwY9f7ZnEJy3I XZE8kQ4xY+UHOXIArv3C7CY4umWz9JLkMqDkGkl4cBL6xRDR0OqVTK5sMq2qjH6f/I0R Bmbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516677; x=1711121477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eGBFnnncz6Ltll0EYPFfWplud6SFUjJrE80gGTXdd+Q=; b=gibHdyGt2MjJv3hL6v4tnuSv0hfQplrr722GV643xuYiPNIGg76Nm8UjkufdQ7xEgD f5M4181c44pul4yYPqWRE1g3LpS4LxpQm2ybGGLJtuDmTWwJHOMXKHAj6w7mEprHlzJN 4hWFwhpEhX6gbvHJPgn4VUehHVqQbY7n+37KBFQDx4l4lurmVXmuw9VRH8xV+nGQ4C5g zo8S8KpK2Q77jN9lfoTYu6v0waHNlaHY2/+ZvjjgxVeh/b5ci8IvLSqh88Z4TVZWLNMr PyLCqDJIi4/EUtEB5ddgc2MJxA6MVTE2YyC7vJzYxPzx+nGSc3ooN589d49d2SnCfjQY vD4w== X-Gm-Message-State: AOJu0YxIJxvHyArpP9Zb53bY7mF4hgNCk2jhMSCPNoJEMcWgWkGOZ9iD ynE4v+rLJEtcVRU/z5AFB2fLo+hyI0aJC9qoPSc102BJSsO9oQGIrNUErXHC X-Google-Smtp-Source: AGHT+IEHPfIrmHQOGaR2L7bbEgKtH+Dbk8yUbaOIs5ku+arMHilyRycdAH50Ph1DFBF4SUg4CT9sIA== X-Received: by 2002:a5d:680b:0:b0:33e:ae46:f9d with SMTP id w11-20020a5d680b000000b0033eae460f9dmr3639618wru.4.1710516677558; Fri, 15 Mar 2024 08:31:17 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:17 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 07/11] io_uring/rw: avoid punting to io-wq directly Date: Fri, 15 Mar 2024 15:29:57 +0000 Message-ID: <595021a813bebf0256e80b95bcc594377096e08b.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 kiocb_done() should care to specifically redirecting requests to io-wq. Remove the hopping to tw to then queue an io-wq, return -EAGAIN and let the core code io_uring handle offloading. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 8 ++++---- io_uring/io_uring.h | 1 - io_uring/rw.c | 8 +------- 3 files changed, 5 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3ae4bb988906..4ad85460ed2a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -498,7 +498,7 @@ static void io_prep_async_link(struct io_kiocb *req) } } -void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use) +static void io_queue_iowq(struct io_kiocb *req) { struct io_kiocb *link = io_prep_linked_timeout(req); struct io_uring_task *tctx = req->task->io_uring; @@ -1505,7 +1505,7 @@ void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts) if (unlikely(req->task->flags & PF_EXITING)) io_req_defer_failed(req, -EFAULT); else if (req->flags & REQ_F_FORCE_ASYNC) - io_queue_iowq(req, ts); + io_queue_iowq(req); else io_queue_sqe(req); } @@ -2088,7 +2088,7 @@ static void io_queue_async(struct io_kiocb *req, int ret) break; case IO_APOLL_ABORTED: io_kbuf_recycle(req, 0); - io_queue_iowq(req, NULL); + io_queue_iowq(req); break; case IO_APOLL_OK: break; @@ -2135,7 +2135,7 @@ static void io_queue_sqe_fallback(struct io_kiocb *req) if (unlikely(req->ctx->drain_active)) io_drain_req(req); else - io_queue_iowq(req, NULL); + io_queue_iowq(req); } } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 6426ee382276..472ba5692ba8 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -79,7 +79,6 @@ struct file *io_file_get_fixed(struct io_kiocb *req, int fd, void __io_req_task_work_add(struct io_kiocb *req, unsigned flags); bool io_alloc_async_data(struct io_kiocb *req); void io_req_task_queue(struct io_kiocb *req); -void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use); void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts); void io_req_task_queue_fail(struct io_kiocb *req, int ret); void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/rw.c b/io_uring/rw.c index 0585ebcc9773..576934dbf833 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -187,12 +187,6 @@ static inline loff_t *io_kiocb_update_pos(struct io_kiocb *req) return NULL; } -static void io_req_task_queue_reissue(struct io_kiocb *req) -{ - req->io_task_work.func = io_queue_iowq; - io_req_task_work_add(req); -} - #ifdef CONFIG_BLOCK static bool io_resubmit_prep(struct io_kiocb *req) { @@ -405,7 +399,7 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, if (req->flags & REQ_F_REISSUE) { req->flags &= ~REQ_F_REISSUE; if (io_resubmit_prep(req)) - io_req_task_queue_reissue(req); + return -EAGAIN; else io_req_task_queue_fail(req, final_ret); } From patchwork Fri Mar 15 15:29:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593657 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8ED564086B; Fri, 15 Mar 2024 15:31:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516682; cv=none; b=OUZOWyJmi7E4G28jCOEhlXApsH2rewUBt8wqy+/Tmq/zuc7qHKntcp3Krc7CwL0eLr3wBtZZVlP27xjFpDd1g14j/yEtuG3pbn6YCe8oJAqJaNC2gP4MHykwi6SyixGJEbg1zRDSjseQQxMFl6K5wxgYbDnbl4ZkItkbmecbYs4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516682; c=relaxed/simple; bh=pOxqMnA5//xNqR/l/9qyCO0QVi0Nf7fBsjJO69oEKZo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GdM/oDyztSFAZmJw/qENUvaaF2xeqbY/1QoQUYu5M4AJh3BdJ4Sj3YV16Soc2Ck1uOuzQGb5hkdYbwZroXH4sV0TgE9orcPS92Fkq442jQjpkAseqdFTRyTYsvHgKIGJUbVxFAasOoCMGCTRiscEOZFmtcL6xSMMijCAUXy5J1M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q0KO2kzj; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q0KO2kzj" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-33ec8f13c62so1332118f8f.0; Fri, 15 Mar 2024 08:31:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516678; x=1711121478; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E0rH3o8gsWbVH7l8R+hddelxY97sk4QbAKjhLNyqc/Y=; b=Q0KO2kzj3UDjer2cUOkqsfMSpWb0WN0e0bx46wYe6Fv2pVPHTaFeSjKMmseVWgZ0bU NqXyo+CuHdolif3ixyCQqfjw9MX3zpdR/wgl3JXzcjyZNvGLrHj1pqOE1LhRiUyfSoc4 Xm/8JaEj4MR7z2Qyir7AQ+mq+YnqefhJpGHTJp9CsvI6PRmXBSDRAvw1vkT7nj740C5W 3BGLJ+G0gJH0lZXrNZctdVX0ZP14wOSo27Lvbp6XZ0RofVugZXbHlwesztSR96kGxm6j /Vj2MSxW9ablCffJNHiIPxhytbQNkwGQ3MPTlnZvuyNlSyQkDjbTTqGe+Se82UEj/RdA QuMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516678; x=1711121478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E0rH3o8gsWbVH7l8R+hddelxY97sk4QbAKjhLNyqc/Y=; b=Y9uJqgSOGEAu3FzDJxhuijlW1fffv0P/R3VSpxMfYqTSi19PMTZhbiLdc19DJ1Evrj GDrjAcO/pJN7A4n0VLzbiwf8NZuh3IaSrcXJWKUM535Th6uOWr1/Ja0LHtEM98X66tyy Z1+m/6+m6J2GVUf97pin3tYBqcN4kv5ufPrmpBlFEHwpNOPdHvkh6MSL4ancrJ/kEmVj JZIntbGj6/ZUcG81+Z99G9gMt7nCNQwCq29GpQ9LZgYD1G+H6ItVYP/5Y7eeCBMzeHYT 8aKPLfZuCqqKApHeg0VlgdErv8uF8irvEFaMQUBQR41V2BXeHm9imNZxKHbsJa38BX0g 7e7Q== X-Gm-Message-State: AOJu0Ywo0DaqKZa8t+iv00cIMwPQoFBIL7ogrrDKIb1IeOQaVFwgaw29 6REM1Arj3GmOQ4LW7Sidp1pnHbm+SUSmriiotID4gaaEY4MNuG+qvNVNh3cx X-Google-Smtp-Source: AGHT+IGMIhW+A9ETRPyFAXld8B50pHwELGumJxK14YyAzes6IvnsEUiztGAiRG27dVnZNImv0tC3uw== X-Received: by 2002:adf:e253:0:b0:33e:d244:4870 with SMTP id bl19-20020adfe253000000b0033ed2444870mr1707979wrb.59.1710516678577; Fri, 15 Mar 2024 08:31:18 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:18 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 08/11] io_uring: force tw ctx locking Date: Fri, 15 Mar 2024 15:29:58 +0000 Message-ID: <1f7f31f4075e766343055ff0d07482992038d467.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We can run normal task_work without locking the ctx, however we try to lock anyway and most handlers prefer or require it locked. It might have been interesting to multi-submitter ring with high contention completing async read/write requests via task_work, however that will still need to go through io_req_complete_post() and potentially take the lock for rsrc node putting or some other case. In other words, it's hard to care about it, so alawys force the locking. The case described would also because of various io_uring caches. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4ad85460ed2a..0cef5c4ddc98 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1191,8 +1191,9 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - /* if not contended, grab and improve batching */ - ts.locked = mutex_trylock(&ctx->uring_lock); + + ts.locked = true; + mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } INDIRECT_CALL_2(req->io_task_work.func, @@ -1453,11 +1454,9 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, if (io_run_local_work_continue(ctx, ret, min_events)) goto again; - if (ts->locked) { - io_submit_flush_completions(ctx); - if (io_run_local_work_continue(ctx, ret, min_events)) - goto again; - } + io_submit_flush_completions(ctx); + if (io_run_local_work_continue(ctx, ret, min_events)) + goto again; trace_io_uring_local_work_run(ctx, ret, loops); return ret; @@ -1481,14 +1480,12 @@ static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = {}; + struct io_tw_state ts = { .locked = true }; int ret; - ts.locked = mutex_trylock(&ctx->uring_lock); + mutex_lock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &ts, min_events); - if (ts.locked) - mutex_unlock(&ctx->uring_lock); - + mutex_unlock(&ctx->uring_lock); return ret; } From patchwork Fri Mar 15 15:29:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593658 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98C2A40BED; Fri, 15 Mar 2024 15:31:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516683; cv=none; b=jUPHjqpxUrSQmLgNANL9YlQgbMbPk4DRt2aezFSkstCQbZ/5nqXzaE/e0XItWLs9hbOcjCp+Dkv2iGRXzXZoT5hc9fATKD5ejSH7jaqwIFnOz9E56SfoU05tNnExDyQArxtMpndw6Wt3OAnzilQ0/rlD1JnrwzqzbLlaUVtELec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516683; c=relaxed/simple; bh=2IVgJelfwSrUZxl7VJW3YC+UA79e6CehL+UHqoNILCM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f8e0rFue95XnypkhY2do6dUGMgdBwb07jVOEHYN5SAn/4Tef5yOiZYUrSMNGivXNLakXuOKPukEP67a9ps9xZhSPUwVPjJ+yk2GpD11actXloSQMQUvfa62OHaOvzPI1Ss3WGORp7xba9nTcAcKsIjHJu3wiBf7OWQCXD6f7I6E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mbpE9O8b; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mbpE9O8b" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-33e5978fc1bso1113763f8f.3; Fri, 15 Mar 2024 08:31:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516680; x=1711121480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8u+FxQNKf+A4K+mkGN1fRUo4gZebAM7P2pFv3PfIxx8=; b=mbpE9O8bLvJRtRcVpsaiV4tOJG5BauSH46hCaO2o8g8q9JB2VdmusPMO6YNKWmjjhg 0mm+WBWHkEWjpK9YH9mAcLyNkS7TxBU8CUkOQK7CX6ln5Lw1lEQm46SktTOn9i7mdxzS 97/lrBu2BqSzC0pfCmRFeN8syeWuuk9SXBUPFVVX9Zvy0IWRV8wxNY/eF+NAVnoe6IMU ayDVYpIuFCEZxkJw5KdaRquauh0d1L2vWi3TV13TCKkB30T3lBWkkuIHMEnou7MADghN lIiJTmrEhyX/Ti1/g82EFQRSmWYMGA0TW7v7Xk/q1fvXEHiBCSoBngdn34UzvaiDNZHa +OGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516680; x=1711121480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8u+FxQNKf+A4K+mkGN1fRUo4gZebAM7P2pFv3PfIxx8=; b=YcsPjO5lPItQ8qP9hXdVxWVUtPMu3sqT5CvvUZSmPjOGPrGIjTdu4dzsrZ6eYkRs42 uwfXH6v+ClDplM+heDqkBEP8hAMKMet07d0IiFWLUIkV6EZYX9QURZ9qelDTyI+fREwT gbKY39Uv+3YLZfhE3D8SaTHrU9vmeU7p14vqX/Owso3qJFj4T2taPen/+mFDp8Yi6Xcz SuWw5BLfooCZFcOYgGCG9EvxalGuRINIGwwl9LIvjWHwyMcRtCxC6PftXb4KP4uYv0aD jAu9bxVY+ZvlTdjA8FsDc34ge8FfKEHRAQwQk0JQDIt1dQRW+vgoMnHBZVxJssBYltoS Nttg== X-Gm-Message-State: AOJu0YwS3ne8MjOsDNIMomCq20/7cjh/dieO29kCPXuqt/tvHyi0FuuP oREmcmIsrdqMYT9jF6+8XMzRpSBUQ0J4nCvZxfpc6orGwQjvLSjoDjMp2Yai X-Google-Smtp-Source: AGHT+IFBvbBF/MveCOkzmUn3I+LiMMKkuRvn/X5whcqxDAiWgGUz8B7q5d268IZY5IPVrw+cgr62yA== X-Received: by 2002:a05:6000:d49:b0:33e:6db4:e89f with SMTP id du9-20020a0560000d4900b0033e6db4e89fmr4302469wrb.22.1710516679681; Fri, 15 Mar 2024 08:31:19 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:19 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 09/11] io_uring: remove struct io_tw_state::locked Date: Fri, 15 Mar 2024 15:29:59 +0000 Message-ID: <04482ca7bfea9eb47ba483c590aa46c83315a6ea.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 ctx is always locked for task_work now, so get rid of struct io_tw_state::locked. Note I'm stopping one step before removing io_tw_state altogether, which is not empty, because it still serves the purpose of indicating which function is a tw callback and forcing users not to invoke them carelessly out of a wrong context. The removal can always be done later. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 2 -- io_uring/io_uring.c | 31 ++++++++----------------------- io_uring/io_uring.h | 5 +---- io_uring/poll.c | 2 +- io_uring/rw.c | 6 ++---- io_uring/timeout.c | 8 ++------ io_uring/uring_cmd.c | 8 ++------ io_uring/waitid.c | 2 +- 8 files changed, 17 insertions(+), 47 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e24893625085..5a2afbc93887 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -439,8 +439,6 @@ struct io_ring_ctx { }; struct io_tw_state { - /* ->uring_lock is taken, callbacks can use io_tw_lock to lock it */ - bool locked; }; enum { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0cef5c4ddc98..1c4bbfc411d1 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -252,14 +252,12 @@ static __cold void io_fallback_req_func(struct work_struct *work) fallback_work.work); struct llist_node *node = llist_del_all(&ctx->fallback_llist); struct io_kiocb *req, *tmp; - struct io_tw_state ts = { .locked = true, }; + struct io_tw_state ts = {}; percpu_ref_get(&ctx->refs); mutex_lock(&ctx->uring_lock); llist_for_each_entry_safe(req, tmp, node, io_task_work.node) req->io_task_work.func(req, &ts); - if (WARN_ON_ONCE(!ts.locked)) - return; io_submit_flush_completions(ctx); mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); @@ -1163,11 +1161,9 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) return; if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_andnot(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); - if (ts->locked) { - io_submit_flush_completions(ctx); - mutex_unlock(&ctx->uring_lock); - ts->locked = false; - } + + io_submit_flush_completions(ctx); + mutex_unlock(&ctx->uring_lock); percpu_ref_put(&ctx->refs); } @@ -1191,8 +1187,6 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - - ts.locked = true; mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } @@ -1465,22 +1459,16 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = { .locked = true, }; - int ret; + struct io_tw_state ts = {}; if (llist_empty(&ctx->work_llist)) return 0; - - ret = __io_run_local_work(ctx, &ts, min_events); - /* shouldn't happen! */ - if (WARN_ON_ONCE(!ts.locked)) - mutex_lock(&ctx->uring_lock); - return ret; + return __io_run_local_work(ctx, &ts, min_events); } static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = { .locked = true }; + struct io_tw_state ts = {}; int ret; mutex_lock(&ctx->uring_lock); @@ -1708,10 +1696,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts) { - if (ts->locked) - io_req_complete_defer(req); - else - io_req_complete_post(req, IO_URING_F_UNLOCKED); + io_req_complete_defer(req); } /* diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 472ba5692ba8..6cad3ef3408b 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -344,10 +344,7 @@ static inline bool io_task_work_pending(struct io_ring_ctx *ctx) static inline void io_tw_lock(struct io_ring_ctx *ctx, struct io_tw_state *ts) { - if (!ts->locked) { - mutex_lock(&ctx->uring_lock); - ts->locked = true; - } + lockdep_assert_held(&ctx->uring_lock); } /* diff --git a/io_uring/poll.c b/io_uring/poll.c index 6db1dcb2c797..8901dd118e50 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -322,7 +322,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_fill_cqe_req_aux(req, ts->locked, mask, + if (!io_fill_cqe_req_aux(req, true, mask, IORING_CQE_F_MORE)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; diff --git a/io_uring/rw.c b/io_uring/rw.c index 576934dbf833..c7f9246ff508 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -305,11 +305,9 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts) io_req_io_end(req); - if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; + if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) + req->cqe.flags |= io_put_kbuf(req, 0); - req->cqe.flags |= io_put_kbuf(req, issue_flags); - } io_req_task_complete(req, ts); } diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 7fd7dbb211d6..0a48e6acd0b2 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -72,10 +72,7 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; if (!io_timeout_finish(timeout, data)) { - bool filled; - filled = io_fill_cqe_req_aux(req, ts->locked, -ETIME, - IORING_CQE_F_MORE); - if (filled) { + if (io_fill_cqe_req_aux(req, true, -ETIME, IORING_CQE_F_MORE)) { /* re-arm timer */ spin_lock_irq(&ctx->timeout_lock); list_add(&timeout->list, ctx->timeout_list.prev); @@ -301,7 +298,6 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *ts) { - unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_kiocb *prev = timeout->prev; int ret = -ENOENT; @@ -313,7 +309,7 @@ static void io_req_task_link_timeout(struct io_kiocb *req, struct io_tw_state *t .data = prev->cqe.user_data, }; - ret = io_try_cancel(req->task->io_uring, &cd, issue_flags); + ret = io_try_cancel(req->task->io_uring, &cd, 0); } io_req_set_res(req, ret ?: -ETIME, 0); io_req_task_complete(req, ts); diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index ec38a8d4836d..e45d4cd5ef82 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -56,13 +56,9 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned issue_flags = IO_URING_F_UNLOCKED; - /* locked task_work executor checks the deffered list completion */ - if (ts->locked) - issue_flags = IO_URING_F_COMPLETE_DEFER; - - ioucmd->task_work_cb(ioucmd, issue_flags); + /* task_work executor checks the deffered list completion */ + ioucmd->task_work_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); } void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, diff --git a/io_uring/waitid.c b/io_uring/waitid.c index 6f851978606d..791ea752b0b1 100644 --- a/io_uring/waitid.c +++ b/io_uring/waitid.c @@ -118,7 +118,7 @@ static int io_waitid_finish(struct io_kiocb *req, int ret) static void io_waitid_complete(struct io_kiocb *req, int ret) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); - struct io_tw_state ts = { .locked = true }; + struct io_tw_state ts = {}; /* anyone completing better be holding a reference */ WARN_ON_ONCE(!(atomic_read(&iw->refs) & IO_WAITID_REF_MASK)); From patchwork Fri Mar 15 15:30:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593659 Received: from mail-lf1-f48.google.com (mail-lf1-f48.google.com [209.85.167.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBA1641236; Fri, 15 Mar 2024 15:31:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516684; cv=none; b=bAoaZt1e66REgiikB5FX4vm6z11Ox1HaSXvpQt1hkbY7B1ArCI25i5a+taUYGixgcaZkXt8W6aGyKMpWwYkWTYn38nRQ1H3y2lBkVvvWEb8NI6sY6V7Qyu85bPGfgwJMYDK5oLnFgj+ZGVx0e3gpJsbmhR3OCeeB2WqvkTjG7GY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516684; c=relaxed/simple; bh=EHG9bGiSWjV6Of0fB0gBLFd05b0oTLSBkIQD9NicUNo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OHoN9cIBzWN/mV3b+9rrWULHgAB25Pm+QPQK3UYYEnFwV0RDIMuQjnBoVasqqWw0eki89o33kDy3oiyP+zX9AZOwbUWqnDieXD32bQnL1HKPHDqTn+vmo9de1AiqCh5D8A4Uj+G37qsZTdXOOPTHZp1++6IjUd/oHvn/YWv3N9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cr71W45h; arc=none smtp.client-ip=209.85.167.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cr71W45h" Received: by mail-lf1-f48.google.com with SMTP id 2adb3069b0e04-512e39226efso2154833e87.0; Fri, 15 Mar 2024 08:31:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516681; x=1711121481; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nrsISQ+DEwoL0WD2x8rUh3mvSFRhgV8vcD0vdmOMI8Q=; b=cr71W45hoSQPPI7C32pWg0Mm/yF77m7re5oCpSqw7rvKKjbk/nMFEPh2KxXwXm85Ig p0JPJ5CKwJ6KkM9ZSh23Dj8SDhuhmNZn4THNJqHvIgQIoPW6snoczVNDHFHcCbbBjOLM n9oFcy96mhjX6sMD5uukfg77+HxzQoA88XlFJhGYLu8IyAu0jTK/1ehW1zHQKY3HlZJ9 7NxNP24zzOlzM8HW/GZ568Hjw2RHD5zvZOfQZwnEtu5AcWvdTPCXYL2uScty2dlCb9Q4 i44izMFQZCKD77QC6GHvlEekiSBKW1tTkVvfG7PIbM9RBmHrXZluqJWC7W9T3v7MFmjg O9aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516681; x=1711121481; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nrsISQ+DEwoL0WD2x8rUh3mvSFRhgV8vcD0vdmOMI8Q=; b=XhcA6zRldH218fhtJgUcN7ylq8dTrwyMXoaDMlIzwhAUu1SbqzpqWe7X3fOJlixS+V 2uOJVsiNO6pW/aFilPuBVwwumpz7Z+brVVkHZA3U5szhOCrFDqW07WUje/diX+EzM97v a+NmQUHqG6RdedFZIG3Nj2+dl9rf0by+PuStqjETz0jRtPZylXTIOyilLWpUkpZJoCmV 5JvD2A+Z/166kzap46rPoPakJ5gnfEN62YBg2BNXZtTj/Tu8RoHW4F2AzWxf1Bm1fB9k a9qh8ykPCVD9D8Y4cBLz/6v92WMIwqBPr2ZcUPE15WV3ves9AnCMCXzxxAzE0cFJ3bfS s+GQ== X-Gm-Message-State: AOJu0Yy7m0bNLcLtsVkdnjDpjaFHoRpmjjazjC22pZmStfGOl6DqGsJH 2+H0xKj5ITOPn001hK7RG6/1mwDi0rcNJL1eg0gfUCnMPiL0Nlfg1ZfGH6M1 X-Google-Smtp-Source: AGHT+IE63B1JvzA/d9wOCVO4Ny+trUpkfhDlLpCm8caS5IovYC48fnKiXokWIqjRLR+C/TIKgp4jOQ== X-Received: by 2002:ac2:5b10:0:b0:513:dd66:d5ed with SMTP id v16-20020ac25b10000000b00513dd66d5edmr595907lfn.29.1710516680653; Fri, 15 Mar 2024 08:31:20 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:20 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 10/11] io_uring: refactor io_fill_cqe_req_aux Date: Fri, 15 Mar 2024 15:30:00 +0000 Message-ID: <777fb7fbd2a3ba526a876fc422288c5f65283f12.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The restriction on multishot execution context disallowing io-wq is driven by rules of io_fill_cqe_req_aux(), it should only be called in the master task context, either from the syscall path or in task_work. Since task_work now always takes the ctx lock implying IO_URING_F_COMPLETE_DEFER, we can just assume that the function is always called with its defer argument set to true. Kill the argument. Also rename the function for more consistency as "fill" in CQE related functions was usually meant for raw interfaces only copying data into the CQ without any locking, waking the user and other accounting "post" functions take care of. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 16 +++------------- io_uring/io_uring.h | 2 +- io_uring/net.c | 6 ++---- io_uring/poll.c | 3 +-- io_uring/rw.c | 4 +--- io_uring/timeout.c | 2 +- 6 files changed, 9 insertions(+), 24 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 1c4bbfc411d1..167a3429a056 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -913,40 +913,30 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) state->cqes_count = 0; } -static bool __io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags, - bool allow_overflow) +bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { bool filled; io_cq_lock(ctx); filled = io_fill_cqe_aux(ctx, user_data, res, cflags); - if (!filled && allow_overflow) + if (!filled) filled = io_cqring_event_overflow(ctx, user_data, res, cflags, 0, 0); io_cq_unlock_post(ctx); return filled; } -bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) -{ - return __io_post_aux_cqe(ctx, user_data, res, cflags, true); -} - /* * A helper for multishot requests posting additional CQEs. * Should only be used from a task_work including IO_URING_F_MULTISHOT. */ -bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags) +bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) { struct io_ring_ctx *ctx = req->ctx; u64 user_data = req->cqe.user_data; struct io_uring_cqe *cqe; lockdep_assert(!io_wq_current_is_worker()); - - if (!defer) - return __io_post_aux_cqe(ctx, user_data, res, cflags, false); - lockdep_assert_held(&ctx->uring_lock); if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 6cad3ef3408b..4bc96470e591 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -67,7 +67,7 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); -bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags); +bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags); void __io_commit_cqring_flush(struct io_ring_ctx *ctx); struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages); diff --git a/io_uring/net.c b/io_uring/net.c index 19451f0dbf81..b2890eeea6a8 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -699,8 +699,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, * receive from this socket. */ if ((req->flags & REQ_F_APOLL_MULTISHOT) && !mshot_finished && - io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - *ret, cflags | IORING_CQE_F_MORE)) { + io_req_post_cqe(req, *ret, cflags | IORING_CQE_F_MORE)) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; @@ -1421,8 +1420,7 @@ int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (ret < 0) return ret; - if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - ret, IORING_CQE_F_MORE)) + if (io_req_post_cqe(req, ret, IORING_CQE_F_MORE)) goto retry; io_req_set_res(req, ret, 0); diff --git a/io_uring/poll.c b/io_uring/poll.c index 8901dd118e50..5d55bbf1de15 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -322,8 +322,7 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_fill_cqe_req_aux(req, true, mask, - IORING_CQE_F_MORE)) { + if (!io_req_post_cqe(req, mask, IORING_CQE_F_MORE)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; } diff --git a/io_uring/rw.c b/io_uring/rw.c index c7f9246ff508..35216e8adc29 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -955,9 +955,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) cflags = io_put_kbuf(req, issue_flags); rw->len = 0; /* similarly to above, reset len to 0 */ - if (io_fill_cqe_req_aux(req, - issue_flags & IO_URING_F_COMPLETE_DEFER, - ret, cflags | IORING_CQE_F_MORE)) { + if (io_req_post_cqe(req, ret, cflags | IORING_CQE_F_MORE)) { if (issue_flags & IO_URING_F_MULTISHOT) { /* * Force retry, as we might have more data to diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 0a48e6acd0b2..3458ca550b83 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -72,7 +72,7 @@ static void io_timeout_complete(struct io_kiocb *req, struct io_tw_state *ts) struct io_ring_ctx *ctx = req->ctx; if (!io_timeout_finish(timeout, data)) { - if (io_fill_cqe_req_aux(req, true, -ETIME, IORING_CQE_F_MORE)) { + if (io_req_post_cqe(req, -ETIME, IORING_CQE_F_MORE)) { /* re-arm timer */ spin_lock_irq(&ctx->timeout_lock); list_add(&timeout->list, ctx->timeout_list.prev); From patchwork Fri Mar 15 15:30:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13593660 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 730CE3D96E; Fri, 15 Mar 2024 15:31:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516685; cv=none; b=kO9y+SxUBtU3qwWD+7O7l9RRKS9UXa3qBgh7v5DOy2jZIf4xGsnYW9qdaZeRUQcM6tlndjIe8yFZ84mfj2pcEhhJYPoGRJBqL9v8+m6v2i8e9e9G0iIOHoOdnJXpme5/nLbHpmCKBWEd0G9oy4APw7QqMX7S6HOMZ3uFZRHnzU0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710516685; c=relaxed/simple; bh=0b0xANT5rn57TIxscmYXYbG3B66r7OZmchFYYxTy2TU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qqR82c0nCRtnJjtvhDzGIFG4DpopMYwyNpz7MLDggaZu/dSwPBVkNcxt5hUsvr/Lj4zPqMkFxSW/lofD9wbc6ULp6WIXHo1D/er+4vwDT0e9uifyom0990aBWmFe01JvmzOXZNCIbta+vtNb4hYvC7wqR2sityYlW5xk+urbpaM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JquzFlcO; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JquzFlcO" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-33ed7ba1a42so78685f8f.2; Fri, 15 Mar 2024 08:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710516681; x=1711121481; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LyQ7TwSuB2UzAPYUsilzhWfnsZo/NWLW8qWu01BUebQ=; b=JquzFlcOVH47iNZRCHCXkptz7OdQRu8n2FsRxryv3o9dhYZVeM284HrcPYEzR72INC tJD+0eDRwPGtYa0QVYuBZzoKu95C6O4ajnnF+DcdVecaUS/vAZ1RiZQGyfoYZZvcaTAm +ioxjArcgMxR72R/k6SVm98/wLO3tAS3tXfR2XSdE1UtPy7aJLz9K8MO2NkxhBDiABdp gcifM08yDtIhGK3QBEnSXzVjOWPV8zFLiJ1IjY5Iq9A0S7ZYYCCH/MJfHMP9VXYCQeNi bC4ZpyhdMs7ckJ9p/k/uPQKGaDgoCtz4B/Z1ty2FeP2Qe6Ew4ox8Ib0S6U4PNuSmihhW fGGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710516681; x=1711121481; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LyQ7TwSuB2UzAPYUsilzhWfnsZo/NWLW8qWu01BUebQ=; b=h7/jAqBC1XeL5UXYz++SIMWCL6DYcbZmDb7YG7BYruekFObNV7FetW9YZGy0MIK8hX Pa26O2kNSvAVcZJprd8iEe/Bo+DJaNtmAo9/cWPuob/TMtWJiiT+4Rqf+1Lfh+qFWkPE BNQLiEgyD8J1Q+yFNtNd+3+vS+67vnQXg4moG33afPgeNJFkmJIema9gGyJXAjiHbKgZ qQYq0Emve8NgjOkcXKRUYUN/BMJJ58Q+u8eU0Q5OT4XXgMCbLe6fFC7p6jtTQNCNRLth XN9EqNbfnh47xdmvtqmUIPmiGBpCZVqIVav4uBUI0o0bjvmbar1nRktWfWwC34MRucG4 mr9g== X-Gm-Message-State: AOJu0Yw849ZQmvg9/Nhork24U3WpDV9/Q0AWlxcbwIK9q8GpjnxVtlXx LJcBa/QMqVL61pPvH7M4pDSwEFFdwcqi0UeXRmtZpiGMlL6ne29vI8ALCGwn X-Google-Smtp-Source: AGHT+IGp37IIkAgab01oSdxVqT+IQ8QLTYPVVW227ELd/el7CBClhRd0EA4n5tOjGuXQQYiuE3HthQ== X-Received: by 2002:adf:e049:0:b0:33e:b73a:4a4b with SMTP id w9-20020adfe049000000b0033eb73a4a4bmr3259734wrh.28.1710516681658; Fri, 15 Mar 2024 08:31:21 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.141.58]) by smtp.gmail.com with ESMTPSA id u3-20020a5d6ac3000000b0033dd2c3131fsm3415671wrw.65.2024.03.15.08.31.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 08:31:21 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH 11/11] io_uring: get rid of intermediate aux cqe caches Date: Fri, 15 Mar 2024 15:30:01 +0000 Message-ID: <0eb3f55722540a11b036d3c90771220eb082d65e.1710514702.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_post_aux_cqe(), which is used for multishot requests, delays completions by putting CQEs into a temporary array for the purpose completion lock/flush batching. DEFER_TASKRUN doesn't need any locking, so for it we can put completions directly into the CQ and defer post completion handling with a flag. That leaves !DEFER_TASKRUN, which is not that interesting / hot for multishot requests, so have conditional locking with deferred flush for them. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +- io_uring/io_uring.c | 64 ++++++++-------------------------- 2 files changed, 15 insertions(+), 52 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5a2afbc93887..ea7e5488b3be 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -205,6 +205,7 @@ struct io_submit_state { bool plug_started; bool need_plug; + bool cq_flush; unsigned short submit_nr; unsigned int cqes_count; struct blk_plug plug; @@ -342,8 +343,6 @@ struct io_ring_ctx { unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; - struct io_uring_cqe completion_cqes[16]; - spinlock_t completion_lock; /* IRQ completion list, under ->completion_lock */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 167a3429a056..023fcf5d52c1 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -176,7 +176,7 @@ static struct ctl_table kernel_io_uring_disabled_table[] = { static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { if (!wq_list_empty(&ctx->submit_state.compl_reqs) || - ctx->submit_state.cqes_count) + ctx->submit_state.cq_flush) __io_submit_flush_completions(ctx); } @@ -636,6 +636,12 @@ static inline void __io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } +static inline void __io_cq_unlock(struct io_ring_ctx *ctx) +{ + if (!ctx->lockless_cq) + spin_unlock(&ctx->completion_lock); +} + static inline void io_cq_lock(struct io_ring_ctx *ctx) __acquires(ctx->completion_lock) { @@ -888,31 +894,6 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, return false; } -static void __io_flush_post_cqes(struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) -{ - struct io_submit_state *state = &ctx->submit_state; - unsigned int i; - - lockdep_assert_held(&ctx->uring_lock); - for (i = 0; i < state->cqes_count; i++) { - struct io_uring_cqe *cqe = &ctx->completion_cqes[i]; - - if (!io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags)) { - if (ctx->lockless_cq) { - spin_lock(&ctx->completion_lock); - io_cqring_event_overflow(ctx, cqe->user_data, - cqe->res, cqe->flags, 0, 0); - spin_unlock(&ctx->completion_lock); - } else { - io_cqring_event_overflow(ctx, cqe->user_data, - cqe->res, cqe->flags, 0, 0); - } - } - } - state->cqes_count = 0; -} - bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags) { bool filled; @@ -933,31 +914,16 @@ bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) { struct io_ring_ctx *ctx = req->ctx; - u64 user_data = req->cqe.user_data; - struct io_uring_cqe *cqe; + bool posted; lockdep_assert(!io_wq_current_is_worker()); lockdep_assert_held(&ctx->uring_lock); - if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { - __io_cq_lock(ctx); - __io_flush_post_cqes(ctx); - /* no need to flush - flush is deferred */ - __io_cq_unlock_post(ctx); - } - - /* For defered completions this is not as strict as it is otherwise, - * however it's main job is to prevent unbounded posted completions, - * and in that it works just as well. - */ - if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) - return false; - - cqe = &ctx->completion_cqes[ctx->submit_state.cqes_count++]; - cqe->user_data = user_data; - cqe->res = res; - cqe->flags = cflags; - return true; + __io_cq_lock(ctx); + posted = io_fill_cqe_aux(ctx, req->cqe.user_data, res, cflags); + ctx->submit_state.cq_flush = true; + __io_cq_unlock_post(ctx); + return posted; } static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) @@ -1551,9 +1517,6 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_wq_work_node *node; __io_cq_lock(ctx); - /* must come first to preserve CQE ordering in failure cases */ - if (state->cqes_count) - __io_flush_post_cqes(ctx); __wq_list_for_each(node, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); @@ -1575,6 +1538,7 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) io_free_batch_list(ctx, state->compl_reqs.first); INIT_WQ_LIST(&state->compl_reqs); } + ctx->submit_state.cq_flush = false; } static unsigned io_cqring_events(struct io_ring_ctx *ctx)