From patchwork Fri Apr 5 15:50:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13619233 Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B804116F85A for ; Fri, 5 Apr 2024 15:50:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332216; cv=none; b=AOMGziDFz/OxAojNA9R1hEZVTpVM/RzBdKxKEX+/W/U5svFVFsTJGeM0/eQB36lki1gIucof9a7im866T6pFbD4SRCLl285h8oGyIsC0/U/hSV+E6fUJR76dfa9zpQjCVzmNhaUaYOZwwPGYz9Fs1JNloRqydj42635aPhbtVTg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332216; c=relaxed/simple; bh=ivG4baHe8jvZDHz51NYdk34fk/ji3JHYkVbRpuIa3H4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H4bKGZn6whTNeZZT3LipJeDFSfiLjxUo3meVAh5qwMn78Bx2Yr6paLjNWeMe+WAtNm8RmsxbBTsoCO/XOQsSQwOBEBMnNGs9HYkrSyaTAlaeWHAnGjhyVKeZgmUDp4lF3jU8dCLi71nxGgUwEUt46jvtdDvwz51W5gxC0UNwyzw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bNvmUNl9; arc=none smtp.client-ip=209.85.218.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bNvmUNl9" Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-a468226e135so330984466b.0 for ; Fri, 05 Apr 2024 08:50:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712332212; x=1712937012; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FKMEblkDI7QPYIeMCqCL6ua5DWNpr+kCM+xZPnX6/xc=; b=bNvmUNl9q9qFtiGk3iIL1cJ5XepvqW42NnPrwvJhGRMq66KIGFlwijj/Yry/CoxmGo MoAHFR1FVrVI07S/PaTTUQC300zL/KS97mb//QGne1+mHF8sJACAAw69ILqhzDSSZVrc Njz4GZRnqauedEmJlufTZcyoMSZpBDWL2L7+rXi/n69U1+wQvqOTmNCuJyyXJFzEXozf uEPTU7Fl19rNIluDKL2uH7/iCNtlNAf1QZ6qX69PTPXCXMvv/Gojj4GU7FyGquTcUckt QXlXBqKPSkf7fRldCruz116oGZlkVVXcuDhMy011z1732M3uBreO0WxS3Ec3o12X5yCk rL5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712332212; x=1712937012; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FKMEblkDI7QPYIeMCqCL6ua5DWNpr+kCM+xZPnX6/xc=; b=qvNqYtP8XUex0jWgJ53zZ50Q4GzqWmA92djRnVjJyU1iwhJQodpKIhaPdlW3ey3war zPzPVGao4+zu+4cFS2Mc8sdfKmzC6N+S1jn0ABun6Zwn0ydAhMNYHMq67is1C2hzON2d nMoACnA6WLnRWH5WHJ+i+FETFvNUrrksCcGfg9qw2PL6LceANGsYw5DczMPftpRO052e O2i42Gqq5jMllxDW7pXVB2XNrcdERGTBmVyiYqF1t/I1XXaNes2R5AwUPfA4cM/OMS8z IaQeTway7m+U0xMPw0NuhfOb0jKOzBKBOR4du8/+WgqRcNTnmjV/Prp7FnCB1jDvNlAR 1cWw== X-Gm-Message-State: AOJu0Yylr3q42x7pI9qTuy9hvWsuR3KTG1vWGpCXAB7AQd2gFFn0oZSH 96wGFgy7HYMoV0KMRbdJPnN/17mY6+ZkPWcfpwjUTITYHE2a9JFPkhCvF2Xj X-Google-Smtp-Source: AGHT+IG7XmzxUYgDaSUyPNK+5KDrKWIP7xWbpjExkbPKLsA3iB4G9Y3paPQIGmrtOHzywspKybOLVA== X-Received: by 2002:a17:907:985b:b0:a4e:109f:7b4b with SMTP id jj27-20020a170907985b00b00a4e109f7b4bmr1513990ejc.41.1712332212337; Fri, 05 Apr 2024 08:50:12 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id lc24-20020a170906f91800b00a46c8dbd5e4sm966105ejb.7.2024.04.05.08.50.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Apr 2024 08:50:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Ming Lei Subject: [PATCH for-next 1/4] io_uring: kill dead code in io_req_complete_post Date: Fri, 5 Apr 2024 16:50:02 +0100 Message-ID: <1d8297e2046553153e763a52574f0e0f4d512f86.1712331455.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Ming Lei Since commit 8f6c829491fe ("io_uring: remove struct io_tw_state::locked"), io_req_complete_post() is only called from io-wq submit work, where the request reference is guaranteed to be grabbed and won't drop to zero in io_req_complete_post(). Kill the dead code, meantime add req_ref_put() to put the reference. Cc: Pavel Begunkov Signed-off-by: Ming Lei Reviewed-by: Pavel Begunkov Signed-by: Pavel Begunkov Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 37 ++----------------------------------- io_uring/refs.h | 7 +++++++ 2 files changed, 9 insertions(+), 35 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8a9584c5c8ce..b7f742fe9d41 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -930,7 +930,6 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags) static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; - struct io_rsrc_node *rsrc_node = NULL; /* * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires @@ -947,42 +946,10 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) if (!io_fill_cqe_req(ctx, req)) io_req_cqe_overflow(req); } - - /* - * If we're the last reference to this request, add to our locked - * free_list cache. - */ - if (req_ref_put_and_test(req)) { - if (req->flags & IO_REQ_LINK_FLAGS) { - if (req->flags & IO_DISARM_MASK) - io_disarm_next(req); - if (req->link) { - io_req_task_queue(req->link); - req->link = NULL; - } - } - io_put_kbuf_comp(req); - if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS)) - io_clean_op(req); - io_put_file(req); - - rsrc_node = req->rsrc_node; - /* - * Selected buffer deallocation in io_clean_op() assumes that - * we don't hold ->completion_lock. Clean them here to avoid - * deadlocks. - */ - io_put_task_remote(req->task); - wq_list_add_head(&req->comp_list, &ctx->locked_free_list); - ctx->locked_free_nr++; - } io_cq_unlock_post(ctx); - if (rsrc_node) { - io_ring_submit_lock(ctx, issue_flags); - io_put_rsrc_node(ctx, rsrc_node); - io_ring_submit_unlock(ctx, issue_flags); - } + /* called from io-wq submit work only, the ref won't drop to zero */ + req_ref_put(req); } void io_req_defer_failed(struct io_kiocb *req, s32 res) diff --git a/io_uring/refs.h b/io_uring/refs.h index 1336de3f2a30..63982ead9f7d 100644 --- a/io_uring/refs.h +++ b/io_uring/refs.h @@ -33,6 +33,13 @@ static inline void req_ref_get(struct io_kiocb *req) atomic_inc(&req->refs); } +static inline void req_ref_put(struct io_kiocb *req) +{ + WARN_ON_ONCE(!(req->flags & REQ_F_REFCOUNT)); + WARN_ON_ONCE(req_ref_zero_or_close_to_overflow(req)); + atomic_dec(&req->refs); +} + static inline void __io_req_set_refcount(struct io_kiocb *req, int nr) { if (!(req->flags & REQ_F_REFCOUNT)) { From patchwork Fri Apr 5 15:50:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13619234 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DE8916F901 for ; Fri, 5 Apr 2024 15:50:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332217; cv=none; b=Nm8CQb7voEX5DgjfmK+oXcyRlCuKV1NrWsXUcECLrkhH99/zZYhzObkfZQTudWJ7Tcnc7HOLEjtGuFwvOSH7z7UCB1I241h1gGveOCJoWbl5RlIKRFPY68tgtvSAn2WGvejKtS7VUW892+9jeVXmNTRQWjhXgrTVYPx0yUJAuBY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332217; c=relaxed/simple; bh=7oALiLajvRVhDyG5FY3/RAJ0HrbZAI6k6shViVT7YLQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Mhj1zCw19Zf94um/q6lXEc3Mg5jsWQFX0NQLjv8owLh1NFNJ6XoUI7GyjqaUlfPViOvU0685/tqTdqpljzNe9e96rmwrbjlNYG2uK4cFBxg5cQ72c5SJxFo3T6VWXWVSED32e953uCSts117iXDN36Y469Tu1eVUKAA8qvrB8d0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DE7J21/M; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DE7J21/M" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a51ae869113so48165666b.0 for ; Fri, 05 Apr 2024 08:50:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712332213; x=1712937013; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7nnkNMcai0DLhA6/2LRzuS5PCHUKa84UwC0Mph9cl+E=; b=DE7J21/M75349hgilB9yZglwd7qabP/X2ksM3/GYSF90MlDZOJw6TfnqbShRk64tp7 tXJoDFHYk2y+PxKjOhWgcUbrythXT+KhYZQJ5XJqe6HOCoAu3nS6Ufu1aST8h/nBXK3m tBsA7DaBJOBtGMMkratOARcZBRTXK4qPCaiNq9SIrj2OuJ1foiEp3hhz08UaBgGsIrYv 6PfB/S1sDA17d1nP85oweoS8rRAypaLx39J9I5eqGaKm7OKuU75a4mVuucEmZmZLzWRU yRLAgLz0RI7Tssz9Qe+Bg4fzAUtdUuQfHLUBcU6RVkA3fC8mw/x6SXw9xde4nGae3ga5 6Aow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712332213; x=1712937013; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7nnkNMcai0DLhA6/2LRzuS5PCHUKa84UwC0Mph9cl+E=; b=W/SgOpwrLsisZ5zPWA0zUaxA1aACdh4rW9tW1d6EOCpWDYhjbGb/6hx4GMBk6AYrjp c0O1PNMvq1lwa1w3VqZdhT/zc9V79dFzEtxat3Ra5R9OorjWBLElVGfKkdG1fLiMnNB5 ypggGFIX/buorz+ArqGYdnfK/OUg+OBWNanCRXkJvRHQzcaAasALHLeUUUh1hBaINHHj NUsdIaJNY6jHFnkxBfF1TymYawJYQeN4sVYV7f5oqALU55FZT6JUavzYsjdo/I0xjL3I dZMTd3Xz13N7ZcM2AHNvgT/btIrvxmPjGctpqq+0Rgw9zsF47TDPEpelpLVoNXgbbRal EuLA== X-Gm-Message-State: AOJu0YwqN57r//mVsM9n8ANbvOPBw+5/K0uFBSwv5hO3wqhjAx2zuTHb a0XZd80bjLvb6FrBNVKNt1UllIBXlaaaUvs+NY/scrhc4ZDelISZh2s4lmKG X-Google-Smtp-Source: AGHT+IGKmG+lPuAjjllQizcCvtBAuharT0T+gQN6KwtZxQUjdrIEPdDkKsykS2XcN3ygROWADJnCWg== X-Received: by 2002:a17:906:16c9:b0:a4a:3d08:bd7b with SMTP id t9-20020a17090616c900b00a4a3d08bd7bmr1373130ejd.26.1712332212812; Fri, 05 Apr 2024 08:50:12 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id lc24-20020a170906f91800b00a46c8dbd5e4sm966105ejb.7.2024.04.05.08.50.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Apr 2024 08:50:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Ming Lei Subject: [PATCH for-next 2/4] io_uring: turn implicit assumptions into a warning Date: Fri, 5 Apr 2024 16:50:03 +0100 Message-ID: <1013b60c35d431d0698cafbc53c06f5917348c20.1712331455.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_req_complete_post() is now io-wq only and shouldn't be used outside of it, i.e. it relies that io-wq holds a ref for the request as explained in a comment below. Let's add a warning to enforce the assumption and make sure nobody would try to do anything weird. Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- io_uring/io_uring.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b7f742fe9d41..c84650b0f7f2 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -931,6 +931,13 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; + /* + * All execution paths but io-wq use the deferred completions by + * passing IO_URING_F_COMPLETE_DEFER and thus should not end up here. + */ + if (WARN_ON_ONCE(!(issue_flags & IO_URING_F_IOWQ))) + return; + /* * Handle special CQ sync cases via task_work. DEFER_TASKRUN requires * the submitter task context, IOPOLL protects with uring_lock. @@ -948,7 +955,10 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) } io_cq_unlock_post(ctx); - /* called from io-wq submit work only, the ref won't drop to zero */ + /* + * We don't free the request here because we know it's called from + * io-wq only, which holds a reference, so it cannot be the last put. + */ req_ref_put(req); } From patchwork Fri Apr 5 15:50:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13619235 Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4723216FF4B for ; Fri, 5 Apr 2024 15:50:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332218; cv=none; b=pK8LoAPd+tt291QS98JlctnSSHP4n00W1e/zweg6i1CEZDnQM/FpC4E0iC/ms+sUzPsc+aRclbt9nkXWpeII6MlHmVaXSxaDvvcFoUqti6aO2VeOWNdbz6LsroaL5YB6Wf1M9N7ny+jq6PdtmEJ92yqTa1RuhKY9eBOuKtB4o3U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332218; c=relaxed/simple; bh=ARdPazd7WB6upcxA5AejUGvyIrW+OVCN8RQdiCudRgY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KgU2TB9VyhCsfNI+n70F8JxaHmDWyRmAPAoSH3nvs5BnR0j57wGnt9EZ1rz+1Q6LR3QaGanRw7GW1hSx2CaQkYa909HX3OlHshnND7uPxzQU5x1/5wfF2LLlpQidWzJjL/WdSugxbZ/FDk+2NGKKeaBJYbX0diPK2BlzsOiEnRs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=STAVmwH1; arc=none smtp.client-ip=209.85.218.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="STAVmwH1" Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-a4e60a64abcso595662066b.1 for ; Fri, 05 Apr 2024 08:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712332214; x=1712937014; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HV9Mw5Jjz68HHXUpaAbRtDprLc4zzD8EsXydMkjUPWQ=; b=STAVmwH1P/EFzcsy+39CxF2w7czFjNCigKhJCaoheQqnEQQ+PfqFZo0WAyL9JrxOah oSpntCqTmaKGOUPphKzELIGgnzyDWiUjZ6lToWB7dzs36cllZi5OYMo3SVABB2LSjM4x 4/JEJVe0+oTcUNBoC2bC3s6Ev+F3vtU8JjXatD1foonmwbgGgFlTa0pb1tBY+J1EpE2R UTLt+I3CyUp0JFnsyRstOntou3aNUvekscVL0zmMWCQrNdGh/M/zmT7CR9F9Ns4UHbgE IzqE9MUIDzIp1edm1ptlPbSi9by/pHlak2W8g0LBvNE1X7V9xbT2N49LFgHgR0cvTu6O FyNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712332214; x=1712937014; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HV9Mw5Jjz68HHXUpaAbRtDprLc4zzD8EsXydMkjUPWQ=; b=YIqfRSzHbTREQHdk65/MHim6KprLRZgAEmI6VpGqOgLDQcHaT8F1hzNxxA8vv6t7rZ caw9ANOx1zJ0NCJ4nhvMpQlq31xS4c7JSB0ATBVHMYH3YIRrrpOWfTWonRWLHZQ4dPOq V9dVZFSGJFDLi4y6sOvBBw3tJCdZ6iZuqpXbpTsIRMmsv8LFxIdLlocaKMlXnmCmSL+6 Sdhkk26weW1oFN0ibqyGZlJqitJGAqA9vERr7VBlDx3QUnV5ihzEqY/lr8LZLrwghHVi WzabXbrpjqNqlvSRztjaUKmJQFVuAGfzXcwKNeYoYzqYKXcy1PMIkUWdHBGmaxkivt4i RQfw== X-Gm-Message-State: AOJu0Yw3A1ZbQIiSzXRy8IqKRz6/MmL6JLGpvfHMO0Dmb4fT3f9Ey29u /Yfib8hvaVdJ2SEgOBmMeu4t9rGlo0RpB2NxTHLeM19sM23ujjuBal3/2xVS X-Google-Smtp-Source: AGHT+IGIrSP/RndXMLxuJLdypqBzCg+wsIgmvgVcATX+4WQAeeQwEuOA3xDjKDmdUJSRmV36LZ2r1w== X-Received: by 2002:a17:906:b311:b0:a51:9056:b878 with SMTP id n17-20020a170906b31100b00a519056b878mr1508723ejz.35.1712332214495; Fri, 05 Apr 2024 08:50:14 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id lc24-20020a170906f91800b00a46c8dbd5e4sm966105ejb.7.2024.04.05.08.50.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Apr 2024 08:50:13 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Ming Lei Subject: [PATCH for-next 3/4] io_uring: remove async request cache Date: Fri, 5 Apr 2024 16:50:04 +0100 Message-ID: <7bffccd213e370abd4de480e739d8b08ab6c1326.1712331455.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_req_complete_post() was a sole user of ->locked_free_list, but since we just gutted the function, the cache is not used anymore and can be removed. ->locked_free_list served as an asynhronous counterpart of the main request (i.e. struct io_kiocb) cache for all unlocked cases like io-wq. Now they're all forced to be completed into the main cache directly, off of the normal completion path or via io_free_req(). Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- include/linux/io_uring_types.h | 4 ---- io_uring/io_uring.c | 22 ---------------------- 2 files changed, 26 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index b191710bec4f..9c49aa2dac38 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -347,10 +347,6 @@ struct io_ring_ctx { spinlock_t completion_lock; - /* IRQ completion list, under ->completion_lock */ - unsigned int locked_free_nr; - struct io_wq_work_list locked_free_list; - struct list_head io_buffers_comp; struct list_head cq_overflow_list; struct io_hash_table cancel_table; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c84650b0f7f2..b20ee6a0e32e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -335,7 +335,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) init_llist_head(&ctx->work_llist); INIT_LIST_HEAD(&ctx->tctx_list); ctx->submit_state.free_list.next = NULL; - INIT_WQ_LIST(&ctx->locked_free_list); INIT_HLIST_HEAD(&ctx->waitid_list); #ifdef CONFIG_FUTEX INIT_HLIST_HEAD(&ctx->futex_list); @@ -990,15 +989,6 @@ static void io_preinit_req(struct io_kiocb *req, struct io_ring_ctx *ctx) memset(&req->big_cqe, 0, sizeof(req->big_cqe)); } -static void io_flush_cached_locked_reqs(struct io_ring_ctx *ctx, - struct io_submit_state *state) -{ - spin_lock(&ctx->completion_lock); - wq_list_splice(&ctx->locked_free_list, &state->free_list); - ctx->locked_free_nr = 0; - spin_unlock(&ctx->completion_lock); -} - /* * A request might get retired back into the request caches even before opcode * handlers and io_issue_sqe() are done with it, e.g. inline completion path. @@ -1012,17 +1002,6 @@ __cold bool __io_alloc_req_refill(struct io_ring_ctx *ctx) void *reqs[IO_REQ_ALLOC_BATCH]; int ret; - /* - * If we have more than a batch's worth of requests in our IRQ side - * locked cache, grab the lock and move them over to our submission - * side cache. - */ - if (data_race(ctx->locked_free_nr) > IO_COMPL_BATCH) { - io_flush_cached_locked_reqs(ctx, &ctx->submit_state); - if (!io_req_cache_empty(ctx)) - return true; - } - ret = kmem_cache_alloc_bulk(req_cachep, gfp, ARRAY_SIZE(reqs), reqs); /* @@ -2741,7 +2720,6 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) int nr = 0; mutex_lock(&ctx->uring_lock); - io_flush_cached_locked_reqs(ctx, &ctx->submit_state); while (!io_req_cache_empty(ctx)) { req = io_extract_req(ctx); From patchwork Fri Apr 5 15:50:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13619236 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED89416F901 for ; Fri, 5 Apr 2024 15:50:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332219; cv=none; b=Iwu5YbNnX3Ys3FBoHp/S+0beHpImRLfL4Pc6J5QUtmSc3z9tXUmzRxeESBCeJmNvrHCLqgbAw5A4Em99uoaDDEXzv7ydDsQASvhVJL/Am2GRJ6D3S+0qmT3IsItM6+aJMD3D2wYEOnyNwf6zRndj/luEPJXgJkLx/qPlqODLA0s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712332219; c=relaxed/simple; bh=PW/UvTZciwyckou/rsg/Z4Tl6ixRaIRr4C8xP/RXwyQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fDmdWjn1L6IaatToheO/FpF1IcH/Lu/zVfRU4xPE7EHqde8fFW1uKjPzyjBxbiR0TJQ9SLFeN4+u1K8Irm5P6puFLWwm4HmGz2ABf980frS0tlb6BaHH1Zv68d6kLRJAkW2uMTdAsbhj30lak6WlIodinWe4W8ZTNZaG3eaOZ3Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=R2Qg6rHa; arc=none smtp.client-ip=209.85.218.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="R2Qg6rHa" Received: by mail-ej1-f52.google.com with SMTP id a640c23a62f3a-a519ef3054bso134019666b.1 for ; Fri, 05 Apr 2024 08:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712332216; x=1712937016; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rbECNWgvyLz4lhV88KMe5XuGXgBJQ2wLT/wNj0ZcIHU=; b=R2Qg6rHaXzdcBlkKAMQiwGgfRGChaB/h93WH3hvencHl38bWSlvFtRlJzLvZR75gLi 7i7dppQrNzWTNkgpP6J57H9gu/rXadU7t6TVSK/9S5fAz229dflE0kmJGPce11nMe9WE 0k30Edv+Z8lXuBF8aNrscnWgikSo2MTk4seIEYYI1Xqu/2aTDem4TaEQGHHO54dbEAMy xIexrFdejMV75j5ZpW8OtN7pWLJWkfS5/xyQ+dZ1GIhf6PYI/yH0V08yjXR1PJExfTqT 7fFjsSmANH4OgaJBJvDtLrriQksWNUWsW1+lQp/UYAeVvvL4aNUbG7dodO1wNsIi92at fRBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712332216; x=1712937016; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rbECNWgvyLz4lhV88KMe5XuGXgBJQ2wLT/wNj0ZcIHU=; b=E0hXReUIJR/gKN9LOQB6G7pQHK6xcZUhXev00PoeCWJdJ0uPrSyntVU4Stu4SxVjMr DKZ8WYHMHW1Eyp8Hv9hq8yS9EU0VCi4uY/ZiJDW/kxs3hFRlbOnD3IeBKeayL2PYSGqx YQDubjs+S/DeT9r8hgEAAuFA5bbyZ9aaPkUwQxieb6SVzi6riEa1ADF20kkoK6uV/7qi U5deTE9UuFVqQAS+mguwjdOjF02hdl/pMgg41CI/f+T2x6TFfCr0b3oBJJzphjCDZ/jB GVHszPbiPfMZJCMn9bUUCW6RJyxzVRqOC/iPqs/V5qJd9LNH5kRydjYnxTvjHKs07u54 fDiA== X-Gm-Message-State: AOJu0YykaU5mGtZr6Jw4nGGpMfdmaSASTrs9pIf3ih3xJ2v1tO2DCFsq KNNC1NjgYNuOFGimvMVtgnCBsA4XfkyOOaiNJJzMbNEYFfwjqKflAm5BEZYa X-Google-Smtp-Source: AGHT+IFzKsPrSJBg0Bhirk6U4AjuZ9ORTDJv6ro4m29ttMbh7f7XO+A1EylBks21K+xn1dIXtXKahw== X-Received: by 2002:a17:906:709:b0:a4e:9190:eb18 with SMTP id y9-20020a170906070900b00a4e9190eb18mr1274668ejb.29.1712332215428; Fri, 05 Apr 2024 08:50:15 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id lc24-20020a170906f91800b00a46c8dbd5e4sm966105ejb.7.2024.04.05.08.50.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Apr 2024 08:50:14 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Ming Lei Subject: [PATCH for-next 4/4] io_uring: remove io_req_put_rsrc_locked() Date: Fri, 5 Apr 2024 16:50:05 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_req_put_rsrc_locked() is a weird shim function around io_req_put_rsrc(). All calls to io_req_put_rsrc() require holding ->uring_lock, so we can just use it directly. Signed-off-by: Pavel Begunkov Reviewed-by: Ming Lei --- io_uring/io_uring.c | 5 ++--- io_uring/rsrc.h | 6 ------ 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b20ee6a0e32e..909842cb1436 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1451,10 +1451,9 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, io_clean_op(req); } io_put_file(req); - - io_req_put_rsrc_locked(req, ctx); - + io_put_rsrc_node(ctx, req->rsrc_node); io_put_task(req->task); + node = req->comp_list.next; io_req_add_to_cache(req, ctx); } while (node); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 83c079a707f8..c032ca3436ca 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -83,12 +83,6 @@ static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node io_rsrc_node_ref_zero(node); } -static inline void io_req_put_rsrc_locked(struct io_kiocb *req, - struct io_ring_ctx *ctx) -{ - io_put_rsrc_node(ctx, req->rsrc_node); -} - static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) {