From patchwork Wed Apr 10 01:26:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13623452 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D92865C for ; Wed, 10 Apr 2024 01:27:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712433; cv=none; b=MSkjhjoYlag+iaZ4pkdwrse5vYdQmt366cw9NrW6ti69+Q13QzwMeETCbSvs63VWMf+Qtkb+p6Qn1bMZn7z+6qbyfhAJfEqSkWX4Wyj5+R980jj45wqp2s8RSsDpE2ybIaLLykxRk9dpQa0rid4g2PY/0k/vkdetWE3iIFlnuTI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712433; c=relaxed/simple; bh=3ukB7rgy9eN+tu/3UVQoUkyNBH6lutv+Ep5PTj8gU3g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eSgbf9Jj1UzbScZx+9lxkgpOPvoqdKQNXMiuu/lYX3DgILVQKeafjrFQwb1ipy4Etx/wgK0zYG5Up4+dYmMNJCbMstBxfwb/3W2u4sNrkW7zOrB/K/PEyaT3LFTpU6eZnXkJtmaFppeZ2qX/DPku/4PQBO41Bj3oZ6u7kXYNxNY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Onr6nkP5; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Onr6nkP5" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-3462178fbf9so582421f8f.1 for ; Tue, 09 Apr 2024 18:27:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712712429; x=1713317229; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D0LSk2ZnzPF0+Dkd4LZNo7M6iJ8DiupkfPWyBVZ3fxk=; b=Onr6nkP5C188UNjeSi/pHDNsAMLYEBOgSOFcHGa4hFmLRMwoJNaDANeK57oRdX9Ha3 zePUhKvQ6fcqSCZlkmibQp4U+36v6vjLVEF5YSj5W6Wagbu8iBhZ27lex6HfHf3TMJaT zA2kCMwHhXl3PwFEZgMu/4xVycnE4nbKyV6f8OFNWvKzVvGOF+LRmpUt6XzQfKt1a0cP cov0LaV7NQIQX1u7KBQsexXPtNj5I4EL571iWtJH3u9vjO9f7XGY6QptKiKNfKF3Podq T1EpSKPIUrbYjcGn8chcF6SXqrYM0vRrleMCdVQR5b4097ikArfynO8I8iieT4rkyd9A La/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712712429; x=1713317229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D0LSk2ZnzPF0+Dkd4LZNo7M6iJ8DiupkfPWyBVZ3fxk=; b=vjoUk52mLhuIP5necjkBwl8biOYRkptVGhOazzrzaHkSi6B+pGdGzh7uc8BcMueXGJ gOLQmSes/TbxOg3CKxkfBr0FEIWxnLnINzBUfKQTWss5RN8URDkgqEUUtfnTxRJZXZro Fla72JjqEENchY17etNzb8jk3I7tkHCE1iVK9akHDYoKwcUM6oKsjaaSivhJGBkGYYtZ E4hsIWBY+hl3DVyoU2ruWSNXEqy/9Uwj+97DbGNy7Kg2PoICevpNdaoGr0n7YkWk1xea xMrCdBSi+Ip3NuBD3jsT23e5rCgxXmboHhcMPrFSj7dbzIHbdVyuduS00j7Hz6d3nYKZ 9vXw== X-Gm-Message-State: AOJu0Yy260onIww65rco45CSMtnoQbP1ap7zRyp3Aw1ma89dzDolt9KU LUKU8H1lZfUL4RUMU0tSxyId0ZPOIpdS8BmWam4QWzkc+tYe9AGJewuhOcDc X-Google-Smtp-Source: AGHT+IGmEqFw/p0OeGWUDzJVLGovCFdd0MpgfGSE2JSkNDcS+Jj7R0aejjbaPk3tDoNuYubE/h6neQ== X-Received: by 2002:adf:f783:0:b0:33e:69db:bf9e with SMTP id q3-20020adff783000000b0033e69dbbf9emr1097470wrp.67.1712712429483; Tue, 09 Apr 2024 18:27:09 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.24]) by smtp.gmail.com with ESMTPSA id r4-20020a5d6944000000b00343b09729easm12737693wrw.69.2024.04.09.18.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 18:27:08 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/5] io_uring: unexport io_req_cqe_overflow() Date: Wed, 10 Apr 2024 02:26:51 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There are no users of io_req_cqe_overflow() apart from io_uring.c, make it static. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8df9ad010803..7e90c37084a9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -819,7 +819,7 @@ static bool io_cqring_event_overflow(struct io_ring_ctx *ctx, u64 user_data, return true; } -void io_req_cqe_overflow(struct io_kiocb *req) +static void io_req_cqe_overflow(struct io_kiocb *req) { io_cqring_event_overflow(req->ctx, req->cqe.user_data, req->cqe.res, req->cqe.flags, diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 1eb65324792a..624ca9076a50 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -62,7 +62,6 @@ static inline bool io_should_wake(struct io_wait_queue *iowq) } bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); -void io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); bool io_post_aux_cqe(struct io_ring_ctx *ctx, u64 user_data, s32 res, u32 cflags); From patchwork Wed Apr 10 01:26:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13623453 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A43FDEC2 for ; Wed, 10 Apr 2024 01:27:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712434; cv=none; b=B+RXC3Jft897RpFHeoCsjit7stfMeVdTVhx8thTKcO7eAY99QSXXOhQ+OB9u1cOV5+Up2pMEPbMDGftYbaXDLGVf0RTQDxyraK53hc/6Cz7nZNnguSYpa2tYKcjbratj3iQ/DvXZVND2jJol5w//errZ4c3aSbXZ/PemTOcun5k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712434; c=relaxed/simple; bh=W7w2Q9T23ojdDHLhaaU3uq16x93wWqFxci1pZqGoMQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RtUqfZ0s5fN7N+qjxqEWO2zPpyQ4AllD08coEzeT1Zk7XuROECvxx+vGwb0seERo4J7nRH5e7Fs4z723jduP/UFmJAvxkNBWhLyJwqhiDjyFuvZvxvkWelFOL8LGdBojbGc1utZ5wgtJbtwQ+7h3gHbioaUWtjatL9u9rp+SO/Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VDqeh88D; arc=none smtp.client-ip=209.85.221.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VDqeh88D" Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-343c7fae6e4so4106760f8f.1 for ; Tue, 09 Apr 2024 18:27:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712712430; x=1713317230; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IuWE7ZWtFQ3OwiocCsLF5BJ6o/QQL/vOgTFWzXUf7Vs=; b=VDqeh88DJI8T07eY88vaseL1i77WpL+/ufzaG1OLHgaRyIMbRmmPpOB9dlN1ofQ7TI ghxweDPRyS2nry6XPcpbWJdOWooeHxVPCiF6pHfPmBwUZAvmhgaNlLjw2oWG5goOFt2S 6Z93eXRbCRohMdft8TPBAe+JdquYLKTE0PI71kJoK8JnxM08LWmCsZeU1VxLyIbcPqU4 PIdwuQcXj533ppoLqN3dUBzLLnn7zaDngOmljWj998c1m/oPVnznT1oGPjyKo5xMd4NB kqSD0JRDvzuXQFND9iDd+k3sgP9w7PIRzMCbL5zYD0Te3um99Iv0N07Ih9Xn+z7I7xHV FErg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712712430; x=1713317230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IuWE7ZWtFQ3OwiocCsLF5BJ6o/QQL/vOgTFWzXUf7Vs=; b=UZR3AU6Dlcjw6TrGM9er5EZLSuHX7L+P0cIGgqujFiCt7VGIyCs+gV+/hYUccJr6pp IdItXMl235sLskVwYdzRmbwwuYNRXNNVVEReIs64pOIQ6gXQj2NSLzGFt53NzFa/jg7f 84RRQmDdgLXmdWumYVftSVduE2VA58g7mMp+BB13VXcP8tMfPqSEPFG5LEXNnVunXq/I RIWzualZIfy6gF+WMPd28kgaPpQQBwMvyHVj++Ck8NztntCd/s6YUdfwVSNPMSDcBvud wlUICA2AIFswhXR17IioMOC+rSYHlIkMSwuFOz9DbKxOzlXChMFR3voPE8mfQVyX7zJA iyqw== X-Gm-Message-State: AOJu0Yw7bkhOkjsIV+ENJMUSwOQWjMxqw2YRXSJNabCjv34RrjDVuRvN Ra/7B7vLjUwuU57R9V0mvKEWyD3JPLniQgqzTORnPRq1/s2RPxwPxwwZ+duw X-Google-Smtp-Source: AGHT+IF+P4MxFok1WcfGuYpSNlGwnfnCjRwLb06i7SR8XFJPuwgktkzMzzbr4RttKr/wrVOc7Tk3ag== X-Received: by 2002:a5d:634a:0:b0:33e:c7e9:3360 with SMTP id b10-20020a5d634a000000b0033ec7e93360mr736318wrw.18.1712712430383; Tue, 09 Apr 2024 18:27:10 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.24]) by smtp.gmail.com with ESMTPSA id r4-20020a5d6944000000b00343b09729easm12737693wrw.69.2024.04.09.18.27.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 18:27:10 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/5] io_uring: remove extra SQPOLL overflow flush Date: Wed, 10 Apr 2024 02:26:52 +0100 Message-ID: <2a83b0724ca6ca9d16c7d79a51b77c81876b2e39.1712708261.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 c1edbf5f081be ("io_uring: flag SQPOLL busy condition to userspace") added an extra overflowed CQE flush in the SQPOLL submission path due to backpressure, which was later removed. Remove the flush and let io_cqring_wait() / iopoll handle it. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 7e90c37084a9..55838813ac3e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3238,8 +3238,6 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, */ ret = 0; if (ctx->flags & IORING_SETUP_SQPOLL) { - io_cqring_overflow_flush(ctx); - if (unlikely(ctx->sq_data->thread == NULL)) { ret = -EOWNERDEAD; goto out; From patchwork Wed Apr 10 01:26:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13623454 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43EDDEDE for ; Wed, 10 Apr 2024 01:27:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712434; cv=none; b=W12BK4a5dcOxsM/mIZDDLkhKJwcN2R7PN4T1akPwuDyrqDPqOeEjPmhTNYN5rp952AYNOZETW9gEBc+g5MQM6cJVLBW2osKUxloehHMg3BFB4Lkwv7Ka1FQASRp8F2BocfPpTECcVKkdVcGCFZ1pdroc1ym2XuoYsSIPVSkztng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712434; c=relaxed/simple; bh=+cFsA4uEa4mfUPDhX0Ql+HKvPY8/JsBHl53sJDVBfEE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CJRvaIHLbhtti3ue3nj6JgSvMMj42k1Jsl2HO2EPN4CoKO7iXZiH26lDsZz88E0+e70OiQTU4TjswvGlg9zBz60CypczXZo854fLuM8jrB8bXQjxOdAWnK5S4CT5eUGMmR1JO/m/Lk0j3I/rqY4fnxnsrl3voOD6CFH0MhoeSqc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LH8dTDMB; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LH8dTDMB" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-343bccc0b2cso4520507f8f.1 for ; Tue, 09 Apr 2024 18:27:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712712431; x=1713317231; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J/0EWDa/dNkvJHojCTHVu+tnL215oLyOWobFjajXkfk=; b=LH8dTDMBYbDDtGVzyJfEnsOw7jgyT1Vom0gPhlsCwgZ/h6m0X8K0CzKNiWvbd55wSu uDSC70sP4kM9NDqsgViX2dDDO/SBVmV0+DKaLmYQQ//Flg+mavv7j/XMlfZa9l0+9wK/ D6JgtlFX663CfXIw/9Y3cLx6ipEfANoddgrv8gXijLcOAoBTGl/qU67YhfsC5Xxx+XGk LESTtpufRj6d4mTpzPPep+blqO53/GdB89Y+BGoztIXRoCmmaP8rtD9uaqj3P62Ac9bD +pf5iXI3F7jvCp95iUZdNfkGwXXl8k3wV59kub0rH0JwR4Lado1m2iHvZenTECpQOBqa EeKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712712431; x=1713317231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J/0EWDa/dNkvJHojCTHVu+tnL215oLyOWobFjajXkfk=; b=u+kg0yAKI/fmWfH4lsyUWqipySyb/TP81MmP+LLCD8v6ifcuSzm1fsZ2ECOhQIRMPT 5iijBb9NpPm2moyAo7rDvK3dJ9C8AroCSFYPF1/npSiPDRQ6gDBjIEPJA4B1GEuIq5ma skgJ5D+os8BGoBMWo1QaAGzewMCkK7VS1w+vBB+sQ8J/q0MZ/84Wd86PT6Tk6ZpoN7Vg /NmpVHS5LVKW0YLhOMo237BFDmvA0zmiwtgGjxj4mA2MB/RNEQAD0DnLGyn0OkB5qyqz drInvnXVWC26Pxk4Goe0R0DpGW0FXZtW71N72wqeRyC1zk9W4Y4+MebkhQjL7kWVHnfT DJZg== X-Gm-Message-State: AOJu0YxMXMy+Pf4dLZ4S5KMSJzDL6KyeIGacqfmTZwMxx6WjMEpkJapW UIw3EsdwuCfxcCANwUTOorLe2NRwyKBdBZKHwlVbrdv0QIEf4LTS4WaQQqxg X-Google-Smtp-Source: AGHT+IGuO2v6yRyy+yJzXofgxPkhktJQkszPpktp0NghMACcXHxSjxmfOFQfRC3f+elSzlnf6b3ISg== X-Received: by 2002:a5d:4a45:0:b0:346:408d:4337 with SMTP id v5-20020a5d4a45000000b00346408d4337mr642488wrs.9.1712712431490; Tue, 09 Apr 2024 18:27:11 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.24]) by smtp.gmail.com with ESMTPSA id r4-20020a5d6944000000b00343b09729easm12737693wrw.69.2024.04.09.18.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 18:27:11 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/5] io_uring: open code io_cqring_overflow_flush() Date: Wed, 10 Apr 2024 02:26:53 +0100 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There is only one caller of io_cqring_overflow_flush(), open code it Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 55838813ac3e..9424659c5856 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -726,12 +726,6 @@ static void io_cqring_do_overflow_flush(struct io_ring_ctx *ctx) mutex_unlock(&ctx->uring_lock); } -static void io_cqring_overflow_flush(struct io_ring_ctx *ctx) -{ - if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) - io_cqring_do_overflow_flush(ctx); -} - /* can be called by any task */ static void io_put_task_remote(struct task_struct *task) { @@ -2452,8 +2446,9 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (!llist_empty(&ctx->work_llist)) io_run_local_work(ctx, min_events); io_run_task_work(); - io_cqring_overflow_flush(ctx); - /* if user messes with these they will just get an early return */ + + if (unlikely(test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq))) + io_cqring_do_overflow_flush(ctx); if (__io_cqring_events_user(ctx) >= min_events) return 0; From patchwork Wed Apr 10 01:26:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13623455 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9378A1388 for ; Wed, 10 Apr 2024 01:27:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712436; cv=none; b=aeZrnyCelTcGpdksvnlklCfQggu5P9I2iDO3X8bhGwPyYbL3UtMiFPibVMPB2QPDDD5O2psZ1axe6YjEmOBKe/E032kP+RJdfvr9rFYkeX0P/4A9V+H3IYa6E5h2cMAxVOoKLA7erHb+LXjJ1kA/a1nN9Op7kp0iVsM2dY2Pzfk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712436; c=relaxed/simple; bh=S4NaWumQkuVUStYPiLt2TE9vK9XhYW42xTwrzgxFUBc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PjNRDohumBest+mfrP9iZDBg/dCh258YVNS8NDEHdhA79/hRTYf3YqthuVrGqWQLOP66K/jU57nfIlYWeVWWV8kV6kGzANqJSNT+xV5kq22l7jqJ4NOpJi07rFzeKPJ9D4/2m0kSZwZ7sZc7oBgR5SqjP5Rf/xkKyL1kO7Yhc1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M9u3eUcV; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M9u3eUcV" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-34665dd7610so99350f8f.3 for ; Tue, 09 Apr 2024 18:27:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712712432; x=1713317232; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HUT3VaoONdExdEm2PmzG4eWlFh4iS4s0PUvVjZnVNPk=; b=M9u3eUcVZKo/jAKOJEGn1z/bPZWHOhiaqWMsIJyX/xlV/gbuAetKqUip9x2JJ0B7PD GpL02dctVFlkVeCyTaJhEBtPT2jaE7Y4uGtPuE56Kf6YRRg8jUbtK+CpF1vC8uAWfnGn OPykLZdbrXpTM9CCRHyyfVF3Q9+98S52YQ62GLQNvgXk6R0qvWohsxCQ8jBO5DDifYZE iONzh4wTu04etQdAasJfMa81qQ5h6GJDSHheW8KRObzlyKotw4PU00/voY27Dr79RP/t 2g2xmzC/WXv3OZWXVLvFs4w6FvHAUPonqPwzISFxFopzKRkk7GV9hVLe7SvqeyTEOdpX Z/Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712712432; x=1713317232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HUT3VaoONdExdEm2PmzG4eWlFh4iS4s0PUvVjZnVNPk=; b=PuwUlWH4y/Ibjtl7yC970SLdP6IvA0+T9HCgqMMozeVcL8Nvjm+69HyiUV0ojwyCoC ltyqj1PWjgAX2T24GWxaVPNNmN0reP6tNWCfm9UJe/QG6H8J+xhlJBQ420wS6LsfSlzB ThF7WJK8JHHZcJ3+G2q8ivhj9in+PQICykbl7Gi4Frbd+Sc6yaAjwtva4ZmwMcNoi6Ba h5D4kn1KroLV5LYJGmMZh3OJ+TOGl5x89mT55INfei5vLuWAlcm/gw6uLztDCFOKtwOh VkDBYHyaR9BfMnaowrmyCgyTqpqoFpfg3Be0e1NXsHyn5/eE2JIVARyNEZKwX4YU7YoZ PCbg== X-Gm-Message-State: AOJu0YyBs/OSVlAAUElfVQPUGt7P7RvomMI7qZAPeQVdH0hwBV1Eon3D 7SkocTYxZt4VeHv2QFAlkzPCm179THZBp0s4Am1uw17DjDpvllFZZdPTuizr X-Google-Smtp-Source: AGHT+IG2h0/oPUulM0TF1mLS1gnMHcNfhHejxtqOTQnHl/ZDXH9LrW7dsw7bQvonrDSRf2sKJMOdsg== X-Received: by 2002:a05:6000:1549:b0:345:c359:d34a with SMTP id 9-20020a056000154900b00345c359d34amr886618wry.59.1712712432295; Tue, 09 Apr 2024 18:27:12 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.24]) by smtp.gmail.com with ESMTPSA id r4-20020a5d6944000000b00343b09729easm12737693wrw.69.2024.04.09.18.27.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 18:27:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/5] io_uring: always lock __io_cqring_overflow_flush Date: Wed, 10 Apr 2024 02:26:54 +0100 Message-ID: <162947df299aa12693ac4b305dacedab32ec7976.1712708261.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Conditional locking is never great, in case of __io_cqring_overflow_flush(), which is a slow path, it's not justified. Don't handle IOPOLL separately, always grab uring_lock for overflow flushing. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9424659c5856..d6cb7d0d5e1d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -673,6 +673,8 @@ static void io_cqring_overflow_kill(struct io_ring_ctx *ctx) struct io_overflow_cqe *ocqe; LIST_HEAD(list); + lockdep_assert_held(&ctx->uring_lock); + spin_lock(&ctx->completion_lock); list_splice_init(&ctx->cq_overflow_list, &list); clear_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq); @@ -689,6 +691,8 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) { size_t cqe_size = sizeof(struct io_uring_cqe); + lockdep_assert_held(&ctx->uring_lock); + if (__io_cqring_events(ctx) == ctx->cq_entries) return; @@ -718,12 +722,9 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) static void io_cqring_do_overflow_flush(struct io_ring_ctx *ctx) { - /* iopoll syncs against uring_lock, not completion_lock */ - if (ctx->flags & IORING_SETUP_IOPOLL) - mutex_lock(&ctx->uring_lock); + mutex_lock(&ctx->uring_lock); __io_cqring_overflow_flush(ctx); - if (ctx->flags & IORING_SETUP_IOPOLL) - mutex_unlock(&ctx->uring_lock); + mutex_unlock(&ctx->uring_lock); } /* can be called by any task */ @@ -1522,6 +1523,8 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) unsigned int nr_events = 0; unsigned long check_cq; + lockdep_assert_held(&ctx->uring_lock); + if (!io_allowed_run_tw(ctx)) return -EEXIST; From patchwork Wed Apr 10 01:26:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13623456 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 306CB15A4 for ; Wed, 10 Apr 2024 01:27:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712436; cv=none; b=r62QOwD2tp07g6XerfrNIhpIXlfT3Cgp2Pt5oAtQpL4VwWsWmtxqY1XlPOb0wea2wdemyLHnAc2UdvmzECOd8bwLBgbshZ1ephMhu3ABsAHGzdjDteTDBg8pi7zRr5jdunESQoNrgg4hJVb4Ucnz4x3geOMSHPmkveG3KzZmbrA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712712436; c=relaxed/simple; bh=M2OwH7UgBGLSXnOA61HS0oQI62bTMeOZlWXGfZVIGp8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MyzWkoUNUj/NkGiH9K2ylXEBugTZS2yi0+9ZtlKH/M7trRPZ9nRs137X2hVqy7RC/68Wy2/CjRu9K2UHThX06z86V/cPfObwhKwyjqVUJC+sG8gBPU2xUIzNOSvFBc797fxVQ9BKJsokrpg6wfCo4Au4WCVaNAQVBqCMnMRFGA4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q71b6cAZ; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q71b6cAZ" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-346359c8785so676395f8f.0 for ; Tue, 09 Apr 2024 18:27:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712712433; x=1713317233; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FNziGC8hKSnzMEKFyxJNNwyB2Vte56ZYQCL3ALYbRl8=; b=Q71b6cAZYgfYr1aBVP5LzrEHT2bhevfRbnMc9TPP1zJ8c2TKZES3915PXurOu+rxVr z0Ms8q1iFuWu0OorrtaqOhrf4HWNPD+KV5j3f+CKNM4RIE+infHjqlQv6qxvEF7wY2i9 b65Kl9Vtt7BVcOUMv3UZLq89XO1gAZs5ZvbhVFZES/aIZ25+ZP+WCz4ocdXlZv0Fkoir qLC2FO28lTKqhVFl+V8cWB9/zQKnPMmFh++lEfbS+TDi0pZkmaSIxiEZJgsZnAyvngAp +ct1lJeBqNV7mzyhXDw15aIa0B8KLWqz16tyXpEqlblH5kWVRRs2TMR9enmodjuHP70g Upzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712712433; x=1713317233; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FNziGC8hKSnzMEKFyxJNNwyB2Vte56ZYQCL3ALYbRl8=; b=KHS6jmGEUhHJkSBqhbm83o4K+9/jRdEG3O9KkSTT+nKCDldMAE5spfSxnB0hI0ywf2 qdpYyV+wKdMyNdO1M9UhBqggNIP07GslZLlb9Tu8j8ponRtM+rtDLxpV/mj8d5hzFQpW VgG+bQ1CUMHszuBmVS2IWYP1gpifx2Huk38sn3XM8mHeF9nei7aCIaAIIiuWMOoB9l/1 WVvS9AxL3vCsHaObJcNsvK5NZxAbV+21ZinKBVi5HhIFfguu0QFhEt43pLudcW3b2rdE nzZU1tVDmAiGK+EgEK3/C6k7Z6ZSjJKSY0yFHwLm2C7Jm+74yt6xp/7FnuUBMO5s8CLm +s3g== X-Gm-Message-State: AOJu0YxynHuWzlAjEVDHDHq0A02XC13fqgB/kpozRQv5u6ByweV+LX9s YXtBO6gtXZmUa4OMIuOcRE7B1Jnna1uRwK8Bmpj5Tg8jJFjw+qZtV6ja1gfE X-Google-Smtp-Source: AGHT+IEFDxfVSDSAhGK3F5PE9uR8uNE2dPD/XP/KJzJjPEegQ5nBm73L+dnpuGN3TQBFzrkfMJ6goA== X-Received: by 2002:a5d:64ea:0:b0:345:6c39:5f4c with SMTP id g10-20020a5d64ea000000b003456c395f4cmr983296wri.12.1712712433436; Tue, 09 Apr 2024 18:27:13 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.132.24]) by smtp.gmail.com with ESMTPSA id r4-20020a5d6944000000b00343b09729easm12737693wrw.69.2024.04.09.18.27.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 18:27:12 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/5] io_uring: consolidate overflow flushing Date: Wed, 10 Apr 2024 02:26:55 +0100 Message-ID: <986b42c35e76a6be7aa0cdcda0a236a2222da3a7.1712708261.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Consolidate __io_cqring_overflow_flush and io_cqring_overflow_kill() into a single function as it once was, it's easier to work with it this way. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 40 +++++++++++++++------------------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d6cb7d0d5e1d..7a9bfbc1c080 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -668,26 +668,7 @@ static void io_cq_unlock_post(struct io_ring_ctx *ctx) io_commit_cqring_flush(ctx); } -static void io_cqring_overflow_kill(struct io_ring_ctx *ctx) -{ - struct io_overflow_cqe *ocqe; - LIST_HEAD(list); - - lockdep_assert_held(&ctx->uring_lock); - - spin_lock(&ctx->completion_lock); - list_splice_init(&ctx->cq_overflow_list, &list); - clear_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq); - spin_unlock(&ctx->completion_lock); - - while (!list_empty(&list)) { - ocqe = list_first_entry(&list, struct io_overflow_cqe, list); - list_del(&ocqe->list); - kfree(ocqe); - } -} - -static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) +static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool dying) { size_t cqe_size = sizeof(struct io_uring_cqe); @@ -704,11 +685,14 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) struct io_uring_cqe *cqe; struct io_overflow_cqe *ocqe; - if (!io_get_cqe_overflow(ctx, &cqe, true)) - break; ocqe = list_first_entry(&ctx->cq_overflow_list, struct io_overflow_cqe, list); - memcpy(cqe, &ocqe->cqe, cqe_size); + + if (!dying) { + if (!io_get_cqe_overflow(ctx, &cqe, true)) + break; + memcpy(cqe, &ocqe->cqe, cqe_size); + } list_del(&ocqe->list); kfree(ocqe); } @@ -720,10 +704,16 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) io_cq_unlock_post(ctx); } +static void io_cqring_overflow_kill(struct io_ring_ctx *ctx) +{ + if (ctx->rings) + __io_cqring_overflow_flush(ctx, true); +} + static void io_cqring_do_overflow_flush(struct io_ring_ctx *ctx) { mutex_lock(&ctx->uring_lock); - __io_cqring_overflow_flush(ctx); + __io_cqring_overflow_flush(ctx, false); mutex_unlock(&ctx->uring_lock); } @@ -1531,7 +1521,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - __io_cqring_overflow_flush(ctx); + __io_cqring_overflow_flush(ctx, false); /* * Similarly do not spin if we have not informed the user of any * dropped CQE.