From patchwork Wed Aug 7 14:18:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13756354 Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 739725820E for ; Wed, 7 Aug 2024 14:17:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040263; cv=none; b=ogeEcDeOoZeY0iR9Gh7fCpppvOhrgVyxbJSCWZIbFXUuoFQ3sJMNqscpSnMAWH7i0Jc0wDJoXVgu8k1Gyz1b01ZQe0GFc230BT4CXv7J2JMsRNW4H90ghd6h627JPUx7RQHvPwYPvUafvoOV45vT64XtQBw4PwFUFYYu2dpcWHQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040263; c=relaxed/simple; bh=yMnMp6sYGz4/6Z7wV25uL/n2ygpp95FpW6CIeQqXrF8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JtksAQrvohDIRNZZg+mLSi9OBjUmS0eSPHtkDAUYMR/vl/xBPoWKVX37FeUsTc7e+e8pdQktu0NBbIhmlb8fBP0MhGS+Yw1d3V9fYdVq3WOgwf0K79I1fWk7chYQGTBFSval5fLmNqfG/6bv0O5PnhtrItXTlHZ6cba7e6LgIhY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Qv21fLbv; arc=none smtp.client-ip=209.85.208.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Qv21fLbv" Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-5a108354819so2445513a12.0 for ; Wed, 07 Aug 2024 07:17:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723040260; x=1723645060; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aCDFSYLRz664AJ+uUNg7sg3G8Ma7AxzpoUVwna2Z1b0=; b=Qv21fLbvVonDg8pOkNtluTdcz1xxnFJl8DAKXjyEX9dIbu9TR0Jp56PdFDEKxmZ26I UTqel6ETRANTH1Kuj6zDy4MJNm29GsX6LkpjD4J79PTqdXXulRgl2rQxKAjhAT9xgoRM Tr0NXfk9eZv1qljG5Uzax/1IaEIEFXAkUtLgnvl2/ApiOIImyViYSUQzDAmJsItHQPYU C4xEG9jCsAs8mEMRsgGKlsdmQGksv6aNjmPH10Do9A2fDmye4OhnerMsLYNWsfuzX4IT A1Hc7qhxBHHB5EFkWnE1g9T9L94kVeHvub1jYKcR4Lbpy2bLrdXmvLncrsbAOW/6hbqW eteg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723040260; x=1723645060; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aCDFSYLRz664AJ+uUNg7sg3G8Ma7AxzpoUVwna2Z1b0=; b=kIaQsxsiqjd/WReUOR8vueo6nG9PazTeBRT3IkeQbrycfaZuXI1ok+5wEpM5vwa687 ngy7/++Un+GmdLe8hEaeGGz5inele4QRP+15kxVln867hsnXIuCOSfYr7DyCRhMCUJ+q iaF/y0PbB6sa+mM9dBkPzrdG2x8rbONBQjOIC1lYSbUpO6WsBuih67Fu8BsKjdBf76H8 tGVCgjx2tmrof2P3ERuGWiyDyZOD32BCqXGWPvXZ3CYHJUuaO49qbm6uJeUdMoimrBQU cLkcBgM9R88+5Zeylj2vbe2/xFHnU7GUAvybOiXuV1AruvEqtoGsz1K1vshbrjuaatPV 5zNg== X-Gm-Message-State: AOJu0YyfjWx/fMF3QUpRyxeoWmiCO8ps/mUBQ6WxwHueeHu34dl3rEQz 2YuFfrB/TLqDoztOfAznxohGKcMrRzcKiKk1sSvuxka1nvcC3ZoVyXStEw== X-Google-Smtp-Source: AGHT+IEaSrWL+TV0mKJWa1lWd1/aGzHypK8qXU6b1bHsupGG92k1WyW+r1xpxjS8roqUjPWdnhUFwQ== X-Received: by 2002:a05:6402:1a42:b0:5a2:2ecc:2f0 with SMTP id 4fb4d7f45d1cf-5b7f36f5964mr13223321a12.1.1723040259134; Wed, 07 Aug 2024 07:17:39 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5b83bf3a37bsm7176928a12.88.2024.08.07.07.17.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 07:17:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH v2 1/4] io_uring/napi: refactor __io_napi_busy_loop() Date: Wed, 7 Aug 2024 15:18:11 +0100 Message-ID: <2ad7ede8cc7905328fc62e8c3805fdb11635ae0b.1723039801.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 we don't need to set ->napi_prefer_busy_poll if we're not going to poll, do the checks first and all polling preparation after. Signed-off-by: Pavel Begunkov --- io_uring/napi.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/io_uring/napi.c b/io_uring/napi.c index 4fd6bb331e1e..a670f49e30ef 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -301,10 +301,11 @@ void __io_napi_adjust_timeout(struct io_ring_ctx *ctx, struct io_wait_queue *iow */ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { - iowq->napi_prefer_busy_poll = READ_ONCE(ctx->napi_prefer_busy_poll); + if ((ctx->flags & IORING_SETUP_SQPOLL) || !ctx->napi_enabled) + return; - if (!(ctx->flags & IORING_SETUP_SQPOLL) && ctx->napi_enabled) - io_napi_blocking_busy_loop(ctx, iowq); + iowq->napi_prefer_busy_poll = READ_ONCE(ctx->napi_prefer_busy_poll); + io_napi_blocking_busy_loop(ctx, iowq); } /* From patchwork Wed Aug 7 14:18:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13756355 Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68FC4339B1 for ; Wed, 7 Aug 2024 14:17:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040264; cv=none; b=ugHTOxryFKml/U4TZb68o4jP+St+KwFeDqOnB7ZzXh/nLrgjhAwUGIHqvKRgcekMU9pjoDrKmdXTa2zq6wh+yMJwr580cE3E5yqs8PLjQJKWk04OD8JTfM+XF5z+wWicQnOtJ87JEu4YHXlNmaGdcU97P4qma2i0Ft1cpZk9LU0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040264; c=relaxed/simple; bh=kbQc3ZU5Pa8Xs4JzWsnWeqlC1DUBEYEF1zOqqUrpdNE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GS693buu0vTOqr9+/jEBIYCmWGlEwB2XgbDvToRB3zZe3GwAlZJfcoqI/bQymZ798bTZmiUe7goiA08LonX++n9wCYKzCd9h84aOg3AqXdgxUrRTO6Uw+sZ/da+G+fN0NNOafp1wOXwCwcxAQRLiKuYz2A49yHnINHvOoG5XfXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZOTW/Fci; arc=none smtp.client-ip=209.85.208.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZOTW/Fci" Received: by mail-ed1-f53.google.com with SMTP id 4fb4d7f45d1cf-5b8c2a6135eso1136582a12.0 for ; Wed, 07 Aug 2024 07:17:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723040260; x=1723645060; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cd+9inN4Ie9i79QY0gLMmfXUkICUzIAd6tOIMvHR4GE=; b=ZOTW/FciOAgqmIwci+12OD+7r/e6GAx6qhXo02G5lbsRXFnhUO8ETJQJV0UXspYJc0 wkdheG1z0OBSfws/oAFeYLbfU5ciFSfhPhhp5s3NoNOL+j4vsHkRlPwu4+oxsZuTpyEN JToQ1Up/DSaj0p178PYPUzkAFrquvbLUkFPI4LnscaQN0v1XPeOnx6SFkmfuI74KIdwr dMFBVw+87SVeK0GY2+LyBbHs1PfIlJtukXlqHesvDQwBy7IUVZBKXBhZb5OuP2VcJJXo bzpAQ95oISMTOj6Zr6sSL2IEpGsoDdH69DARyKhkOrPkUJ5NAdLQxt2abeNZJ0wdzIqE ZphA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723040260; x=1723645060; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cd+9inN4Ie9i79QY0gLMmfXUkICUzIAd6tOIMvHR4GE=; b=Oa7nt0PISpE29hxBXRN1z37holn0OjuxOo6P/iY1UUoLTYZID909ndo59FTWOE9j9M 8TFFSTDs3S7v92PyOTjnKQ8GEZ2dEzHf78E6gqRmwYEB3iKYI9WFe/A11YnM0721+fIS TnnUlOGGibySpKxGEfYzeCrnu8VFThv9Z1L3dT1dQO+Qj9/KIh60uPyVAkfuGPTLNZMF H8XIN7c0ayA71dnSgE94xMc52WfhW1bNloJ3lOS9TlZwXyuzMMG2VkYbXa0A62l+piAA MubP6roK5uB1YokB90SZAQcAg1XH8ga4vuUBzThZylX1+NRh2nzGXXWZfc8Zp7TZpRL+ iBIA== X-Gm-Message-State: AOJu0Yxcin6qgNR6p6Ju7ENA7VuEkHvXJty3grTOndEPxx6xvK3e1t9u zw4qX/7LFKR8FSExQV9LSj6kKfnNFif6JyRSij+RUb4hEMscccljUiki8w== X-Google-Smtp-Source: AGHT+IE/b7JT3Pr91hHDIQwZT5/vXLt6lNP7yWspRnsMq+L8H/awCdRIRN8/2y73g60DtAy0GXeOOw== X-Received: by 2002:aa7:d953:0:b0:5af:b1d8:f3d5 with SMTP id 4fb4d7f45d1cf-5bba36ab44bmr1953799a12.12.1723040260082; Wed, 07 Aug 2024 07:17:40 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5b83bf3a37bsm7176928a12.88.2024.08.07.07.17.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 07:17:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH v2 2/4] io_uring/napi: postpone napi timeout adjustment Date: Wed, 7 Aug 2024 15:18:12 +0100 Message-ID: <88e14686e245b3b42ff90a3c4d70895d48676206.1723039801.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Remove io_napi_adjust_timeout() and move the adjustments out of the common path into __io_napi_busy_loop(). Now the limit it's calculated based on struct io_wait_queue::timeout, for which we query current time another time. The overhead shouldn't be a problem, it's a polling path, however that can be optimised later by additionally saving the delta time value in io_cqring_wait(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - io_uring/napi.c | 28 +++++++--------------------- io_uring/napi.h | 16 ---------------- 3 files changed, 7 insertions(+), 38 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3942db160f18..9ec07f76ad19 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2423,7 +2423,6 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, dt = timespec64_to_ktime(ts); iowq.timeout = ktime_add(dt, ktime_get()); - io_napi_adjust_timeout(ctx, &iowq, dt); } if (sig) { diff --git a/io_uring/napi.c b/io_uring/napi.c index a670f49e30ef..d39ae20a3db8 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -271,27 +271,6 @@ int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg) return 0; } -/* - * __io_napi_adjust_timeout() - adjust busy loop timeout - * @ctx: pointer to io-uring context structure - * @iowq: pointer to io wait queue - * @ts: pointer to timespec or NULL - * - * Adjust the busy loop timeout according to timespec and busy poll timeout. - * If the specified NAPI timeout is bigger than the wait timeout, then adjust - * the NAPI timeout accordingly. - */ -void __io_napi_adjust_timeout(struct io_ring_ctx *ctx, struct io_wait_queue *iowq, - ktime_t to_wait) -{ - ktime_t poll_dt = READ_ONCE(ctx->napi_busy_poll_dt); - - if (to_wait) - poll_dt = min(poll_dt, to_wait); - - iowq->napi_busy_poll_dt = poll_dt; -} - /* * __io_napi_busy_loop() - execute busy poll loop * @ctx: pointer to io-uring context structure @@ -304,6 +283,13 @@ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) if ((ctx->flags & IORING_SETUP_SQPOLL) || !ctx->napi_enabled) return; + iowq->napi_busy_poll_dt = READ_ONCE(ctx->napi_busy_poll_dt); + if (iowq->timeout != KTIME_MAX) { + ktime_t dt = ktime_sub(iowq->timeout, ktime_get()); + + iowq->napi_busy_poll_dt = min_t(u64, iowq->napi_busy_poll_dt, dt); + } + iowq->napi_prefer_busy_poll = READ_ONCE(ctx->napi_prefer_busy_poll); io_napi_blocking_busy_loop(ctx, iowq); } diff --git a/io_uring/napi.h b/io_uring/napi.h index 88f1c21d5548..87e30b4f8d9e 100644 --- a/io_uring/napi.h +++ b/io_uring/napi.h @@ -17,8 +17,6 @@ int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg); void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock); -void __io_napi_adjust_timeout(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, ktime_t to_wait); void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq); int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx); @@ -27,15 +25,6 @@ static inline bool io_napi(struct io_ring_ctx *ctx) return !list_empty(&ctx->napi_list); } -static inline void io_napi_adjust_timeout(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, - ktime_t to_wait) -{ - if (!io_napi(ctx)) - return; - __io_napi_adjust_timeout(ctx, iowq, to_wait); -} - static inline void io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { @@ -86,11 +75,6 @@ static inline bool io_napi(struct io_ring_ctx *ctx) static inline void io_napi_add(struct io_kiocb *req) { } -static inline void io_napi_adjust_timeout(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, - ktime_t to_wait) -{ -} static inline void io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { From patchwork Wed Aug 7 14:18:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13756356 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B7EE53804 for ; Wed, 7 Aug 2024 14:17:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040264; cv=none; b=XCCJHJYN4okFlT03ahkjPvFI9TsIr+dsQGsASo7eeznI5IfwdfnQ50Iz83ehKWt8bgc45nFsNkTyVJhcFJh1/rSnpe4Ekoj+xFByHgkvPGX18DT4z0ia1rqQCP7CwlHR+L4Iv70GEBMv8Lkogf1KdsSE8rF2AGCKGBzz8jLW1Fk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040264; c=relaxed/simple; bh=0KgL9mAOHnAXHqaLMBB/ti8Vpc9CRu9oziRb/ufYV64=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n24oEswiFu1SPny71Lic7GNhWfTyQdvMYJhFqwY7I/cpT0fmwWbTGM4qGqVYQNZtxKdpyPCcoeJP1q+7RE/mGvjUjeGUCuWgZli6gLXPyScQ068C5JQbSXY+ZoROxU2ZvYPQ0lnpr4mnm7pKzAMDkGcNTGK7UAph0LpCPNDKqr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Tcry4YEB; arc=none smtp.client-ip=209.85.208.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Tcry4YEB" Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-5b8c2a6135dso2695026a12.1 for ; Wed, 07 Aug 2024 07:17:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723040261; x=1723645061; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y87uv01/QSe4u5tLRXTf6I7wzSiuLnMLzrUhMtzt6X8=; b=Tcry4YEBCzb77uqc+Yxa8MhtS+Ob24qrFD1ZgmMK0aMGBmXOvxqor6zGMxQ8nmpQlN WE0LvaDSTXqKxxf55FWq5HM1idWq3F5aSBIHUWBwIu8uQ9I//sBgBh0IBkq92R7rVaI9 hEsrshVKPu1RyAXie0yHs0Qf13KzHPElqJUvYNLHvdr9l8zpJaKIRbckPGUrMEZncgYA UyOBKJ2pVH0T2Mh3QR+VzkYf55V5SlneP+hEWFLuskJ3w2SZgoKPVNydFNWMVR56h+cb iXPpNJ9E7TL2ns+svoaeKqqAH5L/YLJC6Fe8HAys0q/8KUlBWE2k7LAlu5OVU3F+fJe5 HsbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723040261; x=1723645061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y87uv01/QSe4u5tLRXTf6I7wzSiuLnMLzrUhMtzt6X8=; b=kHyTYbZ6dQBLXm0aYYKM3B5vc1654Rw8/+Jn3wbzRAifmT20ra8fmf9opnDZoAivL/ P9y1thanRgYM70/uZcKlPpwaXjYoI1eUYfjEmVf43xtLfnLIAWwYy6tc2JNqUItONJg7 cIzhCBwbPMSVc+9ZT0d8lXbIP/HfAXHItl/gVDwf9bwm/bHRqlvG9+6XhzxZDfG0vVb+ V1LACd37GZzgxxM3wxV0WNSpU853YFEfIc6z2q3MSfI5YKMcx1iI7YISyRIjpP8p66ML BH3j1eqXASQmUrpk5oFeI987Ydtj8RkS2brhymS3/ad+7Ggt0IR+EHDLq4+ZwgToUq2m +qHQ== X-Gm-Message-State: AOJu0Yw0R3fVUNGQrpd4pwu0TbDLloukcuDs3bKqCrUmaKR2/uM9GbDF obxN/cwT/M62vCueoGnq3GM4OsneVDiPp+osaEQFSWznoSCLOzVzOFo4cg== X-Google-Smtp-Source: AGHT+IGnx1wVDScEKFhg43qOXl0vJPYbyCBAlq1YU3PwnOpXjtipPiyvg/gUgYN8nKxoD7AIEB8esg== X-Received: by 2002:a05:6402:5251:b0:5bb:9ae0:4a41 with SMTP id 4fb4d7f45d1cf-5bb9ae04b7dmr3636034a12.28.1723040260533; Wed, 07 Aug 2024 07:17:40 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5b83bf3a37bsm7176928a12.88.2024.08.07.07.17.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 07:17:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH v2 3/4] io_uring: add absolute mode wait timeouts Date: Wed, 7 Aug 2024 15:18:13 +0100 Message-ID: <4d5b74d67ada882590b2e42aa3aa7117bbf6b55f.1723039801.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In addition to current relative timeouts for the waiting loop, where the timespec argument specifies the maximum time it can wait for, add support for the absolute mode, with the value carrying a CLOCK_MONOTONIC absolute time until which we should return control back to the user. Suggested-by: Lewis Baker Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 15 ++++++++------- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 2aaf7ee256ac..afc901502804 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -507,6 +507,7 @@ struct io_cqring_offsets { #define IORING_ENTER_SQ_WAIT (1U << 2) #define IORING_ENTER_EXT_ARG (1U << 3) #define IORING_ENTER_REGISTERED_RING (1U << 4) +#define IORING_ENTER_ABS_TIMER (1U << 5) /* * Passed in for io_uring_setup(2). Copied back with updated info on success diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9ec07f76ad19..5282f9887440 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2387,7 +2387,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, * Wait until events become available, if we don't already have some. The * application must reap them itself, as they reside on the shared cq ring. */ -static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, +static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, const sigset_t __user *sig, size_t sigsz, struct __kernel_timespec __user *uts) { @@ -2416,13 +2416,13 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (uts) { struct timespec64 ts; - ktime_t dt; if (get_timespec64(&ts, uts)) return -EFAULT; - dt = timespec64_to_ktime(ts); - iowq.timeout = ktime_add(dt, ktime_get()); + iowq.timeout = timespec64_to_ktime(ts); + if (!(flags & IORING_ENTER_ABS_TIMER)) + iowq.timeout = ktime_add(iowq.timeout, ktime_get()); } if (sig) { @@ -3153,7 +3153,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP | IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG | - IORING_ENTER_REGISTERED_RING))) + IORING_ENTER_REGISTERED_RING | + IORING_ENTER_ABS_TIMER))) return -EINVAL; /* @@ -3251,8 +3252,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (likely(!ret2)) { min_complete = min(min_complete, ctx->cq_entries); - ret2 = io_cqring_wait(ctx, min_complete, sig, - argsz, ts); + ret2 = io_cqring_wait(ctx, min_complete, flags, + sig, argsz, ts); } } From patchwork Wed Aug 7 14:18:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13756357 Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B4D46BFD4 for ; Wed, 7 Aug 2024 14:17:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040264; cv=none; b=cnm6rKqJecPqKGNuqcD5EKt47WiN8GpUitacpHcDl7vge4C+TBavn2uF1IcGnJv8cy0emA8B987ndIw7Gp0hitZR4p8xqG06h8SLExF3cTNG1rOGiSYPa40x8HA1Y/sgQfkxKENG0pYGOY+RKi4H4lfMhjXirQ4Mbgy7Sy17Ktg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723040264; c=relaxed/simple; bh=CofWqGuEpePArAbEME3kTKzrlGGzPO/8E/k2B9aobuo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X3DYB2cBe4uhowKVVhZ05pM4/E/odrLZadtV6vpV6CtXq9t0aCZHhJZJc2XAlv89JykdtQnf4i8X1T674nMqj7T3crfIWTDYnOD37awNTNp4N7T1QkSdyAwhccqfaXMsVmDBioo/kewoNeFy1lFa2HfNYs5IYrLyas5DxFK/9gk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OxWAZr6f; arc=none smtp.client-ip=209.85.208.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OxWAZr6f" Received: by mail-ed1-f50.google.com with SMTP id 4fb4d7f45d1cf-5a156556fb4so2169133a12.3 for ; Wed, 07 Aug 2024 07:17:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723040261; x=1723645061; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dvi2VSofIFFrnhxlbK+KYhrQHGkaB2+Ul72FbaXCv+Q=; b=OxWAZr6fUKqdMzOHp4YfK1RSvPpqMJ433Hkdt5TBIbcH/E2sqCfq2HLI7fZ99rINSa 6BpWoRJIsd5xWGobrzNkRYluAzYCavAjcl4ygOocBUG85zbKadDgkfje/nx6/L0G7NS+ u8wQRPpvbA58Iy6VrFtG4KuG4V39v8u1uGf7DrKpJ+Ij7cuG+LjTb4iFAWEA7HeL3fzY DsvlJuWxWY+X3c2vYLHybaa6WIuTP79uvoo0+ELgnoWX6XzXVL2UK4iU8MGYdq90Io0U 1Hu1ahAG+RkVzyhbPZHLknXLll3IXKZsmzU1HLjglgrLiU1o4C6VZnqRJuRVijLOze7r 46Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723040261; x=1723645061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dvi2VSofIFFrnhxlbK+KYhrQHGkaB2+Ul72FbaXCv+Q=; b=uIBPo2Zsvol8dexr127Iq8M2yKYHhkWl1NtdpUe/GAe+GKbopQzkhKkz4VdcjaEzfk VwUrywj4TacYs9h5FN22H02Fsqub7GGngWBkEyXuKWuAAHSvyS4kPF51w3kxXSZ4ulK3 5v76kcrI9iW4nnzBQOsNW4KamxAmUOPZmhOzUCjwbK6lo8MKgNX7TTBh8Bj5a5H0XL2j 5OmI0p6BiKUX7LSYmQxYQ41ZPOYPYnV5mkPYaH6wvRtfeeo4bTRyXaxQ9FsNGjOegCOQ UhVp4vzFx5uCgjw/OjHZTgQzXvxV7029EXdUthZQcZtVrDDijv2dr6HtALH0zVRSALUj L/7Q== X-Gm-Message-State: AOJu0YxND39drP9JwKGMIlLdH+Pum87cxgjqTZzKAyv8K8ZM5xzoPN6G ZSY3rayf6J+JOC84eiqfEpDNv0WpX6n1nsy7Td5vu2bSMNyaHal3GMDVHw== X-Google-Smtp-Source: AGHT+IEJPM8Kk4de3cFxvMI8EeqnAh9q43GfLYT7HFAQspVZIqMBqmLmAk4FCgnd5a7uMVTZqpe5QA== X-Received: by 2002:aa7:d309:0:b0:5bb:9ae0:4a3e with SMTP id 4fb4d7f45d1cf-5bb9ae04b08mr3056101a12.29.1723040260950; Wed, 07 Aug 2024 07:17:40 -0700 (PDT) Received: from 127.0.0.1localhost ([163.114.131.193]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5b83bf3a37bsm7176928a12.88.2024.08.07.07.17.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 07:17:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH v2 4/4] io_uring: user registered clockid for wait timeouts Date: Wed, 7 Aug 2024 15:18:14 +0100 Message-ID: <98f2bc8a3c36cdf8f0e6a275245e81e903459703.1723039801.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a new registration opcode IORING_REGISTER_CLOCK, which allows the user to select which clock id it wants to use with CQ waiting timeouts. It only allows a subset of all posix clocks and currently supports CLOCK_MONOTONIC and CLOCK_BOOTTIME. Suggested-by: Lewis Baker Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 +++ include/uapi/linux/io_uring.h | 7 +++++++ io_uring/io_uring.c | 8 ++++++-- io_uring/io_uring.h | 8 ++++++++ io_uring/napi.c | 2 +- io_uring/register.c | 31 +++++++++++++++++++++++++++++++ 6 files changed, 56 insertions(+), 3 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 3315005df117..4b9ba523978d 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -239,6 +239,9 @@ struct io_ring_ctx { struct io_rings *rings; struct percpu_ref refs; + clockid_t clockid; + enum tk_offsets clock_offset; + enum task_work_notify_mode notify_method; unsigned sq_thread_idle; } ____cacheline_aligned_in_smp; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index afc901502804..48c440edf674 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -596,6 +596,8 @@ enum io_uring_register_op { IORING_REGISTER_NAPI = 27, IORING_UNREGISTER_NAPI = 28, + IORING_REGISTER_CLOCK = 29, + /* this goes last */ IORING_REGISTER_LAST, @@ -676,6 +678,11 @@ struct io_uring_restriction { __u32 resv2[3]; }; +struct io_uring_clock_register { + __u32 clockid; + __u32 __resv[3]; +}; + struct io_uring_buf { __u64 addr; __u32 len; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5282f9887440..20229e72b65c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2377,7 +2377,8 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, ret = 0; if (iowq->timeout == KTIME_MAX) schedule(); - else if (!schedule_hrtimeout(&iowq->timeout, HRTIMER_MODE_ABS)) + else if (!schedule_hrtimeout_range_clock(&iowq->timeout, 0, + HRTIMER_MODE_ABS, ctx->clockid)) ret = -ETIME; current->in_iowait = 0; return ret; @@ -2422,7 +2423,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, iowq.timeout = timespec64_to_ktime(ts); if (!(flags & IORING_ENTER_ABS_TIMER)) - iowq.timeout = ktime_add(iowq.timeout, ktime_get()); + iowq.timeout = ktime_add(iowq.timeout, io_get_time(ctx)); } if (sig) { @@ -3424,6 +3425,9 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, if (!ctx) return -ENOMEM; + ctx->clockid = CLOCK_MONOTONIC; + ctx->clock_offset = 0; + if ((ctx->flags & IORING_SETUP_DEFER_TASKRUN) && !(ctx->flags & IORING_SETUP_IOPOLL) && !(ctx->flags & IORING_SETUP_SQPOLL)) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index c2acf6180845..9935819f12b7 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -437,6 +437,14 @@ static inline bool io_file_can_poll(struct io_kiocb *req) return false; } +static inline ktime_t io_get_time(struct io_ring_ctx *ctx) +{ + if (ctx->clockid == CLOCK_MONOTONIC) + return ktime_get(); + + return ktime_get_with_offset(ctx->clock_offset); +} + enum { IO_CHECK_CQ_OVERFLOW_BIT, IO_CHECK_CQ_DROPPED_BIT, diff --git a/io_uring/napi.c b/io_uring/napi.c index d39ae20a3db8..1c3e7562e90b 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -285,7 +285,7 @@ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) iowq->napi_busy_poll_dt = READ_ONCE(ctx->napi_busy_poll_dt); if (iowq->timeout != KTIME_MAX) { - ktime_t dt = ktime_sub(iowq->timeout, ktime_get()); + ktime_t dt = ktime_sub(iowq->timeout, io_get_time(ctx)); iowq->napi_busy_poll_dt = min_t(u64, iowq->napi_busy_poll_dt, dt); } diff --git a/io_uring/register.c b/io_uring/register.c index e3c20be5a198..57cb85c42526 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -335,6 +335,31 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, return ret; } +static int io_register_clock(struct io_ring_ctx *ctx, + struct io_uring_clock_register __user *arg) +{ + struct io_uring_clock_register reg; + + if (copy_from_user(®, arg, sizeof(reg))) + return -EFAULT; + if (memchr_inv(®.__resv, 0, sizeof(reg.__resv))) + return -EINVAL; + + switch (reg.clockid) { + case CLOCK_MONOTONIC: + ctx->clock_offset = 0; + break; + case CLOCK_BOOTTIME: + ctx->clock_offset = TK_OFFS_BOOT; + break; + default: + return -EINVAL; + } + + ctx->clockid = reg.clockid; + return 0; +} + static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -511,6 +536,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_unregister_napi(ctx, arg); break; + case IORING_REGISTER_CLOCK: + ret = -EINVAL; + if (!arg || nr_args) + break; + ret = io_register_clock(ctx, arg); + break; default: ret = -EINVAL; break;