From patchwork Fri Nov 6 23:16:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888365 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F71B921 for ; Fri, 6 Nov 2020 23:18:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 792A42087E for ; Fri, 6 Nov 2020 23:18:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ER8g397G" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728302AbgKFXRy (ORCPT ); Fri, 6 Nov 2020 18:17:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728198AbgKFXRu (ORCPT ); Fri, 6 Nov 2020 18:17:50 -0500 Received: from mail-qv1-xf43.google.com (mail-qv1-xf43.google.com [IPv6:2607:f8b0:4864:20::f43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59FF0C0613CF; Fri, 6 Nov 2020 15:17:49 -0800 (PST) Received: by mail-qv1-xf43.google.com with SMTP id ed14so1258425qvb.4; Fri, 06 Nov 2020 15:17:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7NtkPSt3HryQm3XghUScn8RP5ZLrHE5f6ni874ipj/g=; b=ER8g397GJJr4epHAMeA12ADqf3nTgy9rwlRAE/caCZKTw0SBDh1Eqrt6wYXi4JFPDU /8tpn5V0aGV9spZIvXMrqofeNCsIk0RcpfTrfrDXZybqVM36rDhmczRO69Z7BLxbYtXu 6SbFq8Dm3fE8ozpb291a6Y0M66/N3srAvUdt+uEGtCkWsq4cXveZ5z6DzvP+IrxvRMDf Ax1BfWqG9sW1fsnZ9hKGzzIUzD3d+mhvUR6tiQTLWsez5rHpfm4w30tCWtqc2ypQHgf8 qY/6YILP/BdjzOp/XZ1KP/Vzym0IadykOMhHfm9IkO3gV6GXDuX85lt5v8ZcqsrNtI3U Xl4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7NtkPSt3HryQm3XghUScn8RP5ZLrHE5f6ni874ipj/g=; b=b5DEoHayoQW5iAD9KkPMaFweThVq7Y0886LrhCAg1vjztyBCbAI/7KjxWc8Gnx6v7f 5Kmjbh6diXhIpbnOwFk05d64ht+pYwcMgprh8kpRiWt1yqeKfc7fUDoGUHm3//5sbb/G uHzzH963GUsf05IG/RAQ0s7ajp5H5c6eQmPTu8o4e5zdgLC1ZB28BWFqzRztxADzVyyy yTlY+zeQDuPFkmVAZ6oszfd+2RekZxiPRyD+DGhcTZY2K0FyltcwWKJMRnasxaRB7sjs 435+7uzRvqVrSKsE1eBiEIJauqGOO+9umF11Yz2VAYSHcjPQzlpQID+FBNv94JTSb1I6 zlpQ== X-Gm-Message-State: AOAM533Fcb4DUTPmF79DVyH08al/i27BWaE4a9RynVpkwSboPjQB75sv 7wUOM7nVqzo9BS1O4UfG1xk= X-Google-Smtp-Source: ABdhPJy0F9moY1w/bwCG/g2cZkfdbYV9+C38Y70i8gwnBDiWqROQAC3r9xRoW89JsIFh+V2aAZ9Y8g== X-Received: by 2002:a0c:ecc8:: with SMTP id o8mr3847589qvq.19.1604704668610; Fri, 06 Nov 2020 15:17:48 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:48 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 1/8] epoll: check for events when removing a timed out thread from the wait queue Date: Fri, 6 Nov 2020 18:16:28 -0500 Message-Id: <20201106231635.3528496-2-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh After abc610e01c66 ("fs/epoll: avoid barrier after an epoll_wait(2) timeout"), we break out of the ep_poll loop upon timeout, without checking whether there is any new events available. Prior to that patch-series we always called ep_events_available() after exiting the loop. This can cause races and missed wakeups. For example, consider the following scenario reported by Guantao Liu: Suppose we have an eventfd added using EPOLLET to an epollfd. Thread 1: Sleeps for just below 5ms and then writes to an eventfd. Thread 2: Calls epoll_wait with a timeout of 5 ms. If it sees an event of the eventfd, it will write back on that fd. Thread 3: Calls epoll_wait with a negative timeout. Prior to abc610e01c66, it is guaranteed that Thread 3 will wake up either by Thread 1 or Thread 2. After abc610e01c66, Thread 3 can be blocked indefinitely if Thread 2 sees a timeout right before the write to the eventfd by Thread 1. Thread 2 will be woken up from schedule_hrtimeout_range and, with evail 0, it will not call ep_send_events(). To fix this issue: 1) Simplify the timed_out case as suggested by Linus. 2) while holding the lock, recheck whether the thread was woken up after its time out has reached. Note that (2) is different from Linus' original suggestion: It do not set "eavail = ep_events_available(ep)" to avoid unnecessary contention (when there are too many timed-out threads and a small number of events), as well as races mentioned in the discussion thread. This is the first patch in the series so that the backport to stable releases is straightforward. Link: https://lkml.kernel.org/r/CAHk-=wizk=OxUyQPbO8MS41w2Pag1kniUV5WdD5qWL-gq1kjDA@mail.gmail.com Fixes: abc610e01c66 ("fs/epoll: avoid barrier after an epoll_wait(2) timeout") Tested-by: Guantao Liu Suggested-by: Linus Torvalds Signed-off-by: Soheil Hassas Yeganeh Reported-by: Guantao Liu Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov Reviewed-by: Davidlohr Bueso --- fs/eventpoll.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 4df61129566d..117b1c395ae4 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1902,23 +1902,30 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, } write_unlock_irq(&ep->lock); - if (eavail || res) - break; + if (!eavail && !res) + timed_out = !schedule_hrtimeout_range(to, slack, + HRTIMER_MODE_ABS); - if (!schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS)) { - timed_out = 1; - break; - } - - /* We were woken up, thus go and try to harvest some events */ + /* + * We were woken up, thus go and try to harvest some events. + * If timed out and still on the wait queue, recheck eavail + * carefully under lock, below. + */ eavail = 1; - } while (0); __set_current_state(TASK_RUNNING); if (!list_empty_careful(&wait.entry)) { write_lock_irq(&ep->lock); + /* + * If the thread timed out and is not on the wait queue, it + * means that the thread was woken up after its timeout expired + * before it could reacquire the lock. Thus, when wait.entry is + * empty, it needs to harvest events. + */ + if (timed_out) + eavail = list_empty(&wait.entry); __remove_wait_queue(&ep->wq, &wait); write_unlock_irq(&ep->lock); } From patchwork Fri Nov 6 23:16:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888369 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F85314B4 for ; Fri, 6 Nov 2020 23:18:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 518F72087E for ; Fri, 6 Nov 2020 23:18:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="At3nFnTI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728852AbgKFXSV (ORCPT ); Fri, 6 Nov 2020 18:18:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728217AbgKFXRw (ORCPT ); Fri, 6 Nov 2020 18:17:52 -0500 Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05EC8C0613D2; Fri, 6 Nov 2020 15:17:50 -0800 (PST) Received: by mail-qt1-x842.google.com with SMTP id j31so2005363qtb.8; Fri, 06 Nov 2020 15:17:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q5hM/1XH5XvHZuZiSwPhICBUd5JhPMgu+Z0ZMA4flOI=; b=At3nFnTIYsXNwPEkTl3meLeIk581q8AIkEPUc3D8f/LK+btAgbSnTFkVG0BGLMV5Oa sLhFJC86XyEopJ9sk0+kIPCgfIljTma6CRrNxg08fhQ1dX1Mv+rgheM2z426MTYmq+fw LiLboyqc6V0FYLHe8XbZQdyQ/MOaxwlarqKzmyv+NAB7E8+AzFJu6g5gYdjfY7ZWG3YH DPzhl7t0BNaA3P1yyfm9HLYXnpncl/T/70sDqHP92B0BBE59EvnayYKROrYKOpMXpE8I nM7BMevq+dGObcQVBwlMxo6WaB75PZwmcPnOIWcKpu6oJR/+GcRJGftFQub2CrRsWiVW wy9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q5hM/1XH5XvHZuZiSwPhICBUd5JhPMgu+Z0ZMA4flOI=; b=m/cCLPDOL4cngLIiTIdqzo3FAJ6thzN7kYQ7Pi/drLtaEZjOxTNlV4fp9f6UDSn4E7 /e4o2J1pf/P7XPSVz8ld/wSq3YmgIEOTxCtrPJzyLY5raaMAZ2hGszHEH58S6lc1xcwD WTr0+FEMeut29i8F8crfkkL4Wddt5gMnvdGo2n/ON+BmCwHuF4ODa1YZw3DC9NzoUrVJ AtxEkcS4guTyjg1oIrWeNs+FwSEIcWBQn+3P/hTGWAAbZQt1ApKGh6dMcA9i65IPb77E KllZziwMGw3SOLO0echVcxzfKSUXJAuWfwOnkg8YmWE4Y8boruBHaT1gJz4BfGi5jt49 kxXg== X-Gm-Message-State: AOAM533YdQCUj3A/tYbTxBbbciFKhWaTfg+SO+SioTtOIr4YZrM89mjm IMp0eg8eeCg7tN9AZiwOslg= X-Google-Smtp-Source: ABdhPJxb2W1+iapjcMgS4qk+lBO+LlHf6WeoaT80foR/JXxYT4J+FDsKKkukhUUPCPlU265tgrHRFQ== X-Received: by 2002:ac8:5191:: with SMTP id c17mr4018077qtn.116.1604704669322; Fri, 06 Nov 2020 15:17:49 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:49 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 2/8] epoll: simplify signal handling Date: Fri, 6 Nov 2020 18:16:29 -0500 Message-Id: <20201106231635.3528496-3-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh Check signals before locking ep->lock, and immediately return -EINTR if there is any signal pending. This saves a few loads, stores, and branches from the hot path and simplifies the loop structure for follow up patches. Suggested-by: Linus Torvalds Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 117b1c395ae4..80c560dad6a3 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1818,7 +1818,7 @@ static inline struct timespec64 ep_set_mstimeout(long ms) static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, long timeout) { - int res = 0, eavail, timed_out = 0; + int res, eavail, timed_out = 0; u64 slack = 0; wait_queue_entry_t wait; ktime_t expires, *to = NULL; @@ -1865,6 +1865,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, ep_reset_busy_poll_napi_id(ep); do { + if (signal_pending(current)) + return -EINTR; + /* * Internally init_wait() uses autoremove_wake_function(), * thus wait entry is removed from the wait queue on each @@ -1894,15 +1897,12 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * important. */ eavail = ep_events_available(ep); - if (!eavail) { - if (signal_pending(current)) - res = -EINTR; - else - __add_wait_queue_exclusive(&ep->wq, &wait); - } + if (!eavail) + __add_wait_queue_exclusive(&ep->wq, &wait); + write_unlock_irq(&ep->lock); - if (!eavail && !res) + if (!eavail) timed_out = !schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS); @@ -1938,14 +1938,14 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * finding more events available and fetching * repeatedly. */ - res = -EINTR; + return -EINTR; } /* * Try to transfer events to user space. In case we get 0 events and * there's still timeout left over, we go trying again in search of * more luck. */ - if (!res && eavail && + if (eavail && !(res = ep_send_events(ep, events, maxevents)) && !timed_out) goto fetch_events; From patchwork Fri Nov 6 23:16:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888355 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9950921 for ; Fri, 6 Nov 2020 23:17:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 84EF820882 for ; Fri, 6 Nov 2020 23:17:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AznM+dy4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728358AbgKFXRy (ORCPT ); Fri, 6 Nov 2020 18:17:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728203AbgKFXRv (ORCPT ); Fri, 6 Nov 2020 18:17:51 -0500 Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15E7BC0613D3; Fri, 6 Nov 2020 15:17:51 -0800 (PST) Received: by mail-qt1-x842.google.com with SMTP id r8so2007875qtp.13; Fri, 06 Nov 2020 15:17:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E1ilj57B28AycSlcMqSc+BHzcumL+p/zmD4cBPaCkao=; b=AznM+dy4BJNFfqz+PAlzqHHpO6Wuq5mCEvnk6y2QNqYhv6KZtW5ux61uaJcKbJ64lH P0M43uFsyircu/AZa+NWDZk6GVSDnKhqYlq1wi6ok+khhMV5aHKuU3ksu3iA/cNZLUep hY0EilBGBgNOxWJaiYCiLa6kYCEXSFCY2HnA6SUkn0ZSF0+BYTxNJ85EuCqZLh4godo6 1zFjgSQV++PsS1WjdZnxGSmQEhhPE+L4tOApAJKGl3FkckIOPDNwUFdGsjkBibScFLkh NwoOTGei4+5dJc/qyAcGtfMVx0lqwPjX8LJAwoVKeJVWmAmD+VESzRMqi1p8/zOxhc5U lc/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E1ilj57B28AycSlcMqSc+BHzcumL+p/zmD4cBPaCkao=; b=H2m8ZbnRuLctiAXYojP1MfzCFPL+ojjZ0cPlvi5qIxMWDpW4P+uMQOjeywQTewFJLV cYMrDhCxhzb9Y0E4sf0+/FV8+T4RPCsWp9/ZS2M2tjW0/Nl0OGDaKJbs4CXktzgz0unc YniYxFDLCnyVTdl2I83RRL+7fwConx+WhsZC/Ohy2699jPL4ObSyF/sXx7R5zTcggVfR QBN+JWZYAcqSNzgHy5KNixR1kkjjGfGxsc4ADsqS+q0r00a2KQRepKiz7Ew7+MecOk3l P6FrXrY59QBWkW1DwWDLDfpKx+uFBEZVdDf7h+lO4MyAkyegOtE7qsq2I1K8djzAcHRS 2/3Q== X-Gm-Message-State: AOAM531/mQE9/T86LpofCToJn1sEb+z3ZWI8dzu2K94dgU+0gALmsiWO hQ6dyvt28z6Nxb2hnobe2ec= X-Google-Smtp-Source: ABdhPJyapMMEKClAaA5UuC0rq9zDRf3BtSz4Q0EAxoAXnPc29hofhm48hNyFBKAp6jr2cxHwglPDeA== X-Received: by 2002:ac8:5816:: with SMTP id g22mr3697520qtg.316.1604704670377; Fri, 06 Nov 2020 15:17:50 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:49 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 3/8] epoll: pull fatal signal checks into ep_send_events() Date: Fri, 6 Nov 2020 18:16:30 -0500 Message-Id: <20201106231635.3528496-4-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh To simplify the code, pull in checking the fatal signals into ep_send_events(). ep_send_events() is called only from ep_poll(). Note that, previously, we were always checking fatal events, but it is checked only if eavail is true. This should be fine because the goal of that check is to quickly return from epoll_wait() when there is a pending fatal signal. Suggested-by: Willem de Bruijn Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 80c560dad6a3..ed9deeab2488 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1780,6 +1780,14 @@ static int ep_send_events(struct eventpoll *ep, { struct ep_send_events_data esed; + /* + * Always short-circuit for fatal signals to allow threads to make a + * timely exit without the chance of finding more events available and + * fetching repeatedly. + */ + if (fatal_signal_pending(current)) + return -EINTR; + esed.maxevents = maxevents; esed.events = events; @@ -1931,15 +1939,6 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, } send_events: - if (fatal_signal_pending(current)) { - /* - * Always short-circuit for fatal signals to allow - * threads to make a timely exit without the chance of - * finding more events available and fetching - * repeatedly. - */ - return -EINTR; - } /* * Try to transfer events to user space. In case we get 0 events and * there's still timeout left over, we go trying again in search of From patchwork Fri Nov 6 23:16:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888367 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A21BF921 for ; Fri, 6 Nov 2020 23:18:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7AF2F2087E for ; Fri, 6 Nov 2020 23:18:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AIIXVyDh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728908AbgKFXSV (ORCPT ); Fri, 6 Nov 2020 18:18:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728215AbgKFXRv (ORCPT ); Fri, 6 Nov 2020 18:17:51 -0500 Received: from mail-qv1-xf42.google.com (mail-qv1-xf42.google.com [IPv6:2607:f8b0:4864:20::f42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9B7DC0613D4; Fri, 6 Nov 2020 15:17:51 -0800 (PST) Received: by mail-qv1-xf42.google.com with SMTP id g13so1255385qvu.1; Fri, 06 Nov 2020 15:17:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7k6fYQ9K5vRm3PXjQcfxqUpud2NB02Cs/V2CtqqpG6M=; b=AIIXVyDhthOlEuqc7en/uBEzaqrbD7+TQK4buBybf+OyfThOBXyRJqp8H/lk1vrqWe n3xSzNGuMpQcDH2Nk3Gh9YZn1v+SrGcy3Qxdu6uvf/yqPHwYOJaXztKzyNeMMv8HlxwA hg8tmeYIj4Kpo3jaA8UF0sJO99O5zKyf7HhzedIFcZwvx6uZSHfyMKq35jDjLXO6P/tN xc5qJa1D4AMd48phAZYm9yPSw5lq4UogTFg6aFGTAme846WVYyRd7ywPHQaUjTX0biYu xzPxL9G3tIYl9BbMZwPX4tWO0Bmhr4aVnngPZo7c9Vdpp+0bxbnqYg1ahWaJUmAGVAyG FRxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7k6fYQ9K5vRm3PXjQcfxqUpud2NB02Cs/V2CtqqpG6M=; b=QtpxzzBHd3NiRgO6qcqdjm1r+fbiX84q6Uk8D/G8Wt2Y1Y4veO2gmgi3db2KXaX0+K gzEAPIYZCV1YC7kos+x70o8cMGAKLs9vDslNvyB+bjCZz/AnHvPx9IjG5NKy71+S1LyC K52lcs6L9xJGZLTraFXVtunsdmCipfM07Vr4zwLuBaUfYi9Iz0bFrcL2Rb674Td2zUq/ ntGk5TnclrYvM/2HLzS4fZIxPgV0Vv9akTnaLRlc8lG/qnyN+U2SbmHcldOeQyT/c4rb ucr6hQ1wGbrubp/cbOb6rcdLLwNx7U3kJlxO37rR7HQxrROBB02s39x+L7LIK2lgxnju QZVg== X-Gm-Message-State: AOAM53150LvDZeh+WmuN7WIxAjXAxPWMyAZ/zFv4NECn9CqFOoAFLACK IOiX3ldhP8PzdhspeZpkqU8= X-Google-Smtp-Source: ABdhPJyGTIkUWAGtRmr2zgTP+0Zcs2Ik4PTw6YFjewy5dY/Iifjj7lFMowd+sVi3OhEVYdw6/8BReg== X-Received: by 2002:a05:6214:c2:: with SMTP id f2mr3909399qvs.2.1604704670992; Fri, 06 Nov 2020 15:17:50 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:50 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 4/8] epoll: move eavail next to the list_empty_careful check Date: Fri, 6 Nov 2020 18:16:31 -0500 Message-Id: <20201106231635.3528496-5-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh This is a no-op change and simply to make the code more coherent. Suggested-by: Linus Torvalds Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ed9deeab2488..5226b0cb1098 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1913,6 +1913,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (!eavail) timed_out = !schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS); + __set_current_state(TASK_RUNNING); /* * We were woken up, thus go and try to harvest some events. @@ -1922,8 +1923,6 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, eavail = 1; } while (0); - __set_current_state(TASK_RUNNING); - if (!list_empty_careful(&wait.entry)) { write_lock_irq(&ep->lock); /* From patchwork Fri Nov 6 23:16:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888361 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D0C714B4 for ; Fri, 6 Nov 2020 23:18:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 584E0206A5 for ; Fri, 6 Nov 2020 23:18:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JgqfE4j2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728818AbgKFXSK (ORCPT ); Fri, 6 Nov 2020 18:18:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728245AbgKFXRy (ORCPT ); Fri, 6 Nov 2020 18:17:54 -0500 Received: from mail-qt1-x843.google.com (mail-qt1-x843.google.com [IPv6:2607:f8b0:4864:20::843]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51AD1C0613D6; Fri, 6 Nov 2020 15:17:52 -0800 (PST) Received: by mail-qt1-x843.google.com with SMTP id j31so2005413qtb.8; Fri, 06 Nov 2020 15:17:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nOSFREc0POUXsBcyqAN5If5e3Op5s8v2XbVHIhjOP4Y=; b=JgqfE4j2JpxJLmj0zMJ1yzwbdykmApIOch3Xx3jw4p1oJ8+MyRvrmm4GuDvkqsO7lt 7VmVyRlv8D/K/cIE1E4+sU3RTK3pYrLv7dc9IKSine5WpCXlLLVQL6LXxRSyFGk9kQxP SqfXzNUUuYJlPcF5qHqiUv48rq5vE2xph+EJN721IEfZj3jyYdiw3LX1qHu7z3FyaMAP FMcll8IZnZiIbYnj3nedzbXPY8K79d8nfWwntOgG9yFOTl1lejMS2B6uOSvaksCCUR/c cURrpjB8Pps0U3WCwAcB7BoVbU/qND6Qy6PL5fNrMpNRlGtPGfzSGBjRaHzb1TlK331b NPLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nOSFREc0POUXsBcyqAN5If5e3Op5s8v2XbVHIhjOP4Y=; b=J+gRGpghva7rHW2iGZY5M+wnL1EkujbG+073yHLLvBkpRi1D60OCZcRd66pQdt6LqG xi5bKPLlKnf6S7n5tjk3p0ikqkeTMhaIjui7Rm3vDU1XqrMxxEE/tGuqGHgoG/DfQdAx KVDBSMeoZFzgr7uyKX0q90TPz4g9x3iUf38co2tTCY22NFzTGj/2pDAOpH/cU7SW7MnP 6vmvBnzmzm450/vv1A9kdNsYazzRMp+ulzGM5eZsf4P0wW6WGTMeUbgxR+nGDLPBRUyu dlU/4CLcTOWWyqLvakgSxEFP0BcfoOeptjTJvsJfjQHgucBmEhVMRnMntMn/fEhrdnxV PnVQ== X-Gm-Message-State: AOAM532V+o0U3pjtoAvZqThj88onuWOqprk1U4tfwyp9VlWYVGEu1BSY RZ7mPdnHvQJNhvvS6HpScN0= X-Google-Smtp-Source: ABdhPJzxGx8FL5Mjc33i9OrWQxJ0rNlUEGUzIEfMSVyZooARvf49FSMcVCJsrDzETAN5A7A+JM4oYg== X-Received: by 2002:aed:31c5:: with SMTP id 63mr3768102qth.84.1604704671575; Fri, 06 Nov 2020 15:17:51 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:51 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 5/8] epoll: simplify and optimize busy loop logic Date: Fri, 6 Nov 2020 18:16:32 -0500 Message-Id: <20201106231635.3528496-6-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh ep_events_available() is called multiple times around the busy loop logic, even though the logic is generally not used. ep_reset_busy_poll_napi_id() is similarly always called, even when busy loop is not used. Eliminate ep_reset_busy_poll_napi_id() and inline it inside ep_busy_loop(). Make ep_busy_loop() return whether there are any events available after the busy loop. This will eliminate unnecessary loads and branches, and simplifies the loop. Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 42 +++++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 23 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 5226b0cb1098..28c1d341d2e6 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -391,19 +391,26 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) * busy loop will return if need_resched or ep_events_available. * * we must do our busy polling with irqs enabled + * + * Returns whether new events are available after a successful busy loop. */ -static void ep_busy_loop(struct eventpoll *ep, int nonblock) +static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); - if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) + if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep); -} - -static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep) -{ - if (ep->napi_id) + if (ep_events_available(ep)) + return true; + /* + * Busy poll timed out. Drop NAPI ID for now, we can add + * it back in when we have moved a socket with a valid NAPI + * ID onto the ready list. + */ ep->napi_id = 0; + return false; + } + return false; } /* @@ -444,12 +451,9 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) #else -static inline void ep_busy_loop(struct eventpoll *ep, int nonblock) -{ -} - -static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep) +static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) { + return false; } static inline void ep_set_busy_poll_napi_id(struct epitem *epi) @@ -1857,21 +1861,13 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, } fetch_events: - - if (!ep_events_available(ep)) - ep_busy_loop(ep, timed_out); - eavail = ep_events_available(ep); + if (!eavail) + eavail = ep_busy_loop(ep, timed_out); + if (eavail) goto send_events; - /* - * Busy poll timed out. Drop NAPI ID for now, we can add - * it back in when we have moved a socket with a valid NAPI - * ID onto the ready list. - */ - ep_reset_busy_poll_napi_id(ep); - do { if (signal_pending(current)) return -EINTR; From patchwork Fri Nov 6 23:16:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888363 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B8C814B4 for ; Fri, 6 Nov 2020 23:18:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0C79120867 for ; Fri, 6 Nov 2020 23:18:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kE9eYc8O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728808AbgKFXSK (ORCPT ); Fri, 6 Nov 2020 18:18:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728268AbgKFXRy (ORCPT ); Fri, 6 Nov 2020 18:17:54 -0500 Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DF70C0617A7; Fri, 6 Nov 2020 15:17:53 -0800 (PST) Received: by mail-qt1-x844.google.com with SMTP id j31so2005439qtb.8; Fri, 06 Nov 2020 15:17:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+zMkTSpothskT4xefhKBsGTGEZ2rYTh/ZE5qcHm9T8k=; b=kE9eYc8O86zWBxgPy5SobzbDGX5il6e8ZYHqiuzBwPkR+VE8JyYrAohIH/GUAj9/Mg Fa7+hMvHDOIWk+br7/JmFHW27RnnBwCjW+9dZmVdsPndUMT7eiPwAr9T6aEl3sjJLFuw D6873brIDGvJ0XuWpDMiK4LjFGj/c/BgQ+22FPTLn/pS4kJvHqNTSgN7X7oG9WBHVeGf st7B7LFlEZ7lwIPo2ERLOweh06C6vIIIPVuuJkyjGTksz1iTF1L8j0JeWsNDq4pKRdEY xZJ22WW4Ql+RtKhslcKGFhBpW3li9m34VA3x1/bu/VLUyReCtc3Ul2ty8p0LgKRuQ/y/ sGRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+zMkTSpothskT4xefhKBsGTGEZ2rYTh/ZE5qcHm9T8k=; b=AHZ3czv7mOjemDpqISrvy7NeNrZaqI3GCL7WbEhKkBwAip618KfugDjBDsZH8AuDZC lrqtAQ1Xo2arqs+fPVEslnxe82xGUkdPjqLo/Q300flQxDrImXbeLqFjaBWYT1aIdjeS kv1129mqpcG8ppa3iElcgTxuGTNrUaCaoR8c4MyG+K9gSw1BsA0u95EKl4coI5qQ9Axe X6r2xYm1xI7IDXPvGFiGaY2LDS43gLtPlIiebXfkvitLUoTYWPHcYMx1WnaAFOGoL2SJ 97krR8mSwcPLO6jrk66juh+Wu4aFatpmPI7wIxWlwudRANFuUPH35gooxSTDu40hKQ8P bQ5Q== X-Gm-Message-State: AOAM532N07GEDmGGq+DWe2ewLGHrxRgQS9uPVnQuWh2z3bf/nwhUQRxP ZhvbGX2OoHp8KzE/Cg2cemc= X-Google-Smtp-Source: ABdhPJzMIvL1n5gCX0+jwWGXiX3sFbv6kcvcqQmI+eygJb8IC6vRVrH4oYSHW210phUXbAp/pJ46KA== X-Received: by 2002:ac8:43ca:: with SMTP id w10mr3145639qtn.154.1604704672495; Fri, 06 Nov 2020 15:17:52 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:51 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 6/8] epoll: pull all code between fetch_events and send_event into the loop Date: Fri, 6 Nov 2020 18:16:33 -0500 Message-Id: <20201106231635.3528496-7-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh This is a no-op change which simplifies the follow up patches. Suggested-by: Linus Torvalds Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 28c1d341d2e6..b29bbebe8ca4 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1861,14 +1861,14 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, } fetch_events: - eavail = ep_events_available(ep); - if (!eavail) - eavail = ep_busy_loop(ep, timed_out); + do { + eavail = ep_events_available(ep); + if (!eavail) + eavail = ep_busy_loop(ep, timed_out); - if (eavail) - goto send_events; + if (eavail) + goto send_events; - do { if (signal_pending(current)) return -EINTR; @@ -1917,21 +1917,22 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * carefully under lock, below. */ eavail = 1; - } while (0); - if (!list_empty_careful(&wait.entry)) { - write_lock_irq(&ep->lock); - /* - * If the thread timed out and is not on the wait queue, it - * means that the thread was woken up after its timeout expired - * before it could reacquire the lock. Thus, when wait.entry is - * empty, it needs to harvest events. - */ - if (timed_out) - eavail = list_empty(&wait.entry); - __remove_wait_queue(&ep->wq, &wait); - write_unlock_irq(&ep->lock); - } + if (!list_empty_careful(&wait.entry)) { + write_lock_irq(&ep->lock); + /* + * If the thread timed out and is not on the wait queue, + * it means that the thread was woken up after its + * timeout expired before it could reacquire the lock. + * Thus, when wait.entry is empty, it needs to harvest + * events. + */ + if (timed_out) + eavail = list_empty(&wait.entry); + __remove_wait_queue(&ep->wq, &wait); + write_unlock_irq(&ep->lock); + } + } while (0); send_events: /* From patchwork Fri Nov 6 23:16:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888357 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 77EDA921 for ; Fri, 6 Nov 2020 23:18:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54CFE206A5 for ; Fri, 6 Nov 2020 23:18:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U9PzMtZ8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728663AbgKFXR5 (ORCPT ); Fri, 6 Nov 2020 18:17:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728518AbgKFXRz (ORCPT ); Fri, 6 Nov 2020 18:17:55 -0500 Received: from mail-qt1-x843.google.com (mail-qt1-x843.google.com [IPv6:2607:f8b0:4864:20::843]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC83AC061A4A; Fri, 6 Nov 2020 15:17:53 -0800 (PST) Received: by mail-qt1-x843.google.com with SMTP id t5so2025844qtp.2; Fri, 06 Nov 2020 15:17:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fOTmvxvmpQbjOs7fCzqpFaGlP1cxXiVyhe60vEcddRs=; b=U9PzMtZ8qMgubA4tawTN8osLPaaIBfSTcJDcjCPe8/AHrxeYUHRtvP913xVhr/mdkh u3O1C4f/y8ITPpCzkydmH27VkcudGTY9dgf/38QskeIt/Oq69/Z4iXZpEUTJHIBGZqEa NrXXsRAoVHNEp5mZAQkiNMpSuIDp7aeR2TapMI46qff9UKJpeeHUb/l5eQ66cP3HvBpl YIG1Sfn8xJ5Pbs7cCY+uZEaBYTdFBrYp0WbSNWUTa0xNXmRq8GGofOXG+T7tHgcHCFqC mwNL5rAMWTG8MfJ2EE1SBYLQnE5igXqtPJeRrALmeAYX+DnBD7h0qOvcxdXnybQbsZ0R VcrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fOTmvxvmpQbjOs7fCzqpFaGlP1cxXiVyhe60vEcddRs=; b=YynKWg4be3aYUiAYtXx/47mPzqK6J+4miBcSLsRBEA3ZlwOzi/DNSU5RC852zD1eyf 3wJKbTQdnpX/MC+HWNA9OJAWndRh9Z1UTwuxIGBQQp7Ar2jJcwmglKBwJjgLRh3aB43P MuWdCLDQ/CHjR3TDmrWR5VDYnOIJX2mLqlclF/CMm55LhmPrj3ZJwrB9/U8qeCh+Br4v RCI7Oa7mhUN7Eo2UyBd8Lezktp78aAXr67h2I/15wOnfBMr48Iu7/EZ3ZOaj36J9Fj94 ebhhK6UbOFjsWXkv/S3nzrd9OTB3W9u7SwNn39xT1g2RZj4dg4B4/ZdOax7Nr8jHYW7G yoPQ== X-Gm-Message-State: AOAM532NmE2CF0HDP8LHcWIH7da/+FKp0rdLCfKpZ7N3l2PGk7xxF+l0 AO0FPG8xJGHxDURG8V1+wHk= X-Google-Smtp-Source: ABdhPJzEbkgRrEBgDJJc5LVTbXoKCH1aZnuw6tZva49R1/sbWsaowtlXwCjo5oIM0OO0AuW9Vum1uw== X-Received: by 2002:ac8:4897:: with SMTP id i23mr3687277qtq.211.1604704673124; Fri, 06 Nov 2020 15:17:53 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:52 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 7/8] epoll: replace gotos with a proper loop Date: Fri, 6 Nov 2020 18:16:34 -0500 Message-Id: <20201106231635.3528496-8-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh The existing loop is pointless, and the labels make it really hard to follow the structure. Replace that control structure with a simple loop that returns when there are new events, there is a signal, or the thread has timed out. Suggested-by: Linus Torvalds Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index b29bbebe8ca4..f4e1be7ada26 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1830,7 +1830,7 @@ static inline struct timespec64 ep_set_mstimeout(long ms) static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, long timeout) { - int res, eavail, timed_out = 0; + int res, eavail = 0, timed_out = 0; u64 slack = 0; wait_queue_entry_t wait; ktime_t expires, *to = NULL; @@ -1856,18 +1856,30 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, write_lock_irq(&ep->lock); eavail = ep_events_available(ep); write_unlock_irq(&ep->lock); - - goto send_events; } -fetch_events: - do { + while (1) { + if (eavail) { + /* + * Try to transfer events to user space. In case we get + * 0 events and there's still timeout left over, we go + * trying again in search of more luck. + */ + res = ep_send_events(ep, events, maxevents); + if (res) + return res; + } + + if (timed_out) + return 0; + eavail = ep_events_available(ep); - if (!eavail) - eavail = ep_busy_loop(ep, timed_out); + if (eavail) + continue; + eavail = ep_busy_loop(ep, timed_out); if (eavail) - goto send_events; + continue; if (signal_pending(current)) return -EINTR; @@ -1932,19 +1944,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, __remove_wait_queue(&ep->wq, &wait); write_unlock_irq(&ep->lock); } - } while (0); - -send_events: - /* - * Try to transfer events to user space. In case we get 0 events and - * there's still timeout left over, we go trying again in search of - * more luck. - */ - if (eavail && - !(res = ep_send_events(ep, events, maxevents)) && !timed_out) - goto fetch_events; - - return res; + } } /** From patchwork Fri Nov 6 23:16:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Soheil Hassas Yeganeh X-Patchwork-Id: 11888359 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 90A2E921 for ; Fri, 6 Nov 2020 23:18:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6A1BA20867 for ; Fri, 6 Nov 2020 23:18:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UexX2EnJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728651AbgKFXR5 (ORCPT ); Fri, 6 Nov 2020 18:17:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728203AbgKFXR4 (ORCPT ); Fri, 6 Nov 2020 18:17:56 -0500 Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C08F4C0613CF; Fri, 6 Nov 2020 15:17:54 -0800 (PST) Received: by mail-qt1-x844.google.com with SMTP id v11so1632933qtq.12; Fri, 06 Nov 2020 15:17:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ngXMQxuAhgumkyikvz2rZCM+zAxLKnIx7tkSsw/SgKA=; b=UexX2EnJhZuRHIEeAiADPldZoeSIkLaPZt8mWzfAPQj7JXwBp2wEWAiu6pYnGwmXSA 2It5A3R22Pv1efGCvdKGau3qVG3DCG4jBGZkjiHTDZ9xF1cweozIRmgtLi25jLMMaeRI 2MsH+lFcUWp6gCyOEnDfEwgISddFVfUdNLbjGQg0ByXrZSCdyyQjG1lszNXzAg3PZx8u +PIJgJ1Um0QOApSP4VxN4Yt4moCJ4uV/gv1kbsegVTEJS2yrEjq9ynaxiVmRE9Hc9Yhd TRcCk0k2hMe7hebGjd9VbZ0x5RkN9OnMm6cthhoXEsaUyrLZ/Fbu45wMUTndeVZPPXyl kaig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ngXMQxuAhgumkyikvz2rZCM+zAxLKnIx7tkSsw/SgKA=; b=RLzuBhuMVgvcRj7zjiQsZnohp+eoHURT5lFW+F/y0Lm4T0sSNrd0GpiCy8S5FdElcl G+Gl6ljv2scdu2NDCLJFYI5lccSryuKvfI2sxkVsx4ahzPZULe0/lI4+NGFv+EwO5dkv yAvuOBQqiVt6HyP7/ZkXlhRFLfgYVgH99U5vSb7p+3nOdo7+wA7bUzqkHCcBPVIXO/3W GmgznhTwU5mzBe4OZf4c2kyW78k3oW1AeyEN6lmFRrfMaxryFeU6Wub+L0r5nVe1yxJa E0Eaxot4mCJ5JAlC7vqHUSv7p2Q7HinaE51sXt4Q/oW2VZ5YdsHPBQ6LhNMUQk0WA3i3 up6g== X-Gm-Message-State: AOAM531c+LSo8X5sWmcZXoQNoJZDbOgyBy3+oAEd1rp9mUpWTBYbZKV6 NMPkntCBtJ66qpPwKkGFF7KRW1iSK9E= X-Google-Smtp-Source: ABdhPJws5NozzSrlgNetDzJAnOcgLr4wFEq258CTkeG0QZ7xV3qoQSbh9IH+v109SBJ23U66BYJOpg== X-Received: by 2002:aed:3ba6:: with SMTP id r35mr3907750qte.269.1604704674064; Fri, 06 Nov 2020 15:17:54 -0800 (PST) Received: from soheil4.nyc.corp.google.com ([2620:0:1003:312:a6ae:11ff:fe18:6946]) by smtp.gmail.com with ESMTPSA id p136sm1519357qke.25.2020.11.06.15.17.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Nov 2020 15:17:53 -0800 (PST) From: Soheil Hassas Yeganeh To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, dave@stgolabs.net, edumazet@google.com, willemb@google.com, khazhy@google.com, guantaol@google.com, Soheil Hassas Yeganeh Subject: [PATCH 8/8] epoll: eliminate unnecessary lock for zero timeout Date: Fri, 6 Nov 2020 18:16:35 -0500 Message-Id: <20201106231635.3528496-9-soheil.kdev@gmail.com> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com> References: <20201106231635.3528496-1-soheil.kdev@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Soheil Hassas Yeganeh We call ep_events_available() under lock when timeout is 0, and then call it without locks in the loop for the other cases. Instead, call ep_events_available() without lock for all cases. For non-zero timeouts, we will recheck after adding the thread to the wait queue. For zero timeout cases, by definition, user is opportunistically polling and will have to call epoll_wait again in the future. Note that this lock was kept in c5a282e9635e9 because the whole loop was historically under lock. This patch results in a 1% CPU/RPC reduction in RPC benchmarks. Suggested-by: Eric Dumazet Signed-off-by: Soheil Hassas Yeganeh Reviewed-by: Eric Dumazet Reviewed-by: Willem de Bruijn Reviewed-by: Khazhismel Kumykov --- fs/eventpoll.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index f4e1be7ada26..1aa23b0be72b 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1830,7 +1830,7 @@ static inline struct timespec64 ep_set_mstimeout(long ms) static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, long timeout) { - int res, eavail = 0, timed_out = 0; + int res, eavail, timed_out = 0; u64 slack = 0; wait_queue_entry_t wait; ktime_t expires, *to = NULL; @@ -1846,18 +1846,21 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, } else if (timeout == 0) { /* * Avoid the unnecessary trip to the wait queue loop, if the - * caller specified a non blocking operation. We still need - * lock because we could race and not see an epi being added - * to the ready list while in irq callback. Thus incorrectly - * returning 0 back to userspace. + * caller specified a non blocking operation. */ timed_out = 1; - - write_lock_irq(&ep->lock); - eavail = ep_events_available(ep); - write_unlock_irq(&ep->lock); } + /* + * This call is racy: We may or may not see events that are being added + * to the ready list under the lock (e.g., in IRQ callbacks). For, cases + * with a non-zero timeout, this thread will check the ready list under + * lock and will added to the wait queue. For, cases with a zero + * timeout, the user by definition should not care and will have to + * recheck again. + */ + eavail = ep_events_available(ep); + while (1) { if (eavail) { /* @@ -1873,10 +1876,6 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) return 0; - eavail = ep_events_available(ep); - if (eavail) - continue; - eavail = ep_busy_loop(ep, timed_out); if (eavail) continue;