[13/78] epoll: eliminate unnecessary lock for zero timeout

Message ID	20201218220206.CTX0JqXyM%akpm@linux-foundation.org (mailing list archive)
State	New, archived
Headers	show Return-Path: <SRS0=Y0WR=FW=kvack.org=owner-linux-mm@kernel.org> DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED19223B7E Date: Fri, 18 Dec 2020 14:02:06 -0800 From: Andrew Morton <akpm@linux-foundation.org> To: akpm@linux-foundation.org, edumazet@google.com, guantaol@google.com, khazhy@google.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, soheil@google.com, torvalds@linux-foundation.org, willemb@google.com Subject: [patch 13/78] epoll: eliminate unnecessary lock for zero timeout Message-ID: <20201218220206.CTX0JqXyM%akpm@linux-foundation.org> In-Reply-To: <20201218140046.497484741326828e5b5d46ec@linux-foundation.org> User-Agent: s-nail v14.8.16 Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	[01/78] mm/memcg: bail early from swap accounting if memcg disabled \| expand [01/78] mm/memcg: bail early from swap accounting if memcg disabled [02/78] mm/memcg: warning on !memcg after readahead page charged [03/78] mm/memcg: remove unused definitions [04/78] mm, kvm: account kvm_vcpu_mmap to kmemcg [05/78] mm/memcontrol:rewrite mem_cgroup_page_lruvec() [06/78] epoll: check for events when removing a timed out thread from the wait queue [07/78] epoll: simplify signal handling [08/78] epoll: pull fatal signal checks into ep_send_events() [09/78] epoll: move eavail next to the list_empty_careful check [10/78] epoll: simplify and optimize busy loop logic [11/78] epoll: pull all code between fetch_events and send_event into the loop [12/78] epoll: replace gotos with a proper loop [13/78] epoll: eliminate unnecessary lock for zero timeout [14/78] kasan: drop unnecessary GPL text from comment headers [15/78] kasan: KASAN_VMALLOC depends on KASAN_GENERIC [16/78] kasan: group vmalloc code [17/78] kasan: shadow declarations only for software modes [18/78] kasan: rename (un)poison_shadow to (un)poison_range [19/78] kasan: rename KASAN_SHADOW_* to KASAN_GRANULE_* [20/78] kasan: only build init.c for software modes [21/78] kasan: split out shadow.c from common.c [22/78] kasan: define KASAN_MEMORY_PER_SHADOW_PAGE [23/78] kasan: rename report and tags files [24/78] kasan: don't duplicate config dependencies [25/78] kasan: hide invalid free check implementation [26/78] kasan: decode stack frame only with KASAN_STACK_ENABLE [27/78] kasan, arm64: only init shadow for software modes [28/78] kasan, arm64: only use kasan_depth for software modes [29/78] kasan, arm64: move initialization message [30/78] kasan, arm64: rename kasan_init_tags and mark as __init [31/78] kasan: rename addr_has_shadow to addr_has_metadata [32/78] kasan: rename print_shadow_for_address to print_memory_metadata [33/78] kasan: rename SHADOW layout macros to META [34/78] kasan: separate metadata_fetch_row for each mode [35/78] kasan: introduce CONFIG_KASAN_HW_TAGS [36/78] arm64: enable armv8.5-a asm-arch option [37/78] arm64: mte: add in-kernel MTE helpers [38/78] arm64: mte: reset the page tag in page->flags [39/78] arm64: mte: add in-kernel tag fault handler [40/78] arm64: kasan: allow enabling in-kernel MTE [41/78] arm64: mte: convert gcr_user into an exclude mask [42/78] arm64: mte: switch GCR_EL1 in kernel entry and exit [43/78] kasan, mm: untag page address in free_reserved_area [44/78] arm64: kasan: align allocations for HW_TAGS [45/78] arm64: kasan: add arch layer for memory tagging helpers [46/78] kasan: define KASAN_GRANULE_SIZE for HW_TAGS [47/78] kasan, x86, s390: update undef CONFIG_KASAN [48/78] kasan, arm64: expand CONFIG_KASAN checks [49/78] kasan, arm64: implement HW_TAGS runtime [50/78] kasan, arm64: print report from tag fault handler [51/78] kasan, mm: reset tags when accessing metadata [52/78] kasan, arm64: enable CONFIG_KASAN_HW_TAGS [53/78] kasan: add documentation for hardware tag-based mode [54/78] kselftest/arm64: check GCR_EL1 after context switch [55/78] kasan: simplify quarantine_put call site [56/78] kasan: rename get_alloc/free_info [57/78] kasan: introduce set_alloc_info [58/78] kasan, arm64: unpoison stack only with CONFIG_KASAN_STACK [59/78] kasan: allow VMAP_STACK for HW_TAGS mode [60/78] kasan: remove __kasan_unpoison_stack [61/78] kasan: inline kasan_reset_tag for tag-based modes [62/78] kasan: inline random_tag for HW_TAGS [63/78] kasan: open-code kasan_unpoison_slab [64/78] kasan: inline (un)poison_range and check_invalid_free [65/78] kasan: add and integrate kasan boot parameters [66/78] kasan, mm: check kasan_enabled in annotations [67/78] kasan, mm: rename kasan_poison_kfree [68/78] kasan: don't round_up too much [69/78] kasan: simplify assign_tag and set_tag calls [70/78] kasan: clarify comment in __kasan_kfree_large [71/78] kasan: sanitize objects when metadata doesn't fit [72/78] kasan, mm: allow cache merging with no metadata [73/78] kasan: update documentation [74/78] mm/Kconfig: fix spelling mistake "whats" -> "what's" [75/78] epoll: convert internal api to timespec64 [76/78] epoll: add syscall epoll_pwait2 [77/78] epoll: wire up syscall epoll_pwait2 [78/78] selftests/filesystems: expand epoll with epoll_pwait2

Message ID

20201218220206.CTX0JqXyM%akpm@linux-foundation.org (mailing list archive)

State

New, archived

Headers

DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED19223B7E
Date: Fri, 18 Dec 2020 14:02:06 -0800
From: Andrew Morton <akpm@linux-foundation.org>
To: akpm@linux-foundation.org, edumazet@google.com, guantaol@google.com,
 khazhy@google.com, linux-mm@kvack.org, mm-commits@vger.kernel.org,
 soheil@google.com, torvalds@linux-foundation.org, willemb@google.com
Subject: [patch 13/78] epoll: eliminate unnecessary lock for zero
 timeout
Message-ID: <20201218220206.CTX0JqXyM%akpm@linux-foundation.org>
In-Reply-To: <20201218140046.497484741326828e5b5d46ec@linux-foundation.org>
User-Agent: s-nail v14.8.16
Sender: owner-linux-mm@kvack.org
Precedence: bulk

Series

[01/78] mm/memcg: bail early from swap accounting if memcg disabled | expand

Commit Message

Andrew Morton Dec. 18, 2020, 10:02 p.m. UTC

From: Soheil Hassas Yeganeh <soheil@google.com>
Subject: epoll: eliminate unnecessary lock for zero timeout

We call ep_events_available() under lock when timeout is 0, and then call
it without locks in the loop for the other cases.

Instead, call ep_events_available() without lock for all cases.  For
non-zero timeouts, we will recheck after adding the thread to the wait
queue.  For zero timeout cases, by definition, user is opportunistically
polling and will have to call epoll_wait again in the future.

Note that this lock was kept in c5a282e9635e9 because the whole loop was
historically under lock.

This patch results in a 1% CPU/RPC reduction in RPC benchmarks.

Link: https://lkml.kernel.org/r/20201106231635.3528496-9-soheil.kdev@gmail.com
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Suggested-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Khazhismel Kumykov <khazhy@google.com>
Cc: Guantao Liu <guantaol@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---

 fs/eventpoll.c |   25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

--- a/fs/eventpoll.c~epoll-eliminate-unnecessary-lock-for-zero-timeout
+++ a/fs/eventpoll.c
@@ -1743,7 +1743,7 @@  static inline struct timespec64 ep_set_m
 static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 		   int maxevents, long timeout)
 {
-	int res, eavail = 0, timed_out = 0;
+	int res, eavail, timed_out = 0;
 	u64 slack = 0;
 	wait_queue_entry_t wait;
 	ktime_t expires, *to = NULL;
@@ -1759,18 +1759,21 @@  static int ep_poll(struct eventpoll *ep,
 	} else if (timeout == 0) {
 		/*
 		 * Avoid the unnecessary trip to the wait queue loop, if the
-		 * caller specified a non blocking operation. We still need
-		 * lock because we could race and not see an epi being added
-		 * to the ready list while in irq callback. Thus incorrectly
-		 * returning 0 back to userspace.
+		 * caller specified a non blocking operation.
 		 */
 		timed_out = 1;
-
-		write_lock_irq(&ep->lock);
-		eavail = ep_events_available(ep);
-		write_unlock_irq(&ep->lock);
 	}
 
+	/*
+	 * This call is racy: We may or may not see events that are being added
+	 * to the ready list under the lock (e.g., in IRQ callbacks). For, cases
+	 * with a non-zero timeout, this thread will check the ready list under
+	 * lock and will added to the wait queue.  For, cases with a zero
+	 * timeout, the user by definition should not care and will have to
+	 * recheck again.
+	 */
+	eavail = ep_events_available(ep);
+
 	while (1) {
 		if (eavail) {
 			/*
@@ -1786,10 +1789,6 @@  static int ep_poll(struct eventpoll *ep,
 		if (timed_out)
 			return 0;
 
-		eavail = ep_events_available(ep);
-		if (eavail)
-			continue;
-
 		eavail = ep_busy_loop(ep, timed_out);
 		if (eavail)
 			continue;

[13/78] epoll: eliminate unnecessary lock for zero timeout

Commit Message

Patch