From patchwork Fri Mar 4 01:59:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12768432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3133C433F5 for ; Fri, 4 Mar 2022 03:53:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A4378D0003; Thu, 3 Mar 2022 22:53:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 653F78D0002; Thu, 3 Mar 2022 22:53:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F5208D0003; Thu, 3 Mar 2022 22:53:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 3946E8D0002 for ; Thu, 3 Mar 2022 22:53:59 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EA2EB6B8 for ; Fri, 4 Mar 2022 03:53:58 +0000 (UTC) X-FDA: 79205335356.01.8DDACD5 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf21.hostedemail.com (Postfix) with ESMTP id 749CA1C0011 for ; Fri, 4 Mar 2022 03:53:58 +0000 (UTC) Received: by mail-qv1-f44.google.com with SMTP id x3so5739990qvd.8 for ; Thu, 03 Mar 2022 19:53:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=pNg4kYdxlqLlNjGIaXFPv1B8vrhqGS0uOksmeWZwteY=; b=Yn0FTrDQatEF8PVLYqXopPVgnNKeu9lW4Zw8fNZ5BHwT69cz0rxT2QHh2/ntvLX/zK bMgZI3k/5CagA9npMGZnDnn5uaAaTEzMR441qWmN9efL5NC59ZJnQ106kviOBjGgSA9w O0ZM1E5SwNwKwX1yMlJG/6dtOwOk9iHwHZsZ7Tz4bv26+1+gI4LiFR4UX373TwWVg2nz qu9CII6nKmUfew9ROh8F9KQNmMKusTg0d+aQbO61HsHklNlbdxVyaatQ2xGFPnIE9nkh bFWVMwHajfUOYRjHttDpu7EAEplD1wWoHcRB8GE1DH1Lr3ZWgkAib2pyEyEN/fJL+6vk 9Dpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=pNg4kYdxlqLlNjGIaXFPv1B8vrhqGS0uOksmeWZwteY=; b=NGjYQtK45lOrF8AO64TIfd1visey+sGyMa/IddNZjEF8xBCAxj/FBr1AyKlWhIFgdN kJ78H9YpzccN7l/0XpzRjT3JpLEyP216yUx+WUAkEuaWwruF2d6/TX3oLAy7FFrvkItB PC00ZcBDNH4mTkb4lsYTkE1wh6BssM/dhFcbAAKRHpFDG1S41zt4PpWJp4Y7UZ1fZfHN JmSwdJMwh2ZHojTmKeaWeWETkytMNLTkD1eD4iFcoOb/is7/3NBvg76X4CQobUfBhJGp nlikKQqRLnEY0qkjb8uK8r0Ekl+dMcCgi2Ti4wofZkTFTuJYGKL7Eum+q+/oBGh9QSx8 51TQ== X-Gm-Message-State: AOAM532DEoKvo6V1mswYxpyzuGw3JKEXLj/MmgitmBSRcYXIbv62Yq+r iZA3XNG6UrbqDABndv87VMfW2sPhHH4= X-Google-Smtp-Source: ABdhPJwgnJJcohamGBceHBePfAkZJZnry0WsByYaX3qChlXOBgHodh4y3rKqkzAB9D27cneAm4Sx6Q== X-Received: by 2002:a17:902:aa84:b0:150:8d6:bdec with SMTP id d4-20020a170902aa8400b0015008d6bdecmr38486225plr.118.1646359185389; Thu, 03 Mar 2022 17:59:45 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:5388:c313:5e37:a261]) by smtp.gmail.com with ESMTPSA id c5-20020a056a00248500b004f6b5ddcc65sm296707pfv.199.2022.03.03.17.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Mar 2022 17:59:44 -0800 (PST) From: Eric Dumazet To: Andrew Morton Cc: linux-kernel , linux-mm , Eric Dumazet , Eric Dumazet , Mel Gorman , Vlastimil Babka , Michal Hocko , Shakeel Butt , Wei Xu , Greg Thelen , Hugh Dickins , David Rientjes Subject: [PATCH] mm/page_alloc: call check_new_pages() while zone spinlock is not held Date: Thu, 3 Mar 2022 17:59:41 -0800 Message-Id: <20220304015941.1704249-1-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.35.1.616.g0bdcbb4464-goog MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 749CA1C0011 X-Stat-Signature: 45tthb5zahgh1xfromp5qwogx8wqejzr Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yn0FTrDQ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of eric.dumazet@gmail.com designates 209.85.219.44 as permitted sender) smtp.mailfrom=eric.dumazet@gmail.com X-HE-Tag: 1646366038-163238 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Eric Dumazet For high order pages not using pcp, rmqueue() is currently calling the costly check_new_pages() while zone spinlock is held. This is not needed, we can release the spinlock sooner to reduce zone spinlock contention. Signed-off-by: Eric Dumazet Cc: Mel Gorman Cc: Vlastimil Babka Cc: Michal Hocko Cc: Shakeel Butt Cc: Wei Xu Cc: Greg Thelen Cc: Hugh Dickins Cc: David Rientjes --- mm/page_alloc.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3589febc6d31928f850ebe5a4015ddc40e0469f3..0890a65f8cc2259e82bc1f5ba95a592fb30f9fb8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3685,7 +3685,6 @@ struct page *rmqueue(struct zone *preferred_zone, gfp_t gfp_flags, unsigned int alloc_flags, int migratetype) { - unsigned long flags; struct page *page; if (likely(pcp_allowed_order(order))) { @@ -3706,10 +3705,12 @@ struct page *rmqueue(struct zone *preferred_zone, * allocate greater than order-1 page units with __GFP_NOFAIL. */ WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); - spin_lock_irqsave(&zone->lock, flags); do { + unsigned long flags; + page = NULL; + spin_lock_irqsave(&zone->lock, flags); /* * order-0 request can reach here when the pcplist is skipped * due to non-CMA allocation context. HIGHATOMIC area is @@ -3723,13 +3724,13 @@ struct page *rmqueue(struct zone *preferred_zone, } if (!page) page = __rmqueue(zone, order, migratetype, alloc_flags); - } while (page && check_new_pages(page, order)); - if (!page) - goto failed; + spin_unlock_irqrestore(&zone->lock, flags); + if (!page) + return NULL; + } while (check_new_pages(page, order)); __mod_zone_freepage_state(zone, -(1 << order), get_pcppage_migratetype(page)); - spin_unlock_irqrestore(&zone->lock, flags); __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); zone_statistics(preferred_zone, zone, 1); @@ -3743,10 +3744,6 @@ struct page *rmqueue(struct zone *preferred_zone, VM_BUG_ON_PAGE(page && bad_range(zone, page), page); return page; - -failed: - spin_unlock_irqrestore(&zone->lock, flags); - return NULL; } #ifdef CONFIG_FAIL_PAGE_ALLOC