From patchwork Sun Mar 13 23:25:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12779480 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B5B6C433EF for ; Sun, 13 Mar 2022 23:25:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D2526B0072; Sun, 13 Mar 2022 19:25:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27FEA6B0073; Sun, 13 Mar 2022 19:25:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 122A66B0074; Sun, 13 Mar 2022 19:25:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id F374B6B0072 for ; Sun, 13 Mar 2022 19:25:57 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 9FF668249980 for ; Sun, 13 Mar 2022 23:25:57 +0000 (UTC) X-FDA: 79240947954.17.6CD31B1 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf24.hostedemail.com (Postfix) with ESMTP id 3B41B18002E for ; Sun, 13 Mar 2022 23:25:57 +0000 (UTC) Received: by mail-pj1-f53.google.com with SMTP id b8so12916427pjb.4 for ; Sun, 13 Mar 2022 16:25:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=WAKvHaxwkF54RfaaelAcwFZFObo76focrshoyvOHapk=; b=GjRCG7zfWbJT4fsFYqxKXWZHT3O3kiPi70JpCtjT0H8Sumx+BOJZtCQBNsqC88wrGJ 7bQ0FDK5JLUgLdeSSzrY1AMB0OmGO5i5Hp7i6Mnh3u0Pba7VJCL0OGa5ljK8M6m4vTHZ HclFb4lp5en3oYwnz51vlviEkcqil0Y24X9Ssb5z9SLS4AXqBJGq3QKXAbxUi7uJD1e3 lMtHw7hKnqEYx4f9G6IsCnxeBVtyibhcMVRjcx2tH5kKMdf9BZTtGL2nfPcQKj5/5xUZ bWRqW//aKDulyOdLwpqxBRC+SzKPl8tTjfYR4H42MsEMIs9B/OpDLB+ZjoG4UY14p0sk MkLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=WAKvHaxwkF54RfaaelAcwFZFObo76focrshoyvOHapk=; b=Vd/XcnrKWs7gSEuwjowaRfFSvmDCTQmhHK1LewYmx3uY6kC0eH0qXADW4aFhA/JSZW knGvJeAJKTDlHt8+vh/Qaqhg2uhOGboew4Xk76snn9ZLnYjrPvHEvswWqChs19UH8XBi 76fSt4FGfrTpPLdYX8sYlPc3j7wzN5l6Ib4ZRXSW/ytSlqb+/6E6l7b2GAj38KukW34q UjPwGCTyWprzNaZe1/jWANpBwHjWX6hWbTOx0ILJs4Gpa+MO+DPWPOjQ4WkSIm4NjyVG 1VWf5XfTQV+YXhdLE/gI3eIScZ98uerMVBTyFo0hxeXUvwmVqhYW2WDCeAVkPS1LUUPc wdZg== X-Gm-Message-State: AOAM530q7b789iLtjtPvvQSFEMyb0ZczdLfywlQXon5yAvKZsN1qZ2+q VRLkz0MSMAQTz8IEcXcXm9Q= X-Google-Smtp-Source: ABdhPJymcDVZQcoUrN0YqMtWtmK4QwpRlxQGxZlloC71h2nHXamfg0Aj2sdIqnv+kIXYU+PuxrNmPg== X-Received: by 2002:a17:90a:1d05:b0:1bf:6a85:fbf5 with SMTP id c5-20020a17090a1d0500b001bf6a85fbf5mr22125124pjd.205.1647213956089; Sun, 13 Mar 2022 16:25:56 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:c921:4439:3ddf:5385]) by smtp.gmail.com with ESMTPSA id t9-20020a655549000000b0036570f81245sm14475487pgr.84.2022.03.13.16.25.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Mar 2022 16:25:55 -0700 (PDT) From: Eric Dumazet To: Andrew Morton Cc: linux-kernel , linux-mm , Eric Dumazet , Eric Dumazet , Mel Gorman , Matthew Wilcox , Shakeel Butt , David Rientjes , Vlastimil Babka , Michal Hocko , Wei Xu , Greg Thelen , Hugh Dickins Subject: [PATCH] mm/page_alloc: call check_pcp_refill() while zone spinlock is not held Date: Sun, 13 Mar 2022 16:25:47 -0700 Message-Id: <20220313232547.3843690-1-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3B41B18002E X-Stat-Signature: z8ibr88r8cr1q9rtycx9jh5hct9mgnfb Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=GjRCG7zf; spf=pass (imf24.hostedemail.com: domain of eric.dumazet@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=eric.dumazet@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-HE-Tag: 1647213957-904813 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Eric Dumazet check_pcp_refill() is used from rmqueue_bulk() while zone spinlock is held. This used to be fine because check_pcp_refill() was testing only the head page, while its 'struct page' was very hot in the cpu caches. With ("mm/page_alloc: check high-order pages for corruption during PCP operations") check_pcp_refill() will add latencies for high order pages. We can defer the calls to check_pcp_refill() after the zone spinlock has been released. Signed-off-by: Eric Dumazet Cc: Mel Gorman Cc: Matthew Wilcox Cc: Shakeel Butt Cc: David Rientjes Cc: Vlastimil Babka Cc: Michal Hocko Cc: Wei Xu Cc: Greg Thelen Cc: Hugh Dickins --- mm/page_alloc.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c9ebf0635d592c6f58df9793ce9fa213371a9a7f..5f0531c11ad668b1c4426ebddc17821aca824783 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3024,7 +3024,9 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long count, struct list_head *list, int migratetype, unsigned int alloc_flags) { + struct page *page, *tmp; int i, allocated = 0; + int free_cma_pages = 0; /* * local_lock_irq held so equivalent to spin_lock_irqsave for @@ -3032,14 +3034,10 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, */ spin_lock(&zone->lock); for (i = 0; i < count; ++i) { - struct page *page = __rmqueue(zone, order, migratetype, - alloc_flags); + page = __rmqueue(zone, order, migratetype, alloc_flags); if (unlikely(page == NULL)) break; - if (unlikely(check_pcp_refill(page))) - continue; - /* * Split buddy pages returned by expand() are received here in * physical page order. The page is added to the tail of @@ -3052,9 +3050,6 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, */ list_add_tail(&page->lru, list); allocated++; - if (is_migrate_cma(get_pcppage_migratetype(page))) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, - -(1 << order)); } /* @@ -3065,6 +3060,16 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, */ __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); spin_unlock(&zone->lock); + list_for_each_entry_safe(page, tmp, list, lru) { + if (unlikely(check_pcp_refill(page))) { + list_del(&page->lru); + allocated--; + } else if (is_migrate_cma(get_pcppage_migratetype(page))) { + free_cma_pages++; + } + } + if (free_cma_pages) + __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(free_cma_pages << order)); return allocated; }