From patchwork Wed Oct 17 06:33:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10644629 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED9D5157A for ; Wed, 17 Oct 2018 06:33:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE230292F4 for ; Wed, 17 Oct 2018 06:33:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CFED02A441; Wed, 17 Oct 2018 06:33:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14EF1292F4 for ; Wed, 17 Oct 2018 06:33:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 302516B0006; Wed, 17 Oct 2018 02:33:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 25B786B0008; Wed, 17 Oct 2018 02:33:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FEFC6B000A; Wed, 17 Oct 2018 02:33:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id BD01D6B0006 for ; Wed, 17 Oct 2018 02:33:41 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id s24-v6so20085143plp.12 for ; Tue, 16 Oct 2018 23:33:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ieSkXsPkKgXtPUQL/svrPOXc6rvTONrU4g1f5Djtbxk=; b=Tlq02YSV0rFOufvyzRwTTYghhIDnw9G8V20SWxqJ0cfqwq743yV3mTWBGMkNkr2V7V CW7zEqbx1sC3JNYEeAmTcsaDAiOBBkuSi4UHQVF84jSBZhLBqkM+IYwLIIqzfFvHLc3a oqYGB6FJVwFumfOTvN6F3sTzN85adDj/+2jt+0nSZR0XA2Ex1YufhrsaMJ7p5ZN7dbFu f5PTB3rnEVZ9dymt8n0X7KRBqPk9u2I7SJ8y39wYjnGvNOrevCSXk7sP4NdZ3weAqRTF 7uQgB8S0qkXpO2GYGtAteAew2VhAdPaL2kIVaALYxd0MOL//CY0VuQBMusTVNkNoWkEr /W7Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohP0riFCvl2ov2FDPl3zapHPAoUVVggQJwEGpP5i0ngBZZr3P7v OY7cF3vBeFdX/CfOBpt66C56heTLL+f83wLjGShjxzteR7o6zOiLwiqa7ywWU2V+s2zfwe6nqWP I/ZKJH/ZSgkfboCShg+YsX+A44Cqz/YOlxwA5XI5bPSqkAnHxxtXkDrmrc7RIeIz58Q== X-Received: by 2002:a62:aa17:: with SMTP id e23-v6mr25393869pff.211.1539758021438; Tue, 16 Oct 2018 23:33:41 -0700 (PDT) X-Google-Smtp-Source: ACcGV618R+pokCSqVP+ORC2p4x5OzQvOqiA2WpWEhweQwEeKxH5StMfrsnBdvV/Z0CLBho8B4A8S X-Received: by 2002:a62:aa17:: with SMTP id e23-v6mr25393830pff.211.1539758020450; Tue, 16 Oct 2018 23:33:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539758020; cv=none; d=google.com; s=arc-20160816; b=RBJnpAuKUHhXVCc1v8TUwB+sYXJTf3pLMAl1hmv+cWFp40/27SnPbDtXoidiZT3uLo F8Q/GELMHvcK4pSnhCuWC99w0BRkz7uYN9nxZMex7GocDIpCap7QqOXgBNpzc3sUPnh5 hUqdupSxq7h9jNGlU6TMSUPt4FklNAb08s8YPQtcI2GnRAriMC5X7kVB4nVr5vrolR1k EcNw93tXvFT6M0I37HozA39nEmc8fN27SGQmyblQVPRuQ+rG77Jq9g6pD1b3+otG7s4+ vJ+E3JYaHP2Hd1PgT67tCYmJLPgn/gMUdPUsa6vf1pW+Lw5tzl1/8qtwxE50ilmXCtEf xqUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ieSkXsPkKgXtPUQL/svrPOXc6rvTONrU4g1f5Djtbxk=; b=NTqGV+SFo+A9Qd2YWGR2tDiu3FZWtv1zvNM/QPBSd7Qa00KL62+EWm/h72ehPNSyNb wmtU61KPGcu+CieINe3yiVESZk6OZcVCwClsHbOcLm/nuRBY4Z+8Uuua3YEjlctK9qOH +7QtpcJS25KtyYQOo25H7xlbcAB9q7edqVaQz0yygM048pO4vqGMA8x6/vzLH9gTt+LB WV7dBBojRhQDDbwQTrDlQvKm7BeGbA2HgiomCAOCQLxSVOTxdokCjVcVMMTDVhRnjWq1 Jon1Fq+P60csj1z92x+kEuAY2Z5Ep4WeZpDSHgbANJjLlU7YSqK5w15z9YKKl/9VKqli 131Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id f2-v6si16427856pgf.423.2018.10.16.23.33.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 23:33:40 -0700 (PDT) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Oct 2018 23:33:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,391,1534834800"; d="scan'208";a="273090084" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga006.fm.intel.com with ESMTP; 16 Oct 2018 23:33:37 -0700 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Huang Ying , Dave Hansen , Kemi Wang , Tim Chen , Andi Kleen , Michal Hocko , Vlastimil Babka , Mel Gorman , Matthew Wilcox , Daniel Jordan , Tariq Toukan , Jesper Dangaard Brouer Subject: [RFC v4 PATCH 2/5] mm/__free_one_page: skip merge for order-0 page unless compaction failed Date: Wed, 17 Oct 2018 14:33:27 +0800 Message-Id: <20181017063330.15384-3-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181017063330.15384-1-aaron.lu@intel.com> References: <20181017063330.15384-1-aaron.lu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Running will-it-scale/page_fault1 process mode workload on a 2 sockets Intel Skylake server showed severe lock contention of zone->lock, as high as about 80%(42% on allocation path and 35% on free path) CPU cycles are burnt spinning. With perf, the most time consuming part inside that lock on free path is cache missing on page structures, mostly on the to-be-freed page's buddy due to merging. One way to avoid this overhead is not do any merging at all for order-0 pages. With this approach, the lock contention for zone->lock on free path dropped to 1.1% but allocation side still has as high as 42% lock contention. In the meantime, the dropped lock contention on free side doesn't translate to performance increase, instead, it's consumed by increased lock contention of the per node lru_lock(rose from 5% to 37%) and the final performance slightly dropped about 1%. Though performance dropped a little, it almost eliminated zone lock contention on free path and it is the foundation for the next patch that eliminates zone lock contention for allocation path. Suggested-by: Dave Hansen Signed-off-by: Aaron Lu --- include/linux/mm_types.h | 9 +++- mm/compaction.c | 13 +++++- mm/internal.h | 27 ++++++++++++ mm/page_alloc.c | 88 ++++++++++++++++++++++++++++++++++------ 4 files changed, 121 insertions(+), 16 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5ed8f6292a53..aed93053ef6e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -179,8 +179,13 @@ struct page { int units; /* SLOB */ }; - /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ - atomic_t _refcount; + union { + /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ + atomic_t _refcount; + + /* For pages in Buddy: if skipped merging when added to Buddy */ + bool buddy_merge_skipped; + }; #ifdef CONFIG_MEMCG struct mem_cgroup *mem_cgroup; diff --git a/mm/compaction.c b/mm/compaction.c index faca45ebe62d..0c9c7a30dde3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -777,8 +777,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * potential isolation targets. */ if (PageBuddy(page)) { - unsigned long freepage_order = page_order_unsafe(page); + unsigned long freepage_order; + /* + * If this is a merge_skipped page, do merge now + * since high-order pages are needed. zone lock + * isn't taken for the merge_skipped check so the + * check could be wrong but the worst case is we + * lose a merge opportunity. + */ + if (page_merge_was_skipped(page)) + try_to_merge_page(page); + + freepage_order = page_order_unsafe(page); /* * Without lock, we cannot be sure that what we got is * a valid page order. Consider only values in the diff --git a/mm/internal.h b/mm/internal.h index 87256ae1bef8..c166735a559e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -527,4 +527,31 @@ static inline bool is_migrate_highatomic_page(struct page *page) void setup_zone_pageset(struct zone *zone); extern struct page *alloc_new_node_page(struct page *page, unsigned long node); + +static inline bool page_merge_was_skipped(struct page *page) +{ + return page->buddy_merge_skipped; +} + +void try_to_merge_page(struct page *page); + +#ifdef CONFIG_COMPACTION +static inline bool can_skip_merge(struct zone *zone, int order) +{ + /* Compaction has failed in this zone, we shouldn't skip merging */ + if (zone->compact_considered) + return false; + + /* Only consider no_merge for order 0 pages */ + if (order) + return false; + + return true; +} +#else /* CONFIG_COMPACTION */ +static inline bool can_skip_merge(struct zone *zone, int order) +{ + return false; +} +#endif /* CONFIG_COMPACTION */ #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 14c20bb3a3da..76d471e0ab24 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -691,6 +691,16 @@ static inline void clear_page_guard(struct zone *zone, struct page *page, unsigned int order, int migratetype) {} #endif +static inline void set_page_merge_skipped(struct page *page) +{ + page->buddy_merge_skipped = true; +} + +static inline void clear_page_merge_skipped(struct page *page) +{ + page->buddy_merge_skipped = false; +} + static inline void set_page_order(struct page *page, unsigned int order) { set_page_private(page, order); @@ -700,6 +710,7 @@ static inline void set_page_order(struct page *page, unsigned int order) static inline void add_to_buddy_common(struct page *page, struct zone *zone, unsigned int order) { + clear_page_merge_skipped(page); set_page_order(page, order); zone->free_area[order].nr_free++; } @@ -730,6 +741,7 @@ static inline void remove_from_buddy(struct page *page, struct zone *zone, list_del(&page->lru); zone->free_area[order].nr_free--; rmv_page_order(page); + clear_page_merge_skipped(page); } /* @@ -797,7 +809,7 @@ static inline int page_is_buddy(struct page *page, struct page *buddy, * -- nyc */ -static inline void __free_one_page(struct page *page, +static inline void do_merge(struct page *page, unsigned long pfn, struct zone *zone, unsigned int order, int migratetype) @@ -809,16 +821,6 @@ static inline void __free_one_page(struct page *page, max_order = min_t(unsigned int, MAX_ORDER, pageblock_order + 1); - VM_BUG_ON(!zone_is_initialized(zone)); - VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); - - VM_BUG_ON(migratetype == -1); - if (likely(!is_migrate_isolate(migratetype))) - __mod_zone_freepage_state(zone, 1 << order, migratetype); - - VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); - VM_BUG_ON_PAGE(bad_range(zone, page), page); - continue_merging: while (order < max_order - 1) { buddy_pfn = __find_buddy_pfn(pfn, order); @@ -891,6 +893,61 @@ static inline void __free_one_page(struct page *page, add_to_buddy_head(page, zone, order, migratetype); } +void try_to_merge_page(struct page *page) +{ + unsigned long pfn, buddy_pfn, flags; + struct page *buddy; + struct zone *zone; + + /* + * No need to do merging if buddy is not free. + * zone lock isn't taken so this could be wrong but worst case + * is we lose a merge opportunity. + */ + pfn = page_to_pfn(page); + buddy_pfn = __find_buddy_pfn(pfn, 0); + buddy = page + (buddy_pfn - pfn); + if (!PageBuddy(buddy)) + return; + + zone = page_zone(page); + spin_lock_irqsave(&zone->lock, flags); + /* Verify again after taking the lock */ + if (likely(PageBuddy(page) && page_merge_was_skipped(page) && + PageBuddy(buddy))) { + int mt = get_pageblock_migratetype(page); + + remove_from_buddy(page, zone, 0); + do_merge(page, pfn, zone, 0, mt); + } + spin_unlock_irqrestore(&zone->lock, flags); +} + +static inline void __free_one_page(struct page *page, + unsigned long pfn, + struct zone *zone, unsigned int order, + int migratetype) +{ + VM_BUG_ON(!zone_is_initialized(zone)); + VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); + + VM_BUG_ON(migratetype == -1); + if (likely(!is_migrate_isolate(migratetype))) + __mod_zone_freepage_state(zone, 1 << order, migratetype); + + VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); + VM_BUG_ON_PAGE(bad_range(zone, page), page); + + if (can_skip_merge(zone, order)) { + add_to_buddy_head(page, zone, 0, migratetype); + set_page_merge_skipped(page); + return; + } + + do_merge(page, pfn, zone, order, migratetype); +} + + /* * A bad page could be due to a number of fields. Instead of multiple branches, * try and check multiple fields with one check. The caller must do a detailed @@ -1148,9 +1205,14 @@ static void free_pcppages_bulk(struct zone *zone, int count, * can be offset by reduced memory latency later. To * avoid excessive prefetching due to large count, only * prefetch buddy for the first pcp->batch nr of pages. + * + * If merge can be skipped, no need to prefetch buddy. */ - if (prefetch_nr++ < pcp->batch) - prefetch_buddy(page); + if (can_skip_merge(zone, 0) || prefetch_nr > pcp->batch) + continue; + + prefetch_buddy(page); + prefetch_nr++; } while (--count && --batch_free && !list_empty(list)); }