From patchwork Tue Sep 11 05:36:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10595077 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3E2514E0 for ; Tue, 11 Sep 2018 05:36:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3359292A8 for ; Tue, 11 Sep 2018 05:36:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A782D292B6; Tue, 11 Sep 2018 05:36:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3187F292AD for ; Tue, 11 Sep 2018 05:36:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A6598E000C; Tue, 11 Sep 2018 01:36:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 52EC18E0001; Tue, 11 Sep 2018 01:36:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 421428E000C; Tue, 11 Sep 2018 01:36:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id E452A8E0001 for ; Tue, 11 Sep 2018 01:36:46 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id x85-v6so12373407pfe.13 for ; Mon, 10 Sep 2018 22:36:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0EDaHouM/o47pSa2XbLulzepPbv3Y/nvnAumJVbrvwI=; b=H+A1EvdbfVgoiLEzbysllucGZB9+6lmJUEHjtPaEPE3RsGIhSWgk5/v1Av1e8RIxw/ 8usN3bi9QKrBrwYRF15Iq/IAzrCwWTAfQgGWudRgzxNlwsYJ/9JVJjzqYJya7jVpoFJ/ VtCDtQeD6WPNYBPTFdpAKhbHJtH+LKidtHpiMbIisKqdxjd+veh5M+Db7ppgvfD9hX4/ nU4RNmIPrhZaw4yegNIGfMP+EaVsxMRCVXkpplAxFYUhqfn8LXBCmVfUB55RAY9JELjL nn7ZHbhEQmU7cIUKFsUrBNppw0Jpe45u4aJQqiZDBfI0UMJc4g9y78xbMPuPVWTnxN0u fTQA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51DNjxtSljfWau+o2BhX4+pd/6A2eCPnOwbBjW3b4MoqNYXpt0+l BL7I6JLtku9Pk0dw3m5NzvBzWOgNXDZokMNjvpFsD2ivTev8Lv5np+VZzm3e6Efe3M3+O1mb9ZR jnhaqvUrSleB043GSmmhESo89VXvOPHazdzK404AhRbOJVYtv1vbdWWSghcD4b/zdgg== X-Received: by 2002:a62:b0e:: with SMTP id t14-v6mr26533489pfi.36.1536644206625; Mon, 10 Sep 2018 22:36:46 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaE9py5RfWzTeO/mCOhsH4z7w4dNIdK7WOop7m6RYrHqXc6bZbvsVD+NR8sB2GSHm2Q5Zlj X-Received: by 2002:a62:b0e:: with SMTP id t14-v6mr26533424pfi.36.1536644205833; Mon, 10 Sep 2018 22:36:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536644205; cv=none; d=google.com; s=arc-20160816; b=MQwlQx5IJfH5gWhasbhLSNiESYIsOwzu9Bh/m8jNt3a/laF3eESliC2QPUKcDylaZx jkYqlIfbp+e86g7o1wCLBgJp0jF/pHhrQTT5ABK+R7RDTsbEOXRQywqx4UUEXeD5VXn5 FOppaNUwKfUWlGtYPduwDMHtma43iwcLuQE5ZoO/hbPv1LMJfadsMSReqIosBhX3NVhX 7xXEfZKRTbWH2ORGP9nrRQKrrlUMgtRzNyAwueoAfzP7PM394JjCAmaT+Q7wknkTXCh1 +9XUfWvmU3DQLYgrQeLvjroWyEqTCQu/nYLIkQUCzgJgGrImjRFAhG+0jwlhPwhtCfpF O7cA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=0EDaHouM/o47pSa2XbLulzepPbv3Y/nvnAumJVbrvwI=; b=EgT4kXsJhIyAlX+Wv5PCdCJ5TnqEXDFtJTddITy83iYXssCB8oGzqfcGQxvagmiMlY fPMUAfRANSFsLPnhc6NsDmJSQGFxonYhy/kpqXCOmdpnlxWRZ+R4H9XHTw0Yvx9CoP1V jbud8efB8rSSd7p0ppSyyRnCj2wTKMfa4GnbTmrdNqyE8OLOyNjO3beMCKdU+IlpmXmx sCR/UT8Uv37qQQBuwnqL82cVskrUdZwqgcMWeIfEAthktVvslM4NvnRLZY3E4b9JB8hY lncPM0rmskyznHR4rXzMfEB78A+3euF5rKF1QNDrpqRSJSWKeG+czDgHxOMyMXntsO5r OEVw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga11.intel.com (mga11.intel.com. [192.55.52.93]) by mx.google.com with ESMTPS id c19-v6si20646945pfc.18.2018.09.10.22.36.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 22:36:45 -0700 (PDT) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 192.55.52.93 as permitted sender) client-ip=192.55.52.93; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 192.55.52.93 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2018 22:36:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,359,1531810800"; d="scan'208";a="262426344" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga006.fm.intel.com with ESMTP; 10 Sep 2018 22:36:43 -0700 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Dave Hansen , Michal Hocko , Vlastimil Babka , Mel Gorman , Matthew Wilcox , Daniel Jordan , Tariq Toukan , Yosef Lev , Jesper Dangaard Brouer Subject: [RFC PATCH 9/9] mm: page_alloc: merge before sending pages to global pool Date: Tue, 11 Sep 2018 13:36:16 +0800 Message-Id: <20180911053616.6894-10-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180911053616.6894-1-aaron.lu@intel.com> References: <20180911053616.6894-1-aaron.lu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Now that we have mergable pages in Buddy unmerged, this is a step to reduce such things from happening to some extent. Suppose two buddy pages are on the list to be freed in free_pcppages_bulk(), the first page goes to merge but its buddy is not in Buddy yet so we hold it locally as an order0 page; then its buddy page goes to merge and couldn't merge either because we hold the first page locally instead of having it in Buddy. The end result is, we have two mergable buddy pages but failed to merge it. So this patch will attempt merge for these to-be-freed pages before acquiring any lock, it could, to some extent, reduce fragmentation caused by last patch. With this change, the pcp_drain trace isn't easy to use so I removed it. Signed-off-by: Aaron Lu --- mm/page_alloc.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 73 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index df38c3f2a1cc..d3eafe857713 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1098,6 +1098,72 @@ void __init percpu_mergelist_init(void) } } +static inline bool buddy_in_list(struct page *page, struct page *buddy, + struct list_head *list) +{ + list_for_each_entry_continue(page, list, lru) + if (page == buddy) + return true; + + return false; +} + +static inline void merge_in_pcp(struct list_head *list) +{ + int order; + struct page *page; + + /* Set order information to 0 initially since they are PCP pages */ + list_for_each_entry(page, list, lru) + set_page_private(page, 0); + + /* + * Check for mergable pages for each order. + * + * For each order, check if their buddy is also in the list and + * if so, do merge, then remove the merged buddy from the list. + */ + for (order = 0; order < MAX_ORDER - 1; order++) { + bool has_merge = false; + + page = list_first_entry(list, struct page, lru); + while (&page->lru != list) { + unsigned long pfn, buddy_pfn, combined_pfn; + struct page *buddy, *n; + + if (page_order(page) != order) { + page = list_next_entry(page, lru); + continue; + } + + pfn = page_to_pfn(page); + buddy_pfn = __find_buddy_pfn(pfn, order); + buddy = page + (buddy_pfn - pfn); + if (!buddy_in_list(page, buddy, list) || + page_order(buddy) != order) { + page = list_next_entry(page, lru); + continue; + } + + combined_pfn = pfn & buddy_pfn; + if (combined_pfn == pfn) { + set_page_private(page, order + 1); + list_del(&buddy->lru); + page = list_next_entry(page, lru); + } else { + set_page_private(buddy, order + 1); + n = list_next_entry(page, lru); + list_del(&page->lru); + page = n; + } + has_merge = true; + } + + if (!has_merge) + break; + } +} + /* * Frees a number of pages from the PCP lists * Assumes all pages on list are in same zone, and of same order. @@ -1165,6 +1231,12 @@ static void free_pcppages_bulk(struct zone *zone, int count, } while (--count && --batch_free && !list_empty(list)); } + /* + * Before acquiring the possibly heavily contended zone lock, do merge + * among these to-be-freed PCP pages before sending them to Buddy. + */ + merge_in_pcp(&head); + read_lock(&zone->lock); isolated_pageblocks = has_isolate_pageblock(zone); @@ -1182,10 +1254,9 @@ static void free_pcppages_bulk(struct zone *zone, int count, if (unlikely(isolated_pageblocks)) mt = get_pageblock_migratetype(page); - order = 0; + order = page_order(page); merged_page = do_merge(page, page_to_pfn(page), zone, &order, mt); list_add(&merged_page->lru, this_cpu_ptr(&merge_lists[order][mt])); - trace_mm_page_pcpu_drain(page, 0, mt); } for_each_migratetype_order(order, migratetype) {