From patchwork Wed Oct 17 06:33:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Lu X-Patchwork-Id: 10644633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0D55112B for ; Wed, 17 Oct 2018 06:33:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C08E72A76D for ; Wed, 17 Oct 2018 06:33:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B4C402A874; Wed, 17 Oct 2018 06:33:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 426332A76D for ; Wed, 17 Oct 2018 06:33:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD7796B000D; Wed, 17 Oct 2018 02:33:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B5D1F6B000E; Wed, 17 Oct 2018 02:33:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B0E06B0010; Wed, 17 Oct 2018 02:33:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 514586B000D for ; Wed, 17 Oct 2018 02:33:47 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id d69-v6so19012576pgc.22 for ; Tue, 16 Oct 2018 23:33:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=UEWF/oMfwuMtnniRCkmBf5lutGOCwkrA40ose+iQuNY=; b=CL8+lLF4iGT6NGokNJzOSFIakhyvxeTX/ON8d9x9VqbT72ebDtuehDu7ztORKtvMW2 UyMPvx5GTSYjhDzE7VJSnFT/wddSOQ+lzSmiBclZQQugJXVwnkXjJu5aTbyCbDCU7XIp +p6RJQ4ME+RLEUQ1re8e3n7iuF+2xgyeD1d2vRaVGYoulkhHf03xf0wShsP2vBMEldph Px6qjJ0L5Z05XEt8IAs9QU06bX3CeL5zF88s5zqy/pnZS/M9HpCFGiRlhEDT4KcrMMdU 7wJn/uRApgzHtgqbKRBTlUfC5VBEiYEzGarUJN3npP6G7Rut6b6d3T7DyqEvQOMrfEi9 H3Tw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohII7eG8+D0OStEX8OChVsdpAJ5IdJ+tUCZjMfvvoqonq7og9pf RMtReAQyFYi/7sniXYQlfO8gE7tGNrO2omST9ebTICUcGgrvxBLOIlDwFVo+bSQv4jcDyZpV9Tf 6HD9ihacvJEehYArvRZguW+Ji90zakkdHYNtse0zFHf4i/XpoNCfq1pBRAntFY364xQ== X-Received: by 2002:a65:4783:: with SMTP id e3-v6mr22871830pgs.12.1539758027017; Tue, 16 Oct 2018 23:33:47 -0700 (PDT) X-Google-Smtp-Source: ACcGV61paSl0vAYCE5yKz28JE7ewP4zumBjtlxcMTdMzP/D5jI1MOVs2KpvsYAkakUa6tBl1LrEa X-Received: by 2002:a65:4783:: with SMTP id e3-v6mr22871800pgs.12.1539758026330; Tue, 16 Oct 2018 23:33:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539758026; cv=none; d=google.com; s=arc-20160816; b=FekpC11VKEC5xigfKeOCCkWC9km0B09n40AsHAxEyE/3M17CboZqp2V5+L7i3zzBul Oc5mvScFXuck8jMzNPSjPOlLG3UWSXJp8hwXUyXJ9HG0K2ZzFdzIyoJw+rNwMRIlY35J KanGhiXZtR5qt1SkfUQYU4FVzpsJxV0RsgxXFxMMUx4gdR0RWRYHE0acbxBcA/x1dfsF IIFsG64Ad2iIP2Ud9TpgTGCGCPgT6V3uHq5RNhrvzbf72O4Pz/ud5i0ixhR8VMEcJmZU QJUlBqJJtZS1VGmjTGP+FAYB3pYy5HBzodWoc0HXplNCo+TwpCAftXh3s9xAMurHNFBY Q2cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=UEWF/oMfwuMtnniRCkmBf5lutGOCwkrA40ose+iQuNY=; b=c85QrJK099LxzMHBQGmqIAjGIexl5MMASgzgHHB8WigPBeLtV3vgzWs+p5m8Tkdoul gX5zHgPtMsufGPNxlIhf3VptAQk/N1vwCnDgdFAqOdJDKxfPJtJY0LQOduQMI2h/OWqH U5z7YR65MTyGgmGAb4urR2Hq5c58hI3tbL1FDkhq+r+krccQhL6QrYHEbXJiUCGd1YgR X449IZsdqW9JSUjfN2ffYw5jySOyFY1INpcS6dlgtCzzRejvddRHgPyzlK0ijOf/G7sP HF85DzkE07xPJ9NXZqvsm4Xn4hf4BKjgLrd8QIdHY1SiV4cCmqVnGIL8MYqnOqlKJtU8 bx0Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id v202-v6si1081311pgb.96.2018.10.16.23.33.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Oct 2018 23:33:46 -0700 (PDT) Received-SPF: pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of aaron.lu@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=aaron.lu@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Oct 2018 23:33:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,391,1534834800"; d="scan'208";a="273090123" Received: from aaronlu.sh.intel.com ([10.239.159.44]) by fmsmga006.fm.intel.com with ESMTP; 16 Oct 2018 23:33:43 -0700 From: Aaron Lu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrew Morton , Huang Ying , Dave Hansen , Kemi Wang , Tim Chen , Andi Kleen , Michal Hocko , Vlastimil Babka , Mel Gorman , Matthew Wilcox , Daniel Jordan , Tariq Toukan , Jesper Dangaard Brouer Subject: [RFC v4 PATCH 4/5] mm/free_pcppages_bulk: reduce overhead of cluster operation on free path Date: Wed, 17 Oct 2018 14:33:29 +0800 Message-Id: <20181017063330.15384-5-aaron.lu@intel.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20181017063330.15384-1-aaron.lu@intel.com> References: <20181017063330.15384-1-aaron.lu@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP After "no_merge for order 0", the biggest overhead in free path for order 0 pages is now add_to_cluster(). As pages are freed one by one, it caused frequent operation of add_to_cluster(). Ideally, if only one migratetype pcp list has pages to free and count=pcp->batch in free_pcppages_bulk(), we can avoid calling add_to_cluster() one time per page but adding them in one go as a single cluster so this patch just did this. This optimization brings zone->lock contention down from 25% to almost zero again using the parallel free workload. Signed-off-by: Aaron Lu --- mm/page_alloc.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e60a248030dc..204696f6c2f4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1242,6 +1242,36 @@ static inline void prefetch_buddy(struct page *page) prefetch(buddy); } +static inline bool free_cluster_pages(struct zone *zone, struct list_head *list, + int mt, int count) +{ + struct cluster *c; + struct page *page, *n; + + if (!can_skip_merge(zone, 0)) + return false; + + if (count != this_cpu_ptr(zone->pageset)->pcp.batch) + return false; + + c = new_cluster(zone, count, list_first_entry(list, struct page, lru)); + if (unlikely(!c)) + return false; + + list_for_each_entry_safe(page, n, list, lru) { + set_page_order(page, 0); + set_page_merge_skipped(page); + page->cluster = c; + list_add(&page->lru, &zone->free_area[0].free_list[mt]); + } + + INIT_LIST_HEAD(list); + zone->free_area[0].nr_free += count; + __mod_zone_page_state(zone, NR_FREE_PAGES, count); + + return true; +} + /* * Frees a number of pages from the PCP lists * Assumes all pages on list are in same zone, and of same order. @@ -1256,10 +1286,10 @@ static inline void prefetch_buddy(struct page *page) static void free_pcppages_bulk(struct zone *zone, int count, struct per_cpu_pages *pcp) { - int migratetype = 0; - int batch_free = 0; + int migratetype = 0, i, count_mt[MIGRATE_PCPTYPES] = {0}; + int batch_free = 0, saved_count = count; int prefetch_nr = 0; - bool isolated_pageblocks; + bool isolated_pageblocks, single_mt = false; struct page *page, *tmp; LIST_HEAD(head); @@ -1283,6 +1313,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, /* This is the only non-empty list. Free them all. */ if (batch_free == MIGRATE_PCPTYPES) batch_free = count; + count_mt[migratetype] += batch_free; do { page = list_last_entry(list, struct page, lru); @@ -1314,12 +1345,24 @@ static void free_pcppages_bulk(struct zone *zone, int count, } while (--count && --batch_free && !list_empty(list)); } + for (i = 0; i < MIGRATE_PCPTYPES; i++) { + if (count_mt[i] == saved_count) { + single_mt = true; + break; + } + } + spin_lock(&zone->lock); isolated_pageblocks = has_isolate_pageblock(zone); + if (!isolated_pageblocks && single_mt) + free_cluster_pages(zone, &head, migratetype, saved_count); + /* * Use safe version since after __free_one_page(), * page->lru.next will not point to original list. + * + * If free_cluster_pages() succeeds, head will be an empty list here. */ list_for_each_entry_safe(page, tmp, &head, lru) { int mt = get_pcppage_migratetype(page);