From patchwork Wed Jan 22 17:43:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11346189 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78AA6921 for ; Wed, 22 Jan 2020 17:43:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3B3482467B for ; Wed, 22 Jan 2020 17:43:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="frFhI/01" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3B3482467B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 648DF6B026C; Wed, 22 Jan 2020 12:43:56 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F8886B026D; Wed, 22 Jan 2020 12:43:56 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5358E6B026E; Wed, 22 Jan 2020 12:43:56 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 3B6726B026C for ; Wed, 22 Jan 2020 12:43:56 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id CED84181AC9CC for ; Wed, 22 Jan 2020 17:43:55 +0000 (UTC) X-FDA: 76405993230.19.side00_5225162fd847 X-Spam-Summary: 2,0,0,72f72d9ec8c6f11a,d41d8cd98f00b204,alexander.duyck@gmail.com,:kvm@vger.kernel.org:mst@redhat.com:linux-kernel@vger.kernel.org:willy@infradead.org:mhocko@kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:vbabka@suse.cz:yang.zhang.wz@gmail.com:nitesh@redhat.com:konrad.wilk@oracle.com:david@redhat.com:pagupta@redhat.com:riel@surriel.com:lcapitulino@redhat.com:dave.hansen@intel.com:wei.w.wang@intel.com:aarcange@redhat.com:pbonzini@redhat.com:dan.j.williams@intel.com:alexander.h.duyck@linux.intel.com:osalvador@suse.de,RULES_HIT:41:152:355:379:960:966:973:981:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:2731:2898:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4037:4117:4321:4385:4401:4605:5007:6119:6261:6653:6742:7576:7903:8957:9010:9012:9121:9413:10004:10128:10400:11026:11658:11914:12043:12048:12291:12295:12297:1243 8:12517: X-HE-Tag: side00_5225162fd847 X-Filterd-Recvd-Size: 6182 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Wed, 22 Jan 2020 17:43:55 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id i23so192331pfo.2 for ; Wed, 22 Jan 2020 09:43:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=q6OeJXPwWEnUOz8Lo8TsCrdbcxtEJkAXopv2X8QuM5w=; b=frFhI/01UG5hBEhRBNTRsQ8gQ7uui7kbxicIpPPmSOJO9jKyybhVB4WGX7s5C+DLHu Kj3G/zYYUiY/xG/esphxYWD/NWnswj7tpeIx1vZoaf81IaQJNhAxHfe5J87P15Ym9qJn uFoQUNGC5cjE8lxFiDYgyIsRO8LkKbA0ouNEU7cSPcM3gH5TGDCx/vBDPxTcOANC6Pa5 Z90aLXZh8QXXtojTzimMsh21X7hI1QW2bO8AC1RarzJipn8h+O+JEIDpBdB8vICJQQRf KLT5VtRdYJG2pPV2QWIvE6otc+2B9JkLdDnHHC0WajU8i2w2qonU2MCHlxFW6xAPzjhi fuQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=q6OeJXPwWEnUOz8Lo8TsCrdbcxtEJkAXopv2X8QuM5w=; b=Zy9O8XUOUabdIb7tzpb2rNhv3YFxdTrSnmLN93yAf8LlDnvQp5MYBmc2437Ukryyhd JjUeJWUsR835RMfQbfAfzZaESypTlHP6UFwMP8VzAdHDzOYujEvhyNXyOPkUNVspTf9J qyOVxu+PPWNY8/HBNcb3xUKcj+EkDyqfrRiQyR/ck9DvTcZm/eFaA3BeKO4p2aPZLU+s kWx8G4pq08YljFDYsNAFzdk9mSGCtizPIR7L/BXiPjBrGN2shu6JWg1Tvy6tIgI0mksu fpWlCip2poiGSFHW5rndMiCHvJJO2Yd7RCITEp58BVFHQ+BYbwEXeu3Vqtm8LLtijDMC WhWw== X-Gm-Message-State: APjAAAUylnJgTp32UfR/w4kB/O9xAnz6sxlAiv97wbSiGZt5stLo1Zxo JOU0Li3FsqYEeQpecluQSDw= X-Google-Smtp-Source: APXvYqyH85Ir7jeF2L2PGXga7oXOxOc4t10gnYRlYbAXz1IVn0jy4XgCW6Pn8gVQFzUlBQ4rw/q4Tw== X-Received: by 2002:a62:33c6:: with SMTP id z189mr3589982pfz.246.1579715034307; Wed, 22 Jan 2020 09:43:54 -0800 (PST) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id j125sm47173382pfg.160.2020.01.22.09.43.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Jan 2020 09:43:53 -0800 (PST) Subject: [PATCH v16.1 7/9] mm/page_reporting: Rotate reported pages to the tail of the list From: Alexander Duyck To: kvm@vger.kernel.org, mst@redhat.com, linux-kernel@vger.kernel.org, willy@infradead.org, mhocko@kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, mgorman@techsingularity.net, vbabka@suse.cz Cc: yang.zhang.wz@gmail.com, nitesh@redhat.com, konrad.wilk@oracle.com, david@redhat.com, pagupta@redhat.com, riel@surriel.com, lcapitulino@redhat.com, dave.hansen@intel.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com, osalvador@suse.de Date: Wed, 22 Jan 2020 09:43:53 -0800 Message-ID: <20200122174353.6142.56871.stgit@localhost.localdomain> In-Reply-To: <20200122173040.6142.39116.stgit@localhost.localdomain> References: <20200122173040.6142.39116.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Rather than walking over the same pages again and again to get to the pages that have yet to be reported we can save ourselves a significant amount of time by simply rotating the list so that when we have a full list of reported pages the head of the list is pointing to the next non-reported page. Doing this should save us some significant time when processing each free list. This doesn't gain us much in the standard case as all of the non-reported pages should be near the top of the list already. However in the case of page shuffling this results in a noticeable improvement. Below are the will-it-scale page_fault1 w/ THP numbers for 16 tasks with and without this patch. Without: tasks processes processes_idle threads threads_idle 16 8093776.25 0.17 5393242.00 38.20 With: tasks processes processes_idle threads threads_idle 16 8283274.75 0.17 5594261.00 38.15 Signed-off-by: Alexander Duyck --- mm/page_reporting.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index 1047c6872d4f..6885e74c2367 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -131,17 +131,27 @@ void __page_reporting_notify(void) if (PageReported(page)) continue; - /* Attempt to pull page from list */ - if (!__isolate_free_page(page, order)) - break; + /* Attempt to pull page from list and place in scatterlist */ + if (*offset) { + if (!__isolate_free_page(page, order)) { + next = page; + break; + } - /* Add page to scatter list */ - --(*offset); - sg_set_page(&sgl[*offset], page, page_len, 0); + /* Add page to scatter list */ + --(*offset); + sg_set_page(&sgl[*offset], page, page_len, 0); - /* If scatterlist isn't full grab more pages */ - if (*offset) continue; + } + + /* + * Make the first non-processed page in the free list + * the new head of the free list before we release the + * zone lock. + */ + if (&page->lru != list && !list_is_first(&page->lru, list)) + list_rotate_to_front(&page->lru, list); /* release lock before waiting on report processing */ spin_unlock_irq(&zone->lock); @@ -169,6 +179,10 @@ void __page_reporting_notify(void) break; } + /* Rotate any leftover pages to the head of the freelist */ + if (&next->lru != list && !list_is_first(&next->lru, list)) + list_rotate_to_front(&next->lru, list); + spin_unlock_irq(&zone->lock); return err;