From patchwork Fri Jan 3 21:16:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander H Duyck X-Patchwork-Id: 11317425 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A31E1395 for ; Fri, 3 Jan 2020 21:17:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CBCEB2253D for ; Fri, 3 Jan 2020 21:17:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="B6iIFQrM" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBCEB2253D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DF36E8E000C; Fri, 3 Jan 2020 16:17:00 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DA35A8E0003; Fri, 3 Jan 2020 16:17:00 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDFDB8E000C; Fri, 3 Jan 2020 16:17:00 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id B72128E0003 for ; Fri, 3 Jan 2020 16:17:00 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7050A1EFD for ; Fri, 3 Jan 2020 21:17:00 +0000 (UTC) X-FDA: 76337583000.01.scarf00_200033ebe701f X-Spam-Summary: 2,0,0,72f72d9ec8c6f11a,d41d8cd98f00b204,alexander.duyck@gmail.com,:kvm@vger.kernel.org:mst@redhat.com:linux-kernel@vger.kernel.org:willy@infradead.org:mhocko@kernel.org::akpm@linux-foundation.org:mgorman@techsingularity.net:vbabka@suse.cz:yang.zhang.wz@gmail.com:nitesh@redhat.com:konrad.wilk@oracle.com:david@redhat.com:pagupta@redhat.com:riel@surriel.com:lcapitulino@redhat.com:dave.hansen@intel.com:wei.w.wang@intel.com:aarcange@redhat.com:pbonzini@redhat.com:dan.j.williams@intel.com:alexander.h.duyck@linux.intel.com:osalvador@suse.de,RULES_HIT:41:152:355:379:960:965:966:973:981:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:2731:2898:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4037:4117:4321:4385:4390:4395:4401:4605:5007:6119:6261:6653:6742:7576:7903:8957:9010:9012:9121:9413:10004:10128:10400:11026:11658:11914:12043:12048:12291:12 295:1229 X-HE-Tag: scarf00_200033ebe701f X-Filterd-Recvd-Size: 6191 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Fri, 3 Jan 2020 21:16:59 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id 6so23941246pgk.0 for ; Fri, 03 Jan 2020 13:16:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=q6OeJXPwWEnUOz8Lo8TsCrdbcxtEJkAXopv2X8QuM5w=; b=B6iIFQrM8v3I3ajt/BcBzw2sj/0ZMyP4nYVWUfntvKkPYDY0R9APeMJ1WpZRZGkN6N d+IIO3MQwmbBe2pjirF4XfYePTo0LQeSH3a3YHzE//DsxP+V0ySI0l+mcua2uFJxoTad El7TOmniUKXJDw9fBgQ4YbfFAX3cigl3dVx13vGcxo6uhVA3E7Xk4ff2+VsKvCM7j5Rp Y0bh0qkAowgldLhmOpzQjdfWY1UeqnVp201hqLwnJWbVv5TSmHVHHwKhVcCCXHR18tbQ sUTvjofReqQ+o6jfRki1xNMw1dL5XmKL7JWMViVGNibXaZ1w9L/ARxvx2ObyqP8Efdux JyCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=q6OeJXPwWEnUOz8Lo8TsCrdbcxtEJkAXopv2X8QuM5w=; b=spbP7a1r3ZBQ9L+KNZua+kkQcvSdaexXRm9FnLfmvNcCwPql2ngnyB2OS1+24h/jM5 c6n7N3jyK9KYJ0LuawbrGqM4tmSI5vGE7f0FHnSA36h2tX2gN/d3siG1uKnud3poDtjJ n9c8ZhDlagi9Ke5/EHvMLuz2reXxMTdYsoj3TqwkXfLB9QxMQdNrXW1jtgSmEYz8rCkH 5DAV+Lqv6VUBjPj9rvt/Wrqlnwjb4U68fvrpnj/eI/ET3PW88PtmmlF2v7AMRrZI2qNI QbKxaLulj1vr/jXnop+GODG0mIyetNLM1VbBhphtqgEqMnhvpcK2JBwjtY1476OqI8DA ddCQ== X-Gm-Message-State: APjAAAWvopBkjleCQtZ9Vf9CF0AKYviuRCEyH85RP2nMYI78niuuZlNA 11NSnzDHqs84dWQUoYvHw4g= X-Google-Smtp-Source: APXvYqzgh5Xrxh62OI8wBacWpgPuHGMSycLz0VPkIcffysY9nks3kpviBjQt/XeAYm7ykj3kfI4RDA== X-Received: by 2002:a62:2cc1:: with SMTP id s184mr97912791pfs.111.1578086218911; Fri, 03 Jan 2020 13:16:58 -0800 (PST) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id a28sm68177056pfh.119.2020.01.03.13.16.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 03 Jan 2020 13:16:58 -0800 (PST) Subject: [PATCH v16 7/9] mm: Rotate free list so reported pages are moved to the tail of the list From: Alexander Duyck To: kvm@vger.kernel.org, mst@redhat.com, linux-kernel@vger.kernel.org, willy@infradead.org, mhocko@kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, mgorman@techsingularity.net, vbabka@suse.cz Cc: yang.zhang.wz@gmail.com, nitesh@redhat.com, konrad.wilk@oracle.com, david@redhat.com, pagupta@redhat.com, riel@surriel.com, lcapitulino@redhat.com, dave.hansen@intel.com, wei.w.wang@intel.com, aarcange@redhat.com, pbonzini@redhat.com, dan.j.williams@intel.com, alexander.h.duyck@linux.intel.com, osalvador@suse.de Date: Fri, 03 Jan 2020 13:16:57 -0800 Message-ID: <20200103211657.29237.50194.stgit@localhost.localdomain> In-Reply-To: <20200103210509.29237.18426.stgit@localhost.localdomain> References: <20200103210509.29237.18426.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Rather than walking over the same pages again and again to get to the pages that have yet to be reported we can save ourselves a significant amount of time by simply rotating the list so that when we have a full list of reported pages the head of the list is pointing to the next non-reported page. Doing this should save us some significant time when processing each free list. This doesn't gain us much in the standard case as all of the non-reported pages should be near the top of the list already. However in the case of page shuffling this results in a noticeable improvement. Below are the will-it-scale page_fault1 w/ THP numbers for 16 tasks with and without this patch. Without: tasks processes processes_idle threads threads_idle 16 8093776.25 0.17 5393242.00 38.20 With: tasks processes processes_idle threads threads_idle 16 8283274.75 0.17 5594261.00 38.15 Signed-off-by: Alexander Duyck --- mm/page_reporting.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index 1047c6872d4f..6885e74c2367 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -131,17 +131,27 @@ void __page_reporting_notify(void) if (PageReported(page)) continue; - /* Attempt to pull page from list */ - if (!__isolate_free_page(page, order)) - break; + /* Attempt to pull page from list and place in scatterlist */ + if (*offset) { + if (!__isolate_free_page(page, order)) { + next = page; + break; + } - /* Add page to scatter list */ - --(*offset); - sg_set_page(&sgl[*offset], page, page_len, 0); + /* Add page to scatter list */ + --(*offset); + sg_set_page(&sgl[*offset], page, page_len, 0); - /* If scatterlist isn't full grab more pages */ - if (*offset) continue; + } + + /* + * Make the first non-processed page in the free list + * the new head of the free list before we release the + * zone lock. + */ + if (&page->lru != list && !list_is_first(&page->lru, list)) + list_rotate_to_front(&page->lru, list); /* release lock before waiting on report processing */ spin_unlock_irq(&zone->lock); @@ -169,6 +179,10 @@ void __page_reporting_notify(void) break; } + /* Rotate any leftover pages to the head of the freelist */ + if (&next->lru != list && !list_is_first(&next->lru, list)) + list_rotate_to_front(&next->lru, list); + spin_unlock_irq(&zone->lock); return err;