From patchwork Fri Nov 30 21:53:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10707267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E27313A4 for ; Fri, 30 Nov 2018 21:53:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F88330627 for ; Fri, 30 Nov 2018 21:53:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 12E493062B; Fri, 30 Nov 2018 21:53:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8117930627 for ; Fri, 30 Nov 2018 21:53:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69B406B5A77; Fri, 30 Nov 2018 16:53:22 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 64A596B5A78; Fri, 30 Nov 2018 16:53:22 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 539F86B5A79; Fri, 30 Nov 2018 16:53:22 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 0FDE36B5A77 for ; Fri, 30 Nov 2018 16:53:22 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id p9so5519068pfj.3 for ; Fri, 30 Nov 2018 13:53:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=bpl5o0AF0Jqp93Btqfi2CQDNS4Bd8fSa0w8Rb9LrKRY=; b=NoHoqVVF1soby0pPFA/D94kn8DtC9hKMrpvB+0OSiCLz8cv5qzy7YVus8+oQqU4Hw8 QXCWDUL9Rxst7paCiB7E5jQrv14KnTSkkOq16fQHLloJPsuLWhjXXP4C/GSOGWfzdfsF klC0GGhGaaKyG2OuWKgGaKmkWT1bL8VGR6cBQeSZLdDXOB+cn9vS3ucfda1ULMO2wJbI p+EXlPwYdBcq5bWZZKL7mzxzwZ6lN4xX9wYXIAIIlS0tLha+3MwO1Vuqean7JdI/TyAN 1hNzGMEEJGCVCwJi2Al9BnzVNEC+VIRfLuHEwFhS/kdK31KMGCzY0uXu9AP2lqEnSL3Z LdBA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWbuhXb9QD0jyxuKC7wUcodpgiueEo2t5djipMoxdUyi9WR0L7Q7 spid3cEz/nNgI3EL6P0loRBklM2z6C81UkGZOkZKqkqiG5WTSxgCR7mT7aXElRTEYPBRswbiL4+ clW0NPRs/BssO3htUwBCtiRnJG2lhERDKGa2znbW8x7dlthyhMsdXX2zNfmKg7E0jow== X-Received: by 2002:a63:ce50:: with SMTP id r16mr6180068pgi.217.1543614801707; Fri, 30 Nov 2018 13:53:21 -0800 (PST) X-Google-Smtp-Source: AFSGD/X4N+o9DFnbN4fozny6NdoaMeircTKRAZDFcCDRQd0d4L/4wL2Q2Jrqzv69XYqsUbmr/Qef X-Received: by 2002:a63:ce50:: with SMTP id r16mr6180031pgi.217.1543614800772; Fri, 30 Nov 2018 13:53:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543614800; cv=none; d=google.com; s=arc-20160816; b=zBqlD5UUCAj34sH/yEoSFXXDs18ZPC0PAvUZ1yLghH3w8norT2dWKxCzEvlLTYhjoG Gy8qWtYjJLytX8q7FazLOXclrzZ/Tae59oSleqGCloGpKHVDcHeY4eMaDb/Y++v9XOPC gWRiLDtHJkcbFQim7r3yA51ExhsqrUnjZ85DPyVUIIQoqMYp7/V+qLWbWMWDlDHzo821 9uY5jpmIAZZGOso884d6WIBteYjI2jA7NNW1+cB5QMxhdrDefwOLl94l6xGU3+Rt8NCS JZwiONbZszxKtelUKLnRTtkpEY7FL/Sk8yrSXXrNRCBk4slUZloK7RdtNJCXtwE6UXaP 0DSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=bpl5o0AF0Jqp93Btqfi2CQDNS4Bd8fSa0w8Rb9LrKRY=; b=oqvje7OAtKBshqe+d+4FpTZbpmqAKW/Sz5kC2GWo3U4RECCxJHgfEAXpORaS+i4Tf4 0w5PfZ2JwSSqgl4DfmQ2AxFSBMizkGnR4Cq95VnlA+tN23MQbFoZUCpnvXkahLW4nbVW r58ETg55CAKJREp2D+GL6OzRzRT7hF+Sva5ZBZllsMfNIdWeLxWCgqPkP+6XjXYM68kb AXYE5nXQL2b6oVPxKnAzXwf+CjelJeC1M5sA2npbtJICuiwpdZEtfAn8v3pZEHT4A3og mnYrVMwUJvCm25lDHFrEoY9lH0vJRr6vR2xLPdKmz+jR0fGPtM/uUo9pemH6Tw7YVmSe 5oAw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga14.intel.com (mga14.intel.com. [192.55.52.115]) by mx.google.com with ESMTPS id f38si5003973pgf.206.2018.11.30.13.53.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Nov 2018 13:53:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) client-ip=192.55.52.115; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.115 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Nov 2018 13:53:20 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,300,1539673200"; d="scan'208";a="94786691" Received: from ahduyck-desk1.jf.intel.com (HELO ahduyck-desk1.amr.corp.intel.com) ([10.7.198.76]) by orsmga007.jf.intel.com with ESMTP; 30 Nov 2018 13:53:18 -0800 Subject: [mm PATCH v6 6/7] mm: Add reserved flag setting to set_page_links From: Alexander Duyck To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, davem@davemloft.net, pavel.tatashin@microsoft.com, mhocko@suse.com, mingo@kernel.org, kirill.shutemov@linux.intel.com, dan.j.williams@intel.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, rppt@linux.vnet.ibm.com, willy@infradead.org, vbabka@suse.cz, khalid.aziz@oracle.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, yi.z.zhang@linux.intel.com, alexander.h.duyck@linux.intel.com Date: Fri, 30 Nov 2018 13:53:18 -0800 Message-ID: <154361479877.7497.2824031260670152276.stgit@ahduyck-desk1.amr.corp.intel.com> In-Reply-To: <154361452447.7497.1348692079883153517.stgit@ahduyck-desk1.amr.corp.intel.com> References: <154361452447.7497.1348692079883153517.stgit@ahduyck-desk1.amr.corp.intel.com> User-Agent: StGit/unknown-version MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Modify the set_page_links function to include the setting of the reserved flag via a simple AND and OR operation. The motivation for this is the fact that the existing __set_bit call still seems to have effects on performance as replacing the call with the AND and OR can reduce initialization time. Looking over the assembly code before and after the change the main difference between the two is that the reserved bit is stored in a value that is generated outside of the main initialization loop and is then written with the other flags field values in one write to the page->flags value. Previously the generated value was written and then then a btsq instruction was issued. On my x86_64 test system with 3TB of persistent memory per node I saw the persistent memory initialization time on average drop from 23.49s to 19.12s per node. Reviewed-by: Pavel Tatashin Signed-off-by: Alexander Duyck --- include/linux/mm.h | 9 ++++++++- mm/page_alloc.c | 39 +++++++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index eb6e52b66bc2..5faf66dd4559 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1238,11 +1238,18 @@ static inline void set_page_node(struct page *page, unsigned long node) page->flags |= (node & NODES_MASK) << NODES_PGSHIFT; } +static inline void set_page_reserved(struct page *page, bool reserved) +{ + page->flags &= ~(1ul << PG_reserved); + page->flags |= (unsigned long)(!!reserved) << PG_reserved; +} + static inline void set_page_links(struct page *page, enum zone_type zone, - unsigned long node, unsigned long pfn) + unsigned long node, unsigned long pfn, bool reserved) { set_page_zone(page, zone); set_page_node(page, node); + set_page_reserved(page, reserved); #ifdef SECTION_IN_PAGE_FLAGS set_page_section(page, pfn_to_section_nr(pfn)); #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 416bbb6f05ab..61eb9945d805 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1183,10 +1183,16 @@ static void free_one_page(struct zone *zone, static void __meminit __init_struct_page_nolru(struct page *page, unsigned long pfn, - unsigned long zone, int nid) + unsigned long zone, int nid, + bool is_reserved) { mm_zero_struct_page(page); - set_page_links(page, zone, nid, pfn); + + /* + * We can use a non-atomic operation for setting the + * PG_reserved flag as we are still initializing the pages. + */ + set_page_links(page, zone, nid, pfn, is_reserved); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); @@ -1202,14 +1208,15 @@ static void __meminit __init_struct_page_nolru(struct page *page, static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { - __init_struct_page_nolru(page, pfn, zone, nid); + __init_struct_page_nolru(page, pfn, zone, nid, false); INIT_LIST_HEAD(&page->lru); } static void __meminit __init_pageblock(unsigned long start_pfn, unsigned long nr_pages, unsigned long zone, int nid, - struct dev_pagemap *pgmap) + struct dev_pagemap *pgmap, + bool is_reserved) { unsigned long nr_pgmask = pageblock_nr_pages - 1; struct page *start_page = pfn_to_page(start_pfn); @@ -1235,15 +1242,8 @@ static void __meminit __init_pageblock(unsigned long start_pfn, * is not defined. */ for (page = start_page + nr_pages; page-- != start_page; pfn--) { - __init_struct_page_nolru(page, pfn, zone, nid); - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); + __init_struct_page_nolru(page, pfn, zone, nid, is_reserved); + /* * ZONE_DEVICE pages union ->lru with a ->pgmap back * pointer and hmm_data. It is a bug if a ZONE_DEVICE @@ -5780,7 +5780,18 @@ static void __meminit __memmap_init_hotplug(unsigned long size, int nid, pfn = max(ALIGN_DOWN(pfn - 1, pageblock_nr_pages), start_pfn); stride -= pfn; - __init_pageblock(pfn, stride, zone, nid, pgmap); + /* + * The last argument of __init_pageblock is a boolean + * value indicating if the page will be marked as reserved. + * + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * Under certain circumstances ZONE_DEVICE pages may not + * need to be marked as reserved, however there is still + * code that is depending on this being set for now. + */ + __init_pageblock(pfn, stride, zone, nid, pgmap, true); cond_resched(); }