From patchwork Mon Nov 5 21:19:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10669199 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCF9715A6 for ; Mon, 5 Nov 2018 21:20:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D42FD29C39 for ; Mon, 5 Nov 2018 21:20:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C791829C74; Mon, 5 Nov 2018 21:20:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 438A129C5E for ; Mon, 5 Nov 2018 21:20:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97CC86B0273; Mon, 5 Nov 2018 16:19:58 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 92DF06B0274; Mon, 5 Nov 2018 16:19:58 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81D356B0275; Mon, 5 Nov 2018 16:19:58 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 3E2F06B0273 for ; Mon, 5 Nov 2018 16:19:58 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id s24-v6so11318622plp.12 for ; Mon, 05 Nov 2018 13:19:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=63FvyKJIYNFUk9XS9PpzHg15OpsRMDn7WoiTX6FY6bs=; b=ZjE/VWRcoMIa+CPBFsXsgozH0o7w0SKj+jFNcpngwajBbH8gT6Zrxj7L74eeUtFQPa grQwcKnMNejjdkYLhBnS7Ebmv6mryrkIKRBTp0c+Y88qNpHXlDuqZh6FsnaYrCrZ6knv mUkHFxokep1Lyo8YNmD/JuNzl3vtlVAoTM4nR2TERBNucZXPJdGya0Jya9wcu+7OUzHo on/MQDvsdGs07mG69jhb2hUrXxtGuxVU70/ToBTK/Rknxod1kp6RpHQpqXMOjgJLDjUs ccYnzFO9kJt1OkUuaIackGk3EUQ84gNxmxEUR9IT4uAcdqEFTLLuQYzO646cV4G0NQP7 k5/A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gI8niOkkeWZDBoVhpAo4ZDucgogz1b2jpijlxv6BtYqEkhw1D48 pyDtC+WE4uNbGAvBN/DMa7C/xb8+nXffyTmrQ9hbrM11OXlaIygG27rkwIpvpnemXkANeKDxWgf TSIMGgD48QsASB7Lkv2z3lHajovpW1guDDMfqkKbdhYMJKbI07+/0k8+CvAGaCvX9Bg== X-Received: by 2002:a17:902:5a2:: with SMTP id f31-v6mr23321804plf.320.1541452797883; Mon, 05 Nov 2018 13:19:57 -0800 (PST) X-Google-Smtp-Source: AJdET5fHZmtZDRYvPiN/dUeXZiUUueHY2LpoKmMawwS8iITD48EiSU7rXNHpGRPTQHICp0h/xcrp X-Received: by 2002:a17:902:5a2:: with SMTP id f31-v6mr23321755plf.320.1541452797033; Mon, 05 Nov 2018 13:19:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541452797; cv=none; d=google.com; s=arc-20160816; b=j5LWsPEZHMugDBdhyl9NGsBI2Yh5TRSeVbDskJd59eZrXyi9VEW6SzX9Yi5tKsuD9b PPpZ7js4V5cd94BhU3SU7DiQ9w9Fnt5Qqfcl+rl6H5RV3EASw7Gb4Ig7IUJmU7sT5dY4 kvqdnVet/zsiOi+lV14QVjupNpVyFvl9SN3F4nZj7vn0+JTVyIilhEal2vE8UYfVPG4/ jcRPXoUV1VhZ8Tr/jdE4T75G7YOtZKF/NwdkQxnJ3V8SGXU8ojoMRmRdWGqPm/WiYwAi 4Dv8rveDhmcW0pGGMeuQgoAfpGEsE4arVPsZNDjaol2uVEzOpvVVNgv1eiLx4REAC6XT dFAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=63FvyKJIYNFUk9XS9PpzHg15OpsRMDn7WoiTX6FY6bs=; b=a5QXhkyT+vhZ0di1D66uWf/ondpl2FaZ4iPupUGz+ut9kvOuyQZsxYat3746ROcwTP O+qoxT+ZWcCkaJuJek3YLzxFPXGrr2zt2J/w0Dn2O71BOm7sgLNVYGNR5aoP6mhnhRCj MHdTpwnbrzCshZIR+LkwIdQl56W3MwkaFpdQGWT90+BVbD9IKwGL9B9OH+3dp/vzgPCV PTEw9BgdIjsTCNovD4jDEkr6rK0LGhSn0NDZlwpDVLaaajMaLPlT1GSpQEYILOfVSoZx K29v4P6PG8t46hzdmUT+972n8jwywCqm4OyuBLCPfFXa4ui60j/FosZOaZHSBP29sgOf Z/RA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id 61-v6si14602011plb.125.2018.11.05.13.19.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Nov 2018 13:19:56 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 13:19:56 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,469,1534834800"; d="scan'208";a="247211874" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga004.jf.intel.com with ESMTP; 05 Nov 2018 13:19:56 -0800 Subject: [mm PATCH v5 6/7] mm: Add reserved flag setting to set_page_links From: Alexander Duyck To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, davem@davemloft.net, pavel.tatashin@microsoft.com, mhocko@suse.com, mingo@kernel.org, kirill.shutemov@linux.intel.com, dan.j.williams@intel.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, rppt@linux.vnet.ibm.com, willy@infradead.org, vbabka@suse.cz, khalid.aziz@oracle.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, yi.z.zhang@linux.intel.com, alexander.h.duyck@linux.intel.com Date: Mon, 05 Nov 2018 13:19:56 -0800 Message-ID: <154145279604.30046.5646399488589213615.stgit@ahduyck-desk1.jf.intel.com> In-Reply-To: <154145268025.30046.11742652345962594283.stgit@ahduyck-desk1.jf.intel.com> References: <154145268025.30046.11742652345962594283.stgit@ahduyck-desk1.jf.intel.com> User-Agent: StGit/unknown-version MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch modifies the set_page_links function to include the setting of the reserved flag via a simple AND and OR operation. The motivation for this is the fact that the existing __set_bit call still seems to have effects on performance as replacing the call with the AND and OR can reduce initialization time. Looking over the assembly code before and after the change the main difference between the two is that the reserved bit is stored in a value that is generated outside of the main initialization loop and is then written with the other flags field values in one write to the page->flags value. Previously the generated value was written and then then a btsq instruction was issued. On my x86_64 test system with 3TB of persistent memory per node I saw the persistent memory initialization time on average drop from 23.49s to 19.12s per node. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- include/linux/mm.h | 9 ++++++++- mm/page_alloc.c | 29 +++++++++++++++++++---------- 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 288c407c08fc..de6535a98e45 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1171,11 +1171,18 @@ static inline void set_page_node(struct page *page, unsigned long node) page->flags |= (node & NODES_MASK) << NODES_PGSHIFT; } +static inline void set_page_reserved(struct page *page, bool reserved) +{ + page->flags &= ~(1ul << PG_reserved); + page->flags |= (unsigned long)(!!reserved) << PG_reserved; +} + static inline void set_page_links(struct page *page, enum zone_type zone, - unsigned long node, unsigned long pfn) + unsigned long node, unsigned long pfn, bool reserved) { set_page_zone(page, zone); set_page_node(page, node); + set_page_reserved(page, reserved); #ifdef SECTION_IN_PAGE_FLAGS set_page_section(page, pfn_to_section_nr(pfn)); #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index dbe00c1a0e23..9eb993a9be99 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1179,7 +1179,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { mm_zero_struct_page(page); - set_page_links(page, zone, nid, pfn); + set_page_links(page, zone, nid, pfn, false); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); @@ -1195,7 +1195,8 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, static void __meminit __init_pageblock(unsigned long start_pfn, unsigned long nr_pages, unsigned long zone, int nid, - struct dev_pagemap *pgmap) + struct dev_pagemap *pgmap, + bool is_reserved) { unsigned long nr_pgmask = pageblock_nr_pages - 1; struct page *start_page = pfn_to_page(start_pfn); @@ -1231,18 +1232,15 @@ static void __meminit __init_pageblock(unsigned long start_pfn, * call because of the fact that the pfn number is used to * get the section_nr and this function should not be * spanning more than a single section. + * + * We can use a non-atomic operation for setting the + * PG_reserved flag as we are still initializing the pages. */ - set_page_links(page, zone, nid, start_pfn); + set_page_links(page, zone, nid, start_pfn, is_reserved); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); - /* - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - /* * ZONE_DEVICE pages union ->lru with a ->pgmap back * pointer and hmm_data. It is a bug if a ZONE_DEVICE @@ -5612,7 +5610,18 @@ static void __meminit __memmap_init_hotplug(unsigned long size, int nid, pfn = max(ALIGN_DOWN(pfn - 1, pageblock_nr_pages), start_pfn); stride -= pfn; - __init_pageblock(pfn, stride, zone, nid, pgmap); + /* + * The last argument of __init_pageblock is a boolean + * value indicating if the page will be marked as reserved. + * + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * Under certain circumstances ZONE_DEVICE pages may not + * need to be marked as reserved, however there is still + * code that is depending on this being set for now. + */ + __init_pageblock(pfn, stride, zone, nid, pgmap, true); cond_resched(); }