From patchwork Wed Oct 17 23:54:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10646621 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9305613A4 for ; Wed, 17 Oct 2018 23:54:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 82641285A2 for ; Wed, 17 Oct 2018 23:54:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 768F9285C7; Wed, 17 Oct 2018 23:54:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D780285CE for ; Wed, 17 Oct 2018 23:54:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 481FD6B0279; Wed, 17 Oct 2018 19:54:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 40A866B027B; Wed, 17 Oct 2018 19:54:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A88C6B027C; Wed, 17 Oct 2018 19:54:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id EA7A16B0279 for ; Wed, 17 Oct 2018 19:54:33 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id c123-v6so11122638ywf.9 for ; Wed, 17 Oct 2018 16:54:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=12zZk11cx7CQ1HxVNZBiBumXtMxabIabSrImjcrcurI=; b=QYKjYB827e0jmhUTmUg3qytHwMDPP8HH7Qv+iYudewID2vYefjc9/eX3/eg/iHIkBo hyGIm3T50mB2m72IRNnuWKC6nFtC67tdBwnggvCso6QjcVwC1IZvq9zQomGEtMzxor0k XEkqpnrhL98nNqMdKnAMpig2UDZooxbvx6Dot6KXKRR8qgSA1kcYM63QtvtMSU035uaF NbIIKqfEmNH/N/ThsY9Xc1XpM+lCYYc7p80SR7cA9XEIko2j9drf19zTT8TfYt/pzVQ5 8q6eq/HkwwuASvLfR98jHbbq/SKxuJSoMfU2iVBo5giWvy87GJos21Q7kbyDZeqm+kKg Bkrw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoh1ZZoXvCl4haOQh4T7sTSAqCWucepopzSWVuiF52mnqHM4YIY5 qxLUgcOzeMWcaWQlLhB1PGA8QC+Gz/lHwhqxXMdxpfYUtDPHRsRaxQ32axN2TfD5KiuVgxcMnQq pWJpp5MMAjNCBkR9ZZL209z/bjITZrGau3U1WaiJnvvOVr8wCgV7725ieW24BpOLi2A== X-Received: by 2002:a81:b2c9:: with SMTP id q192-v6mr16565496ywh.223.1539820473655; Wed, 17 Oct 2018 16:54:33 -0700 (PDT) X-Google-Smtp-Source: ACcGV62U9ZXJiqpNpQhzsUUTMEsB4EMdkcQMdzxyBPmB+h2VIdRQdDfe6w0r8dSWAGpOzl42Yh98 X-Received: by 2002:a81:b2c9:: with SMTP id q192-v6mr16565475ywh.223.1539820472898; Wed, 17 Oct 2018 16:54:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539820472; cv=none; d=google.com; s=arc-20160816; b=Y8J6TLZ8rGx0MIJvjBY3u59+GMLM/iX4NgnBI5AdmyqYvVf0KI+HLFg7uGj9s3axxi TXutjn5VUQJIMYqxKQQdHabn2t0IIXL0bU3shsy6ZFqCrkozyKUn+jFQ5SZpwcU9V9Jz vge3i3bfX/k3PRJNfRtkfwOmL3GNO3dhnTz9l/nHJvMOGYCqrrupddwFsZJrx/+26ZSZ /f96LtMcRxzkyX1JSxI2SjOLR3kwwh25NsEsb94SBACgFxYgSFQ+j5onvMD93/w+kYot 0nqEEhk14MSJndzFEuHHNA4zcI9G5dqKNRe4qcm56a3v+dnNtmpRHdN51bZJxwGQ1JdD 2M9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=12zZk11cx7CQ1HxVNZBiBumXtMxabIabSrImjcrcurI=; b=vwehc7jKtEBVVmUqGf3R1Cpuxex60lHWy2GOU4j8dt08ENhi8356uQd+eSYNHt41Qy Fl11RXYTV+BSClFlxK5Xqq16hIXdpgauVgVyaqQXLMVLMkXetxOnEPIPbuwSTQHNycQK MsJXy4CSTJnoJ+TDs+SIM1pk017IkbnFETEAtNodfbhx2IEKdxQ8D00FdNhCwK2b/ojR Kvn5SZ5VhmIQ7jPP3pKg6Y/fERGz7fxP9mTsxR8fJz6S7gEQ1AKTvCGn+Uv7nGzoU0IX YlEOwvsJsiqarZpZmNtanpm5ZT/g6u1sGQXDzD6L9hjLb5T3l9Nk7MA/Qj/Vn5ocmICa vvuQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id a142-v6si6680026ywe.179.2018.10.17.16.54.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Oct 2018 16:54:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Oct 2018 16:54:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,393,1534834800"; d="scan'208";a="100366643" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.154]) by orsmga001.jf.intel.com with ESMTP; 17 Oct 2018 16:54:31 -0700 Subject: [mm PATCH v4 5/6] mm: Add reserved flag setting to set_page_links From: Alexander Duyck To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, davem@davemloft.net, yi.z.zhang@linux.intel.com, khalid.aziz@oracle.com, rppt@linux.vnet.ibm.com, vbabka@suse.cz, sparclinux@vger.kernel.org, dan.j.williams@intel.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mingo@kernel.org, kirill.shutemov@linux.intel.com Date: Wed, 17 Oct 2018 16:54:31 -0700 Message-ID: <20181017235431.17213.11512.stgit@localhost.localdomain> In-Reply-To: <20181017235043.17213.92459.stgit@localhost.localdomain> References: <20181017235043.17213.92459.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch modifies the set_page_links function to include the setting of the reserved flag via a simple AND and OR operation. The motivation for this is the fact that the existing __set_bit call still seems to have effects on performance as replacing the call with the AND and OR can reduce initialization time. Looking over the assembly code before and after the change the main difference between the two is that the reserved bit is stored in a value that is generated outside of the main initialization loop and is then written with the other flags field values in one write to the page->flags value. Previously the generated value was written and then then a btsq instruction was issued. On my x86_64 test system with 3TB of persistent memory per node I saw the persistent memory initialization time on average drop from 23.49s to 19.12s per node. Signed-off-by: Alexander Duyck --- include/linux/mm.h | 9 ++++++++- mm/page_alloc.c | 29 +++++++++++++++++++---------- 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6e2c9631af05..14d06d7d2986 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1171,11 +1171,18 @@ static inline void set_page_node(struct page *page, unsigned long node) page->flags |= (node & NODES_MASK) << NODES_PGSHIFT; } +static inline void set_page_reserved(struct page *page, bool reserved) +{ + page->flags &= ~(1ul << PG_reserved); + page->flags |= (unsigned long)(!!reserved) << PG_reserved; +} + static inline void set_page_links(struct page *page, enum zone_type zone, - unsigned long node, unsigned long pfn) + unsigned long node, unsigned long pfn, bool reserved) { set_page_zone(page, zone); set_page_node(page, node); + set_page_reserved(page, reserved); #ifdef SECTION_IN_PAGE_FLAGS set_page_section(page, pfn_to_section_nr(pfn)); #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a0b81e0bef03..e7fee7a5f8a3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1179,7 +1179,7 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { mm_zero_struct_page(page); - set_page_links(page, zone, nid, pfn); + set_page_links(page, zone, nid, pfn, false); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); @@ -1195,7 +1195,8 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, static void __meminit __init_pageblock(unsigned long start_pfn, unsigned long nr_pages, unsigned long zone, int nid, - struct dev_pagemap *pgmap) + struct dev_pagemap *pgmap, + bool is_reserved) { unsigned long nr_pgmask = pageblock_nr_pages - 1; struct page *start_page = pfn_to_page(start_pfn); @@ -1231,19 +1232,16 @@ static void __meminit __init_pageblock(unsigned long start_pfn, * call because of the fact that the pfn number is used to * get the section_nr and this function should not be * spanning more than a single section. + * + * We can use a non-atomic operation for setting the + * PG_reserved flag as we are still initializing the pages. */ - set_page_links(page, zone, nid, start_pfn); + set_page_links(page, zone, nid, start_pfn, is_reserved); init_page_count(page); page_mapcount_reset(page); page_cpupid_reset_last(page); /* - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* * ZONE_DEVICE pages union ->lru with a ->pgmap back * pointer and hmm_data. It is a bug if a ZONE_DEVICE * page is ever freed or placed on a driver-private list. @@ -5612,7 +5610,18 @@ static void __meminit __memmap_init_hotplug(unsigned long size, int nid, pfn = max(ALIGN_DOWN(pfn - 1, pageblock_nr_pages), start_pfn); stride -= pfn; - __init_pageblock(pfn, stride, zone, nid, pgmap); + /* + * The last argument of __init_pageblock is a boolean + * value indicating if the page will be marked as reserved. + * + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * Under certain circumstances ZONE_DEVICE pages may not + * need to be marked as reserved, however there is still + * code that is depending on this being set for now. + */ + __init_pageblock(pfn, stride, zone, nid, pgmap, true); cond_resched(); }