From patchwork Fri Nov 30 21:52:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10707247 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8090E14E2 for ; Fri, 30 Nov 2018 21:52:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 723943061A for ; Fri, 30 Nov 2018 21:52:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6518E3061F; Fri, 30 Nov 2018 21:52:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D00A63061A for ; Fri, 30 Nov 2018 21:52:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8DE46B5A6C; Fri, 30 Nov 2018 16:52:55 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C17536B5A6D; Fri, 30 Nov 2018 16:52:55 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE01C6B5A6E; Fri, 30 Nov 2018 16:52:55 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 68B736B5A6C for ; Fri, 30 Nov 2018 16:52:55 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id bj3so5048295plb.17 for ; Fri, 30 Nov 2018 13:52:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=94em1wpyLIy/AWmMo3OJIDZ0oZJTMB9fqgJ3TnSzj9o=; b=P/tM7atZo57bXMHVulJjGYjSW3AwrH/Sx31cz7/HuZ7cUtNqeZAKIBlV/YRL42l+af kmUXoAaov+RXvngyU8+vKhVG9OCXocVdhxoMqawfWw8kvGjM7q6oN1MBgIpKbCSZVE8I hLoWN0X1EJvmMnOgdfzMF8LAzObziyZnOaVnWmNMTXqkK/H/VXzZVA/W+Os4vO7wt8V3 oQqdwDkT1lvwKzVn8RKM0YweLG3J9ax5STJaiNb+6FIRKh1dVnSBjVhaSOFjxWci/354 /oB4rdW9GWLesBUb1cpEuA8HrIMx8H4vlmU0AGZKD1dxXsu20qfLR5RkKwCrTm0FYHSd OzzQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWacDyzk2OVkG54aJqVY+dU5hwwerYt5Tj3AT/C/CTHKNz6JZUgF FTfX2wtpV5AKL0XLbQiZ57vbelRHzJmSLLdHWpMNhqB4nebbcy4b12jAl8C8e2+KE9XXyaaYrDA cBXXZdmeNBa/Iz8xwYmKOQhoNzBaQkFIZMcAL7UnakhlnUoqV7VwqkjKcL1MDeKPanA== X-Received: by 2002:a17:902:7896:: with SMTP id q22mr7359771pll.280.1543614775043; Fri, 30 Nov 2018 13:52:55 -0800 (PST) X-Google-Smtp-Source: AFSGD/XJFB6xcY/8kn/3Hr50gemniEsbOIarekIRF4pLmOb2R8VmMUpnBOgV+Te5u9upzf6el4Db X-Received: by 2002:a17:902:7896:: with SMTP id q22mr7359711pll.280.1543614773945; Fri, 30 Nov 2018 13:52:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543614773; cv=none; d=google.com; s=arc-20160816; b=S4paHTJyy7ofbDvnMR34b4UESNArwQDIPqN/OFi9M2HA4C1qOT2eDpI4N9BPIYrrSH KFBclu/N+i0g0USNmY/LnqZxY9hHyppFThTOyTU2I7BVMUqCnUvHls1oV7paxkV/Xet2 2QJmyWBUFFKlap1g6mZWZsO7dmqt2eE9fhbXikbRFD0VNPhykFaa87KRtRI3kAK9d8tQ reMciHTgmEK1KiWLYNSIAzVZoYkPJT4XHmqn0a2b+MjuSNUpIl0okifXsDdki2HaybvI X1EvSUo1LOW23PPMYq9ZDkHK3ZwVmPvefYAP0I4h+SXpEn1+5hkgWx5EMmj32MFqhddx 6nmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=94em1wpyLIy/AWmMo3OJIDZ0oZJTMB9fqgJ3TnSzj9o=; b=il+ZMCVhs1mX94remYQNGKM+7Ij1vi8FOZaobZZqT6Nb2wzyGjB9V7ICMuJMYUDrsF esAOvFdmpkjCpUV7gTkXdCSlhWP6/DLoboFjnm93Z8dmRz4KUuo7HDg/AGOfCZiWVV1W hoHJ6uFrhhR9yntJ3H09ChtnxJo4SPestus0ZGCaLglC76MBL+9b/+hqU+6LnQt4F9gj aCnwFTJLzwmYUFKbPZdAuHzqPQRe3h+YjGM3y/BVCAV4p0ZXCI4/SyrnZ0L88sMUfArI xMd80IdUxzseHp66Llc1WOuOsfU/UxdfQLDd6sdbN1aMqFl+83BcChEX+5OZs5+dwHn+ BDnA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id i129si6691798pfb.32.2018.11.30.13.52.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Nov 2018 13:52:53 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Nov 2018 13:52:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,300,1539673200"; d="scan'208";a="114393654" Received: from ahduyck-desk1.jf.intel.com (HELO ahduyck-desk1.amr.corp.intel.com) ([10.7.198.76]) by orsmga001.jf.intel.com with ESMTP; 30 Nov 2018 13:52:53 -0800 Subject: [mm PATCH v6 1/7] mm: Use mm_zero_struct_page from SPARC on all 64b architectures From: Alexander Duyck To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, davem@davemloft.net, pavel.tatashin@microsoft.com, mhocko@suse.com, mingo@kernel.org, kirill.shutemov@linux.intel.com, dan.j.williams@intel.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, rppt@linux.vnet.ibm.com, willy@infradead.org, vbabka@suse.cz, khalid.aziz@oracle.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, yi.z.zhang@linux.intel.com, alexander.h.duyck@linux.intel.com Date: Fri, 30 Nov 2018 13:52:53 -0800 Message-ID: <154361477318.7497.13432441396440493352.stgit@ahduyck-desk1.amr.corp.intel.com> In-Reply-To: <154361452447.7497.1348692079883153517.stgit@ahduyck-desk1.amr.corp.intel.com> References: <154361452447.7497.1348692079883153517.stgit@ahduyck-desk1.amr.corp.intel.com> User-Agent: StGit/unknown-version MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use the same approach that was already in use on Sparc on all the architectures that support a 64b long. This is mostly motivated by the fact that 7 to 10 store/move instructions are likely always going to be faster than having to call into a function that is not specialized for handling page init. An added advantage to doing it this way is that the compiler can get away with combining writes in the __init_single_page call. As a result the memset call will be reduced to only about 4 write operations, or at least that is what I am seeing with GCC 6.2 as the flags, LRU pointers, and count/mapcount seem to be cancelling out at least 4 of the 8 assignments on my system. One change I had to make to the function was to reduce the minimum page size to 56 to support some powerpc64 configurations. This change should introduce no change on SPARC since it already had this code. In the case of x86_64 I saw a reduction from 3.75s to 2.80s when initializing 384GB of RAM per node. Pavel Tatashin tested on a system with Broadcom's Stingray CPU and 48GB of RAM and found that __init_single_page() takes 19.30ns / 64-byte struct page before this patch and with this patch it takes 17.33ns / 64-byte struct page. Mike Rapoport ran a similar test on a OpenPower (S812LC 8348-21C) with Power8 processor and 128GB or RAM. His results per 64-byte struct page were 4.68ns before, and 4.59ns after this patch. Reviewed-by: Pavel Tatashin Acked-by: Michal Hocko Signed-off-by: Alexander Duyck --- arch/sparc/include/asm/pgtable_64.h | 30 -------------------------- include/linux/mm.h | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 38 insertions(+), 33 deletions(-) diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 1393a8ac596b..22500c3be7a9 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -231,36 +231,6 @@ extern unsigned long _PAGE_ALL_SZ_BITS; extern struct page *mem_map_zero; #define ZERO_PAGE(vaddr) (mem_map_zero) -/* This macro must be updated when the size of struct page grows above 80 - * or reduces below 64. - * The idea that compiler optimizes out switch() statement, and only - * leaves clrx instructions - */ -#define mm_zero_struct_page(pp) do { \ - unsigned long *_pp = (void *)(pp); \ - \ - /* Check that struct page is either 64, 72, or 80 bytes */ \ - BUILD_BUG_ON(sizeof(struct page) & 7); \ - BUILD_BUG_ON(sizeof(struct page) < 64); \ - BUILD_BUG_ON(sizeof(struct page) > 80); \ - \ - switch (sizeof(struct page)) { \ - case 80: \ - _pp[9] = 0; /* fallthrough */ \ - case 72: \ - _pp[8] = 0; /* fallthrough */ \ - default: \ - _pp[7] = 0; \ - _pp[6] = 0; \ - _pp[5] = 0; \ - _pp[4] = 0; \ - _pp[3] = 0; \ - _pp[2] = 0; \ - _pp[1] = 0; \ - _pp[0] = 0; \ - } \ -} while (0) - /* PFNs are real physical page numbers. However, mem_map only begins to record * per-page information starting at pfn_base. This is to handle systems where * the first physical page in the machine is at some huge physical address, diff --git a/include/linux/mm.h b/include/linux/mm.h index 692158d6c619..eb6e52b66bc2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -123,10 +123,45 @@ extern int mmap_rnd_compat_bits __read_mostly; /* * On some architectures it is expensive to call memset() for small sizes. - * Those architectures should provide their own implementation of "struct page" - * zeroing by defining this macro in . + * If an architecture decides to implement their own version of + * mm_zero_struct_page they should wrap the defines below in a #ifndef and + * define their own version of this macro in */ -#ifndef mm_zero_struct_page +#if BITS_PER_LONG == 64 +/* This function must be updated when the size of struct page grows above 80 + * or reduces below 56. The idea that compiler optimizes out switch() + * statement, and only leaves move/store instructions. Also the compiler can + * combine write statments if they are both assignments and can be reordered, + * this can result in several of the writes here being dropped. + */ +#define mm_zero_struct_page(pp) __mm_zero_struct_page(pp) +static inline void __mm_zero_struct_page(struct page *page) +{ + unsigned long *_pp = (void *)page; + + /* Check that struct page is either 56, 64, 72, or 80 bytes */ + BUILD_BUG_ON(sizeof(struct page) & 7); + BUILD_BUG_ON(sizeof(struct page) < 56); + BUILD_BUG_ON(sizeof(struct page) > 80); + + switch (sizeof(struct page)) { + case 80: + _pp[9] = 0; /* fallthrough */ + case 72: + _pp[8] = 0; /* fallthrough */ + case 64: + _pp[7] = 0; /* fallthrough */ + case 56: + _pp[6] = 0; + _pp[5] = 0; + _pp[4] = 0; + _pp[3] = 0; + _pp[2] = 0; + _pp[1] = 0; + _pp[0] = 0; + } +} +#else #define mm_zero_struct_page(pp) ((void)memset((pp), 0, sizeof(struct page))) #endif