From patchwork Wed Oct 17 23:54:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10646613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7D4F112B for ; Wed, 17 Oct 2018 23:54:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D916D285A2 for ; Wed, 17 Oct 2018 23:54:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CCAE7285CE; Wed, 17 Oct 2018 23:54:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33D21285A2 for ; Wed, 17 Oct 2018 23:54:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 347BF6B000A; Wed, 17 Oct 2018 19:54:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2D0396B000D; Wed, 17 Oct 2018 19:54:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 173C46B026C; Wed, 17 Oct 2018 19:54:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id C880E6B000A for ; Wed, 17 Oct 2018 19:54:10 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id f17-v6so22368270plr.1 for ; Wed, 17 Oct 2018 16:54:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=OIbxSgYz1QauRJ5KpMHPxP86VVk0hXfD5aRBBonaBJ8=; b=ZIIUkFeF15kbWSQG8EFC6EuGFl625bP7d07lH1ZU/QqEbvs0Co9nQZQ4sVSEAqH3Ok cK+m3utD+unUDgQBAtEjxgTMBQunEQe/TCqlgAjM163WCZZtiX2xRTiivT6Jwb9+9VbO XWxQRBO7jylaJzRMbsuN1O0I4h6Ie1tt0y7jhIS9oNgSP5EdaUhtvG9xE1sU7zxohC+L t2EHs9ZLzsMS76dU1xpC1mbeqKz7rInUiQZ+n1k61emj0s2s91q3Ij65vKnmCkUnxBem 4NEiAiuyRyxKByHTOlEro/hQ2e+4KheQusD+wPuhvLtz9ywEL64f+kTFQAe3sXBF506G 23nw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohVNxgpFjylztOcgB4Az8Zwf12ZfeBg1983KX0mtvkxaQrCjKwr NSSVHb7TEZ18xyWJhLyfVCTQlwccXWJQeBLwg6W0+jsIlKyt3EqS8SybkWQBF/S/ptcFskgexEi lI97uAhuS66hIHSugajKRk0G6qKh/79M0Cfa+svQ+D3tSUVe+MjsK0ZPEIiCDJlbvwQ== X-Received: by 2002:a63:5308:: with SMTP id h8-v6mr25571940pgb.358.1539820450461; Wed, 17 Oct 2018 16:54:10 -0700 (PDT) X-Google-Smtp-Source: ACcGV63HHe4kR5JS+8NaQyKDo12NxqxBZcEZSz7YwHjCE9Gdua94zKROUXWos80lCi+GPQIWwTvg X-Received: by 2002:a63:5308:: with SMTP id h8-v6mr25571909pgb.358.1539820449442; Wed, 17 Oct 2018 16:54:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539820449; cv=none; d=google.com; s=arc-20160816; b=tquWB6glltp/7NtY41EwOAiwC3pfIyMh7cJj4XEjRTWvXz5qfasorV2YS88+Hnr4+P 564P8Fs8fOgJ2xNfPE+KEeEsbBGwvRz/P2/m2jwJr9xGh3EoGiaOoUc2DQaP+YjeWV2d re5QBTGc/99Y5KFHep6D264YFuhpdx5kW3qtke36SumiVe0Re64SCOsYDAlnX3sgkI5M xBhHK8J87AN3INnZVsy4YMFOFs9OQr5cKEtRVSynEf4ajkoMfXsq3emckErrXrS16zp2 6uUVf1pMRD3kMlJ//f3VOtDgXo3dQBwHpXaWFY8Qzc694q76uoF9Aldxx6UFdhZK/fBu Omcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=OIbxSgYz1QauRJ5KpMHPxP86VVk0hXfD5aRBBonaBJ8=; b=twzJLvqDUxeAdvxJ9lNgQpWRyxXZL6OTHO+flvXwDAXxbrs7OEn3zJWEU2bQfbUvEj LLq2R5oufl38jg1CazzbKDDsbr9AEY2sYv+GSx845fXUgV1DHGWCmB0xNLTVKtV7cVyF KQrRS1ZPvK3lVQGM8qHRh0MFYEsO56vv56YJad+yqyJWcOftrDuhriMjgYq/3A6O160J XBNWrl5ne8D+XASbSZPJbryMxUPV62yhIRQzi+v/xy/QBhZeHcqDpMfDTQVtv5XJnkBE e+AZJzGuINe4RJJ1zhG7PqR47U3XIMWefaxw3VcT7oPYbJ6SyS2FyC/YmeHx0gG3DYag N6+Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id g12-v6si19758215pla.70.2018.10.17.16.54.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Oct 2018 16:54:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Oct 2018 16:54:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,393,1534834800"; d="scan'208";a="100366568" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.154]) by orsmga001.jf.intel.com with ESMTP; 17 Oct 2018 16:54:08 -0700 Subject: [mm PATCH v4 1/6] mm: Use mm_zero_struct_page from SPARC on all 64b architectures From: Alexander Duyck To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, davem@davemloft.net, yi.z.zhang@linux.intel.com, khalid.aziz@oracle.com, rppt@linux.vnet.ibm.com, vbabka@suse.cz, sparclinux@vger.kernel.org, dan.j.williams@intel.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mingo@kernel.org, kirill.shutemov@linux.intel.com Date: Wed, 17 Oct 2018 16:54:08 -0700 Message-ID: <20181017235408.17213.38641.stgit@localhost.localdomain> In-Reply-To: <20181017235043.17213.92459.stgit@localhost.localdomain> References: <20181017235043.17213.92459.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This change makes it so that we use the same approach that was already in use on Sparc on all the archtectures that support a 64b long. This is mostly motivated by the fact that 7 to 10 store/move instructions are likely always going to be faster than having to call into a function that is not specialized for handling page init. An added advantage to doing it this way is that the compiler can get away with combining writes in the __init_single_page call. As a result the memset call will be reduced to only about 4 write operations, or at least that is what I am seeing with GCC 6.2 as the flags, LRU poitners, and count/mapcount seem to be cancelling out at least 4 of the 8 assignments on my system. One change I had to make to the function was to reduce the minimum page size to 56 to support some powerpc64 configurations. This change should introduce no change on SPARC since it already had this code. In the case of x86_64 I saw a reduction from 3.75s to 2.80s when initializing 384GB of RAM per node. Pavel Tatashin tested on a system with Broadcom's Stingray CPU and 48GB of RAM and found that __init_single_page() takes 19.30ns / 64-byte struct page before this patch and with this patch it takes 17.33ns / 64-byte struct page. Mike Rapoport ran a similar test on a OpenPower (S812LC 8348-21C) with Power8 processor and 128GB or RAM. His results per 64-byte struct page were 4.68ns before, and 4.59ns after this patch. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin Acked-by: Michal Hocko --- arch/sparc/include/asm/pgtable_64.h | 30 -------------------------- include/linux/mm.h | 41 ++++++++++++++++++++++++++++++++--- 2 files changed, 38 insertions(+), 33 deletions(-) diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h index 1393a8ac596b..22500c3be7a9 100644 --- a/arch/sparc/include/asm/pgtable_64.h +++ b/arch/sparc/include/asm/pgtable_64.h @@ -231,36 +231,6 @@ extern struct page *mem_map_zero; #define ZERO_PAGE(vaddr) (mem_map_zero) -/* This macro must be updated when the size of struct page grows above 80 - * or reduces below 64. - * The idea that compiler optimizes out switch() statement, and only - * leaves clrx instructions - */ -#define mm_zero_struct_page(pp) do { \ - unsigned long *_pp = (void *)(pp); \ - \ - /* Check that struct page is either 64, 72, or 80 bytes */ \ - BUILD_BUG_ON(sizeof(struct page) & 7); \ - BUILD_BUG_ON(sizeof(struct page) < 64); \ - BUILD_BUG_ON(sizeof(struct page) > 80); \ - \ - switch (sizeof(struct page)) { \ - case 80: \ - _pp[9] = 0; /* fallthrough */ \ - case 72: \ - _pp[8] = 0; /* fallthrough */ \ - default: \ - _pp[7] = 0; \ - _pp[6] = 0; \ - _pp[5] = 0; \ - _pp[4] = 0; \ - _pp[3] = 0; \ - _pp[2] = 0; \ - _pp[1] = 0; \ - _pp[0] = 0; \ - } \ -} while (0) - /* PFNs are real physical page numbers. However, mem_map only begins to record * per-page information starting at pfn_base. This is to handle systems where * the first physical page in the machine is at some huge physical address, diff --git a/include/linux/mm.h b/include/linux/mm.h index fcf9cc9d535f..6e2c9631af05 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -98,10 +98,45 @@ static inline void set_max_mapnr(unsigned long limit) { } /* * On some architectures it is expensive to call memset() for small sizes. - * Those architectures should provide their own implementation of "struct page" - * zeroing by defining this macro in . + * If an architecture decides to implement their own version of + * mm_zero_struct_page they should wrap the defines below in a #ifndef and + * define their own version of this macro in */ -#ifndef mm_zero_struct_page +#if BITS_PER_LONG == 64 +/* This function must be updated when the size of struct page grows above 80 + * or reduces below 56. The idea that compiler optimizes out switch() + * statement, and only leaves move/store instructions. Also the compiler can + * combine write statments if they are both assignments and can be reordered, + * this can result in several of the writes here being dropped. + */ +#define mm_zero_struct_page(pp) __mm_zero_struct_page(pp) +static inline void __mm_zero_struct_page(struct page *page) +{ + unsigned long *_pp = (void *)page; + + /* Check that struct page is either 56, 64, 72, or 80 bytes */ + BUILD_BUG_ON(sizeof(struct page) & 7); + BUILD_BUG_ON(sizeof(struct page) < 56); + BUILD_BUG_ON(sizeof(struct page) > 80); + + switch (sizeof(struct page)) { + case 80: + _pp[9] = 0; /* fallthrough */ + case 72: + _pp[8] = 0; /* fallthrough */ + case 64: + _pp[7] = 0; /* fallthrough */ + case 56: + _pp[6] = 0; + _pp[5] = 0; + _pp[4] = 0; + _pp[3] = 0; + _pp[2] = 0; + _pp[1] = 0; + _pp[0] = 0; + } +} +#else #define mm_zero_struct_page(pp) ((void)memset((pp), 0, sizeof(struct page))) #endif