From patchwork Mon Apr 2 02:30:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jia He X-Patchwork-Id: 10319421 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 66B5E60388 for ; Mon, 2 Apr 2018 02:33:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 54D6428D42 for ; Mon, 2 Apr 2018 02:33:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4932E28D4B; Mon, 2 Apr 2018 02:33:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BBC8E28D42 for ; Mon, 2 Apr 2018 02:33:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=z2eUKpBZd0Rn6p2FAMng7MJs82rV6FL8imsASFtk1CA=; b=AG/6G4p+3r6JtCvfdSnUpP11LL wpEVU5lqeEMMO8AL4kfadQMPvqixwLomLl5diTRMyhrPuqNE4N+X4KTfb7ERO63RLl4y71bjPDUvy 1BTyJhM2QM0bxqNdDvdc98ovl79VM9p8t1dx/9EMgil6NgSKep2oroxJ+fdGbGznmuEaEmPYO4bAx aCHoZjnV29G3JWPimn7K4eZezeqt6jD9vjc2W7VPelPuNUUfffifERVdlNsMm6M7hbE5DI63IE6PX CyO5BFnFd+0DxXpbuozNZ/9VFC4QGYPTMlCyg/bvbuXg0Nng2Ti/h3EbyrCujeJ3r8Ab+B8U8Y8Nm vcNaOI9g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1f2pHy-0006Se-Hl; Mon, 02 Apr 2018 02:33:42 +0000 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1f2pH6-0005qZ-TG for linux-arm-kernel@lists.infradead.org; Mon, 02 Apr 2018 02:32:53 +0000 Received: by mail-pf0-x241.google.com with SMTP id m68so8571061pfm.11 for ; Sun, 01 Apr 2018 19:32:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=W245SyIjd9INhYH8aBv4SlHrgmxRg+gEUosAndt3Dso=; b=a6uDOMxWiU1G+TYIE8/+zS7UDwCndaCEiQ5mhfwKm10oGpwZotJWca6C58f8R0PfX7 PjzxVskuYcsEWE+3lkXjs47W5aIvqnZX3MkTkEjriRBFXXwk+2FA2x2tgOS9ygIQHd4b ZLMnlEtQ83qH/h4qZb9LM3qIXjITsbQ93x1JF1aDpO/A7N6w1RNTNR4VmjAJG/+en7D9 IgngIylN2rZv5AcYjrcjS+mmFBPatuz1zfNSyPo3jITPL59UoGR5fshhnVbGLjmcP0Zv FoAYitk4tLoE+I+l2WsZtwqecX5hgcuotppi/v0JOX57014uF6so789il7KquC9yfNG3 Fp/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=W245SyIjd9INhYH8aBv4SlHrgmxRg+gEUosAndt3Dso=; b=fZHsUBxlp0+fOQ9Z9OQSzB3NgJn8XofC+PlhPv0b4vVld3HHI8dlsoZ0dKQsVUzo+j 4RSAnwvI9pvBLFyVkYnG0Eea1GswLMfjozLTjWgaQkk9qzDDhnqH8FddjKpLU6ZSTiW9 RDv7cdOaQdYu1Es/rzxcxJ4uxH5lRR9VrE7rxfOR1vTLv1Q4XTGZuYHLiACKXxiZ2wiA zNmEOMZzDlIq463UKCzBfPK/qjcmGaxJCf7KwSTC1+YvPgdYdbOq3DjqNQaNfj7cyx1W AX22fpJPLYl6KYLtHyun6WlGj2b+gs2HBhGZrHq1mddfl64HGUibGZCycNp7tqRBYXml 6MEg== X-Gm-Message-State: AElRT7E2muz2coLLBHzwYfTTpU++5RceTQubJGt5OIk86sZPxFkSkv3Q k/UznwVXNfWvEbFauOiP1K8= X-Google-Smtp-Source: AIpwx4/r1hjC9hiqsj6LWUZJnbFcvM4fnkuV3rptnrXQx7GQVW+PSJphzoEQnNpvQAggA0CmVj70iA== X-Received: by 10.101.69.5 with SMTP id n5mr5047390pgq.363.1522636356671; Sun, 01 Apr 2018 19:32:36 -0700 (PDT) Received: from localhost.localdomain ([38.106.11.25]) by smtp.gmail.com with ESMTPSA id t20sm25934371pfh.182.2018.04.01.19.32.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 01 Apr 2018 19:32:36 -0700 (PDT) From: Jia He To: Russell King , Catalin Marinas , Will Deacon , Mark Rutland , Ard Biesheuvel , Andrew Morton , Michal Hocko Subject: [PATCH v5 2/5] arm: arm64: page_alloc: reduce unnecessary binary search in memblock_next_valid_pfn() Date: Sun, 1 Apr 2018 19:30:33 -0700 Message-Id: <1522636236-12625-3-git-send-email-hejianet@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1522636236-12625-1-git-send-email-hejianet@gmail.com> References: <1522636236-12625-1-git-send-email-hejianet@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180401_193248_970179_C494447D X-CRM114-Status: GOOD ( 19.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Grygorii Strashko , Wei Yang , linux-mm@kvack.org, Eugeniu Rosca , Petr Tesarik , Nikolay Borisov , Jia He , Pavel Tatashin , Daniel Jordan , AKASHI Takahiro , Gioh Kim , Andrey Ryabinin , Laura Abbott , Daniel Vacek , Mel Gorman , Vladimir Murzin , Kees Cook , Philip Derrin , YASUAKI ISHIMATSU , Jia He , Kemi Wang , Vlastimil Babka , linux-arm-kernel@lists.infradead.org, Steve Capper , linux-kernel@vger.kernel.org, James Morse , Johannes Weiner MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Commit b92df1de5d28 ("mm: page_alloc: skip over regions of invalid pfns where possible") optimized the loop in memmap_init_zone(). But there is still some room for improvement. E.g. if pfn and pfn+1 are in the same memblock region, we can simply pfn++ instead of doing the binary search in memblock_next_valid_pfn. Signed-off-by: Jia He --- arch/arm/include/asm/page.h | 1 + arch/arm/mm/init.c | 28 ++++++++++++++++++++++------ arch/arm64/include/asm/page.h | 1 + arch/arm64/mm/init.c | 28 ++++++++++++++++++++++------ 4 files changed, 46 insertions(+), 12 deletions(-) diff --git a/arch/arm/include/asm/page.h b/arch/arm/include/asm/page.h index 489875c..f38909c 100644 --- a/arch/arm/include/asm/page.h +++ b/arch/arm/include/asm/page.h @@ -157,6 +157,7 @@ extern void copy_page(void *to, const void *from); typedef struct page *pgtable_t; #ifdef CONFIG_HAVE_ARCH_PFN_VALID +extern int early_region_idx; extern int pfn_valid(unsigned long); extern unsigned long memblock_next_valid_pfn(unsigned long pfn); #define skip_to_last_invalid_pfn(pfn) (memblock_next_valid_pfn(pfn) - 1) diff --git a/arch/arm/mm/init.c b/arch/arm/mm/init.c index 0fb85ca..06ed190 100644 --- a/arch/arm/mm/init.c +++ b/arch/arm/mm/init.c @@ -193,6 +193,8 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max_low, } #ifdef CONFIG_HAVE_ARCH_PFN_VALID +int early_region_idx __meminitdata = -1; + int pfn_valid(unsigned long pfn) { return memblock_is_map_memory(__pfn_to_phys(pfn)); @@ -203,28 +205,42 @@ EXPORT_SYMBOL(pfn_valid); unsigned long __init_memblock memblock_next_valid_pfn(unsigned long pfn) { struct memblock_type *type = &memblock.memory; + struct memblock_region *regions = type->regions; unsigned int right = type->cnt; unsigned int mid, left = 0; + unsigned long start_pfn, end_pfn; phys_addr_t addr = PFN_PHYS(++pfn); + /* fast path, return pfn+1 if next pfn is in the same region */ + if (early_region_idx != -1) { + start_pfn = PFN_DOWN(regions[early_region_idx].base); + end_pfn = PFN_DOWN(regions[early_region_idx].base + + regions[early_region_idx].size); + + if (pfn >= start_pfn && pfn < end_pfn) + return pfn; + } + + /* slow path, do the binary searching */ do { mid = (right + left) / 2; - if (addr < type->regions[mid].base) + if (addr < regions[mid].base) right = mid; - else if (addr >= (type->regions[mid].base + - type->regions[mid].size)) + else if (addr >= (regions[mid].base + regions[mid].size)) left = mid + 1; else { - /* addr is within the region, so pfn is valid */ + early_region_idx = mid; return pfn; } } while (left < right); if (right == type->cnt) return -1UL; - else - return PHYS_PFN(type->regions[right].base); + + early_region_idx = right; + + return PHYS_PFN(regions[early_region_idx].base); } EXPORT_SYMBOL(memblock_next_valid_pfn); #endif /*CONFIG_HAVE_ARCH_PFN_VALID*/ diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index e57d3f2..f0d8c8e5 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -38,6 +38,7 @@ extern void clear_page(void *to); typedef struct page *pgtable_t; #ifdef CONFIG_HAVE_ARCH_PFN_VALID +extern int early_region_idx; extern int pfn_valid(unsigned long); extern unsigned long memblock_next_valid_pfn(unsigned long pfn); #define skip_to_last_invalid_pfn(pfn) (memblock_next_valid_pfn(pfn) - 1) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 13e43ff..342e4e2 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -285,6 +285,8 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max) #endif /* CONFIG_NUMA */ #ifdef CONFIG_HAVE_ARCH_PFN_VALID +int early_region_idx __meminitdata = -1; + int pfn_valid(unsigned long pfn) { return memblock_is_map_memory(pfn << PAGE_SHIFT); @@ -295,28 +297,42 @@ EXPORT_SYMBOL(pfn_valid); unsigned long __init_memblock memblock_next_valid_pfn(unsigned long pfn) { struct memblock_type *type = &memblock.memory; + struct memblock_region *regions = type->regions; unsigned int right = type->cnt; unsigned int mid, left = 0; + unsigned long start_pfn, end_pfn; phys_addr_t addr = PFN_PHYS(++pfn); + /* fast path, return pfn+1 if next pfn is in the same region */ + if (early_region_idx != -1) { + start_pfn = PFN_DOWN(regions[early_region_idx].base); + end_pfn = PFN_DOWN(regions[early_region_idx].base + + regions[early_region_idx].size); + + if (pfn >= start_pfn && pfn < end_pfn) + return pfn; + } + + /* slow path, do the binary searching */ do { mid = (right + left) / 2; - if (addr < type->regions[mid].base) + if (addr < regions[mid].base) right = mid; - else if (addr >= (type->regions[mid].base + - type->regions[mid].size)) + else if (addr >= (regions[mid].base + regions[mid].size)) left = mid + 1; else { - /* addr is within the region, so pfn is valid */ + early_region_idx = mid; return pfn; } } while (left < right); if (right == type->cnt) return -1UL; - else - return PHYS_PFN(type->regions[right].base); + + early_region_idx = right; + + return PHYS_PFN(regions[early_region_idx].base); } EXPORT_SYMBOL(memblock_next_valid_pfn); #endif /*CONFIG_HAVE_ARCH_PFN_VALID*/