From patchwork Tue Sep 25 05:42:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arun KS X-Patchwork-Id: 10613359 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1D626CB for ; Tue, 25 Sep 2018 05:42:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 91DCE29A31 for ; Tue, 25 Sep 2018 05:42:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 830F829A37; Tue, 25 Sep 2018 05:42:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D9EAD29A31 for ; Tue, 25 Sep 2018 05:42:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DCF98E005D; Tue, 25 Sep 2018 01:42:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 28CFC8E0041; Tue, 25 Sep 2018 01:42:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17C748E005D; Tue, 25 Sep 2018 01:42:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id C9F228E0041 for ; Tue, 25 Sep 2018 01:42:53 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id f89-v6so561616pff.7 for ; Mon, 24 Sep 2018 22:42:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id; bh=tlJgJl6DTxB+XzuiX6ByJ0I9wIQGqDPWwJBRVMxePxc=; b=MmpXOdgt45GOT3SDsEr/i5u7Llz6RqVlIN+rvRbt7sJx0Fqn+1ltvV+jNR/gsFwNk7 iFVwRbf55HULdWktUyoUvhDZgm9vwk2kijOUAb0woyWkLmlD704xoffBGhLTY/Vgn3aq Go8w7JWp5QGf/ivY5aiGeHn3rLFrJ8x0vOleOTpYpU8IPlgd4L8TnMQmxlGB7ikI07ur wWNph84qll0/AolCCrW57CAbrhwQzHT507UBKo/UW2j/oQOa2uKPT+nmam9oHoaps7+R 7v0xGb+reKMb73rrYAY7PMwYnUGf9jcEEwduklbcmgnOexpP2+WY+e9kRzAdL0ACjS6h lz3A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of arunks@qualcomm.com designates 103.229.18.198 as permitted sender) smtp.mailfrom=arunks@qualcomm.com X-Gm-Message-State: ABuFfoi7NUO/il0CbYmsAqJ8a+7Uhnr6E3E36+BV7e8Go5ytx7Mfqn16 qEou0ZM/XHUNSUAvEA5/jYEFpc4sywnlhvVboLHElW6SkcCp5lPvm6VBl25/p2k4qoX9GMGtqOs qGyzbA0MTwDjhQFbGirdDz61vRzwoVPXYVAhnxMvE+CyT8IBuidBPpuQpnByjCak= X-Received: by 2002:a63:be4a:: with SMTP id g10-v6mr1756522pgo.378.1537854173409; Mon, 24 Sep 2018 22:42:53 -0700 (PDT) X-Google-Smtp-Source: ACcGV63eN2lfzUaY4XgVmEh78MEcKiABXewuK5Jh90pCCGIc3kzouKhQlfljKYbxrWVA0i4750m9 X-Received: by 2002:a63:be4a:: with SMTP id g10-v6mr1756475pgo.378.1537854172084; Mon, 24 Sep 2018 22:42:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537854172; cv=none; d=google.com; s=arc-20160816; b=Hx+MSFwITB9vzMo9zegFNcwRYIn0ZXbppXdHlkRAg+VJ0TMkKAh2iauVOtrL4aeKbB XY6t9p2kiTJnt5AGPdyMY97z/L13RQhxtty74+SyoWaaNc4p8vj8KYsDTkcJPHi8TFaO TcRHacB+i0RjAlN8whOAALcKJ4Np4YS6x4V/bvdRu8h7gcz+5WlifLXeRYQgBQMUL7zU QXCcXakZ+Jp7oHSWSgsdHVx4e+/oKXw7Si/hIg5cRUN1jCoEU8TO0JOuuJszYgKtSTgS vvOLZidSd2zlNeqtzHOTMGJivl71M5yW5nJ6WKfsaCLYyqfu4u5UVe/RZkJPUnm+pf7a wkLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from; bh=tlJgJl6DTxB+XzuiX6ByJ0I9wIQGqDPWwJBRVMxePxc=; b=iHYrCRpS7/gjZwiRCvsu/ZEtMYgCH8NZDz7wpOvpK8dQjeT1KZiXWpy7B9EiRvu+HG TNDlIk4Sz6szxEd2NCFRwndFY0rBBxY3L9bGBarjEisTdhSVHDwPihTm1mFOmuZwT36d 3Bs+RcCNZ+pK+9C2xc0rv0IBERDpq/trb3jAruiOQa4VN0m4PBnm2OHbPgyJnhgqruk1 YQyVY8qWJRg0OZOEaOcJgHPZ7b6uKeUYv2yKPnxKMRADSEHZHm5ZJKKPn2cz08UfKpH3 U/3kWybGOrJNQYIiXkE0A2WEmlPu+g5dNRvW0oRWNvm8a7yGwjLB8NCYWjCyymoA6vxF LaAQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of arunks@qualcomm.com designates 103.229.18.198 as permitted sender) smtp.mailfrom=arunks@qualcomm.com Received: from alexa-out-blr.qualcomm.com (alexa-out-blr-02.qualcomm.com. [103.229.18.198]) by mx.google.com with ESMTPS id 8-v6si1560571pla.252.2018.09.24.22.42.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Sep 2018 22:42:52 -0700 (PDT) Received-SPF: pass (google.com: domain of arunks@qualcomm.com designates 103.229.18.198 as permitted sender) client-ip=103.229.18.198; Authentication-Results: mx.google.com; spf=pass (google.com: domain of arunks@qualcomm.com designates 103.229.18.198 as permitted sender) smtp.mailfrom=arunks@qualcomm.com X-IronPort-AV: E=Sophos;i="5.54,300,1534789800"; d="scan'208";a="177287" Received: from ironmsg04-blr.qualcomm.com ([10.86.208.133]) by alexa-out-blr.qualcomm.com with ESMTP/TLS/AES256-SHA; 25 Sep 2018 11:12:49 +0530 X-IronPort-AV: E=McAfee;i="5900,7806,9026"; a="1554084" Received: from blr-ubuntu-104.ap.qualcomm.com (HELO blr-ubuntu-104.qualcomm.com) ([10.79.40.64]) by ironmsg04-blr.qualcomm.com with ESMTP; 25 Sep 2018 11:12:49 +0530 Received: by blr-ubuntu-104.qualcomm.com (Postfix, from userid 346745) id 3AC832BBB; Tue, 25 Sep 2018 11:12:48 +0530 (IST) From: Arun KS To: kys@microsoft.com, haiyangz@microsoft.com, sthemmin@microsoft.com, boris.ostrovsky@oracle.com, jgross@suse.com, akpm@linux-foundation.org, dan.j.williams@intel.com, mhocko@suse.com, vbabka@suse.cz, iamjoonsoo.kim@lge.com, osalvador@suse.de, malat@debian.org, yasu.isimatu@gmail.com, devel@linuxdriverproject.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, xen-devel@lists.xenproject.org Cc: vatsa@codeaurora.org, vinmenon@codeaurora.org, getarunks@gmail.com, Arun KS Subject: [PATCH v2] memory_hotplug: Free pages as higher order Date: Tue, 25 Sep 2018 11:12:38 +0530 Message-Id: <1537854158-9766-1-git-send-email-arunks@codeaurora.org> X-Mailer: git-send-email 1.9.1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When free pages are done with higher order, time spend on coalescing pages by buddy allocator can be reduced. With section size of 256MB, hot add latency of a single section shows improvement from 50-60 ms to less than 1 ms, hence improving the hot add latency by 60%. Modify external providers of online callback to align with the change. Signed-off-by: Arun KS --- Changes since v1: - Removed prefetch() Changes since RFC: - Rebase. - As suggested by Michal Hocko remove pages_per_block. - Modifed external providers of online_page_callback. v1: https://lore.kernel.org/patchwork/patch/989445/ RFC: https://lore.kernel.org/patchwork/patch/984754/ --- drivers/hv/hv_balloon.c | 6 +++-- drivers/xen/balloon.c | 18 ++++++++++++--- include/linux/memory_hotplug.h | 2 +- mm/memory_hotplug.c | 51 ++++++++++++++++++++++++++++++++---------- 4 files changed, 59 insertions(+), 18 deletions(-) diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c index b1b7880..c5bc0b5 100644 --- a/drivers/hv/hv_balloon.c +++ b/drivers/hv/hv_balloon.c @@ -771,7 +771,7 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size, } } -static void hv_online_page(struct page *pg) +static int hv_online_page(struct page *pg, unsigned int order) { struct hv_hotadd_state *has; unsigned long flags; @@ -783,10 +783,12 @@ static void hv_online_page(struct page *pg) if ((pfn < has->start_pfn) || (pfn >= has->end_pfn)) continue; - hv_page_online_one(has, pg); + hv_bring_pgs_online(has, pfn, (1UL << order)); break; } spin_unlock_irqrestore(&dm_device.ha_lock, flags); + + return 0; } static int pfn_covered(unsigned long start_pfn, unsigned long pfn_cnt) diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index e12bb25..010cf4d 100644 --- a/drivers/xen/balloon.c +++ b/drivers/xen/balloon.c @@ -390,8 +390,8 @@ static enum bp_state reserve_additional_memory(void) /* * add_memory_resource() will call online_pages() which in its turn - * will call xen_online_page() callback causing deadlock if we don't - * release balloon_mutex here. Unlocking here is safe because the + * will call xen_bring_pgs_online() callback causing deadlock if we + * don't release balloon_mutex here. Unlocking here is safe because the * callers drop the mutex before trying again. */ mutex_unlock(&balloon_mutex); @@ -422,6 +422,18 @@ static void xen_online_page(struct page *page) mutex_unlock(&balloon_mutex); } +static int xen_bring_pgs_online(struct page *pg, unsigned int order) +{ + unsigned long i, size = (1 << order); + unsigned long start_pfn = page_to_pfn(pg); + + pr_debug("Online %lu pages starting at pfn 0x%lx\n", size, start_pfn); + for (i = 0; i < size; i++) + xen_online_page(pfn_to_page(start_pfn + i)); + + return 0; +} + static int xen_memory_notifier(struct notifier_block *nb, unsigned long val, void *v) { if (val == MEM_ONLINE) @@ -744,7 +756,7 @@ static int __init balloon_init(void) balloon_stats.max_retry_count = RETRY_UNLIMITED; #ifdef CONFIG_XEN_BALLOON_MEMORY_HOTPLUG - set_online_page_callback(&xen_online_page); + set_online_page_callback(&xen_bring_pgs_online); register_memory_notifier(&xen_memory_nb); register_sysctl_table(xen_root); diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 34a2822..7b04c1d 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -87,7 +87,7 @@ extern int test_pages_in_a_zone(unsigned long start_pfn, unsigned long end_pfn, unsigned long *valid_start, unsigned long *valid_end); extern void __offline_isolated_pages(unsigned long, unsigned long); -typedef void (*online_page_callback_t)(struct page *page); +typedef int (*online_page_callback_t)(struct page *page, unsigned int order); extern int set_online_page_callback(online_page_callback_t callback); extern int restore_online_page_callback(online_page_callback_t callback); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 38d94b7..9f67794 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -47,7 +47,7 @@ * and restore_online_page_callback() for generic callback restore. */ -static void generic_online_page(struct page *page); +static int generic_online_page(struct page *page, unsigned int order); static online_page_callback_t online_page_callback = generic_online_page; static DEFINE_MUTEX(online_page_callback_lock); @@ -655,26 +655,53 @@ void __online_page_free(struct page *page) } EXPORT_SYMBOL_GPL(__online_page_free); -static void generic_online_page(struct page *page) +static int generic_online_page(struct page *page, unsigned int order) { - __online_page_set_limits(page); - __online_page_increment_counters(page); - __online_page_free(page); + unsigned long nr_pages = 1 << order; + struct page *p = page; + unsigned int loop; + + for (loop = 0 ; loop < nr_pages ; loop++, p++) { + __ClearPageReserved(p); + set_page_count(p, 0); + } + + adjust_managed_page_count(page, nr_pages); + set_page_refcounted(page); + __free_pages(page, order); + + return 0; +} + +static int online_pages_blocks(unsigned long start, unsigned long nr_pages) +{ + unsigned long end = start + nr_pages; + int order, ret, onlined_pages = 0; + + while (start < end) { + order = min(MAX_ORDER - 1UL, __ffs(start)); + + while (start + (1UL << order) > end) + order--; + + ret = (*online_page_callback)(pfn_to_page(start), order); + if (!ret) + onlined_pages += (1UL << order); + else if (ret > 0) + onlined_pages += ret; + + start += (1UL << order); + } + return onlined_pages; } static int online_pages_range(unsigned long start_pfn, unsigned long nr_pages, void *arg) { - unsigned long i; unsigned long onlined_pages = *(unsigned long *)arg; - struct page *page; if (PageReserved(pfn_to_page(start_pfn))) - for (i = 0; i < nr_pages; i++) { - page = pfn_to_page(start_pfn + i); - (*online_page_callback)(page); - onlined_pages++; - } + onlined_pages = online_pages_blocks(start_pfn, nr_pages); online_mem_sections(start_pfn, start_pfn + nr_pages);