From patchwork Mon Apr 3 16:50:51 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9659981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EF9E16032D for ; Mon, 3 Apr 2017 16:52:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E14F928495 for ; Mon, 3 Apr 2017 16:52:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D615B28499; Mon, 3 Apr 2017 16:52:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5668028489 for ; Mon, 3 Apr 2017 16:52:12 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cv5An-0002lV-0R; Mon, 03 Apr 2017 16:49:45 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cv5Al-0002kZ-Gb for xen-devel@lists.xen.org; Mon, 03 Apr 2017 16:49:43 +0000 Received: from [193.109.254.147] by server-2.bemta-6.messagelabs.com id FF/0A-19731-62D72E85; Mon, 03 Apr 2017 16:49:42 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrLLMWRWlGSWpSXmKPExsUyZ7p8oK5a7aM Ig8Y5PBZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8asK1OZC44rV7TPfsHWwPhPsouRi0NIYBKT xJbXaxkhnK+MEqeXLWGFcDYwSvR03maBcHoYJdrPbQJyODnYBIwkzh6dzghiiwhIS1z7fBmsn VmggUni+bmDQA4Hh7CAucTSGaIgNSwCqhI7fzQwg9i8Ap4Sj858ZwexJQTkJHasfsIEYnMKeE ksf3YRzBYCqlm19A0LRI2xRPvbi2wTGPkWMDKsYtQoTi0qSy3SNTTRSyrKTM8oyU3MzNE1NDD Ty00tLk5MT81JTCrWS87P3cQIDBYGINjBeH1jwCFGSQ4mJVHeDwqPIoT4kvJTKjMSizPii0pz UosPMcpwcChJ8C6rBsoJFqWmp1akZeYAwxYmLcHBoyTC+wIkzVtckJhbnJkOkTrFqCglztsOk hAASWSU5sG1wWLlEqOslDAvI9AhQjwFqUW5mSWo8q8YxTkYlYR5j4NM4cnMK4Gb/gpoMRPQ4i d3HoIsLklESEk1MO6uz+Lx/C+s3qmlfjDJeuKGwJlPJqgYGXzvC3nwctaXSyJflm5wDGs75Xt eIPLLlLOf1cRY1+v6n8uM/KT3u//347m3tPq3s1U6H2Sd+vBQBJs8RyHXOrb18Z/sPSYVVncx Rz7LrRFXSFowOVln9YP5z79ZdHdKXuQ5Ju9tGXiifeUGf8eJ/EosxRmJhlrMRcWJAMbETQqQA gAA X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-12.tower-27.messagelabs.com!1491238180!93851717!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 59872 invoked from network); 3 Apr 2017 16:49:41 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-12.tower-27.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 3 Apr 2017 16:49:41 -0000 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v33GnaCV031457 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Apr 2017 16:49:36 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v33Gnaso012591 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 3 Apr 2017 16:49:36 GMT Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10]) by aserv0121.oracle.com (8.13.8/8.13.8) with ESMTP id v33GnX39015896; Mon, 3 Apr 2017 16:49:34 GMT Received: from ovs101.us.oracle.com (/10.149.76.201) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 03 Apr 2017 09:49:33 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Mon, 3 Apr 2017 12:50:51 -0400 Message-Id: <1491238256-5517-5-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1491238256-5517-1-git-send-email-boris.ostrovsky@oracle.com> References: <1491238256-5517-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: aserv0022.oracle.com [141.146.126.234] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCH v2 4/9] mm: Scrub memory from idle loop X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Boris Ostrovsky --- Changes in v2: * Added node_to_scrub() * Include softirq_pending() in scrub_free_pages()'s return value xen/arch/arm/domain.c | 13 +++++++---- xen/arch/x86/domain.c | 3 +- xen/common/page_alloc.c | 52 ++++++++++++++++++++++++++++++++++++++-------- xen/include/xen/mm.h | 1 + 4 files changed, 54 insertions(+), 15 deletions(-) diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c index bb327da..fdf06e1 100644 --- a/xen/arch/arm/domain.c +++ b/xen/arch/arm/domain.c @@ -45,13 +45,16 @@ void idle_loop(void) if ( cpu_is_offline(smp_processor_id()) ) stop_cpu(); - local_irq_disable(); - if ( cpu_is_haltable(smp_processor_id()) ) + if ( !scrub_free_pages() ) { - dsb(sy); - wfi(); + local_irq_disable(); + if ( cpu_is_haltable(smp_processor_id()) ) + { + dsb(sy); + wfi(); + } + local_irq_enable(); } - local_irq_enable(); do_tasklet(); do_softirq(); diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c index 90e2b1f..a5f62b5 100644 --- a/xen/arch/x86/domain.c +++ b/xen/arch/x86/domain.c @@ -118,7 +118,8 @@ static void idle_loop(void) { if ( cpu_is_offline(smp_processor_id()) ) play_dead(); - (*pm_idle)(); + if ( !scrub_free_pages() ) + (*pm_idle)(); do_tasklet(); do_softirq(); /* diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index a560d3e..5cc489a 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -1043,16 +1043,44 @@ static struct page_info *merge_chunks(struct page_info *pg, unsigned int node, return pg; } -static void scrub_free_pages(unsigned int node) +static nodemask_t node_scrubbing; +static unsigned int node_to_scrub(bool_t get_node) +{ + nodeid_t node = cpu_to_node(smp_processor_id()), local_node; + + if ( node == NUMA_NO_NODE ) + node = 0; + local_node = node; + + /* + * Check local node fist and then then see if there are any memory-only + * nodes that may need scrubbing + */ + while ( 1 ) + { + if ( node_need_scrub[node] && + (!node_test_and_set(node, node_scrubbing) || !get_node) ) + return node; + do { + node = cycle_node(node, node_online_map); + if ( node == local_node ) + return NUMA_NO_NODE; + } while ( !cpumask_empty(&node_to_cpumask(node)) ); + } +} + +bool_t scrub_free_pages() { struct page_info *pg; unsigned int i, zone; - int order; + int order, cpu = smp_processor_id(); + nodeid_t node; - ASSERT(spin_is_locked(&heap_lock)); + node = node_to_scrub(true); + if ( node == NUMA_NO_NODE ) + return false; - if ( !node_need_scrub[node] ) - return; + spin_lock(&heap_lock); for ( zone = 0; zone < NR_ZONES; zone++ ) { @@ -1066,7 +1094,11 @@ static void scrub_free_pages(unsigned int node) break; for ( i = 0; i < (1UL << order); i++) + { scrub_one_page(&pg[i]); + if ( softirq_pending(cpu) ) + goto out; + } pg->count_info &= ~PGC_need_scrub; @@ -1077,7 +1109,12 @@ static void scrub_free_pages(unsigned int node) } } } - } + + out: + spin_unlock(&heap_lock); + node_clear(node, node_scrubbing); + return softirq_pending(cpu) || (node_to_scrub(false) != NUMA_NO_NODE); +} /* Free 2^@order set of pages. */ @@ -1142,9 +1179,6 @@ static void free_heap_pages( if ( tainted ) reserve_offlined_page(pg); - if ( need_scrub ) - scrub_free_pages(node); - spin_unlock(&heap_lock); } diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index 88de3c1..3d93fcc 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -138,6 +138,7 @@ void init_xenheap_pages(paddr_t ps, paddr_t pe); void xenheap_max_mfn(unsigned long mfn); void *alloc_xenheap_pages(unsigned int order, unsigned int memflags); void free_xenheap_pages(void *v, unsigned int order); +bool_t scrub_free_pages(void); #define alloc_xenheap_page() (alloc_xenheap_pages(0,0)) #define free_xenheap_page(v) (free_xenheap_pages(v,0)) /* Map machine page range in Xen virtual address space. */