From patchwork Mon Apr 3 16:50:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9659993 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 39A1D6032D for ; Mon, 3 Apr 2017 16:52:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2C1B728494 for ; Mon, 3 Apr 2017 16:52:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 20BDF2849A; Mon, 3 Apr 2017 16:52:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9523128494 for ; Mon, 3 Apr 2017 16:52:19 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cv5Al-0002kx-Pm; Mon, 03 Apr 2017 16:49:43 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cv5Aj-0002jY-SV for xen-devel@lists.xen.org; Mon, 03 Apr 2017 16:49:41 +0000 Received: from [85.158.137.68] by server-4.bemta-3.messagelabs.com id 33/2E-03705-52D72E85; Mon, 03 Apr 2017 16:49:41 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrLLMWRWlGSWpSXmKPExsXSO6nOVVel9lG EwfTvChZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8bG/ZfZCs5pVTR1z2JuYJwh18XIxSEkMIFJ 4u/+7YwQzh9GiUVLGpghnA2MEi/P/2ODcHoYJV7tfA7kcHKwCRhJnD06nRHEFhGQlrj2+TJYO 7NAA5PE83MHwRLCAu4SLU9Ps4PYLAKqEpenHQRq5uDgFfCUuNDlDhKWEJCT2LH6CROIzSngJb H82UUwWwioZNXSNywQNYYSnzcuZZ7AyLeAkWEVo0ZxalFZapGuoZFeUlFmekZJbmJmjq6hgbF ebmpxcWJ6ak5iUrFecn7uJkZgsDAAwQ7GV93OhxglOZiURHk/KDyKEOJLyk+pzEgszogvKs1J LT7EKMPBoSTBq1kDlBMsSk1PrUjLzAGGLUxagoNHSYT3RTVQmre4IDG3ODMdInWKUVFKnFcJp E8AJJFRmgfXBouVS4yyUsK8jECHCPEUpBblZpagyr9iFOdgVBLm5QCZwpOZVwI3/RXQYiagxU /uPARZXJKIkJJqYOQ4K7HiqP6vzbMYvEWV9vIkzfq/54LT7PM9T7d08OoXZre+mRLiu2TG6zl sk9Vqtfa1NCe+db7l/6Ta8uXtzZlVRWKzwqwMHN1DZ9cV8FuuKfJcyWnelpiguuKnqu6JzU65 Lg+5XrrXPX62dvn1livrvb3Mri1tuLbcvkhAxHv5Pf6zbwLuciuxFGckGmoxFxUnAgBZ7RmDk AIAAA== X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-4.tower-31.messagelabs.com!1491238178!35775718!1 X-Originating-IP: [141.146.126.69] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTQxLjE0Ni4xMjYuNjkgPT4gMjc3MjE4\n X-StarScan-Received: X-StarScan-Version: 9.2.3; banners=-,-,- X-VirusChecked: Checked Received: (qmail 65317 invoked from network); 3 Apr 2017 16:49:40 -0000 Received: from aserp1040.oracle.com (HELO aserp1040.oracle.com) (141.146.126.69) by server-4.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 3 Apr 2017 16:49:40 -0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v33GnXgZ007986 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Apr 2017 16:49:34 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v33GnXph030617 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 3 Apr 2017 16:49:33 GMT Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v33GnWRX012591; Mon, 3 Apr 2017 16:49:33 GMT Received: from ovs101.us.oracle.com (/10.149.76.201) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 03 Apr 2017 09:49:32 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Mon, 3 Apr 2017 12:50:50 -0400 Message-Id: <1491238256-5517-4-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1491238256-5517-1-git-send-email-boris.ostrovsky@oracle.com> References: <1491238256-5517-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: userv0022.oracle.com [156.151.31.74] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCH v2 3/9] mm: Scrub pages in alloc_heap_pages() if needed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When allocating pages in alloc_heap_pages() first look for clean pages. If none is found then retry, take pages marked as unscrubbed and scrub them. Note that we shouldn't find unscrubbed pages in alloc_heap_pages() yet. However, this will become possible when we stop scrubbing from free_heap_pages() and instead do it from idle loop. Signed-off-by: Boris Ostrovsky --- xen/common/page_alloc.c | 87 ++++++++++++++++++++++++++++++++++------------- 1 files changed, 63 insertions(+), 24 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 56486a8..a560d3e 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -705,34 +705,17 @@ static struct page_info *alloc_heap_pages( unsigned int order, unsigned int memflags, struct domain *d) { - unsigned int i, j, zone = 0, nodemask_retry = 0; - nodeid_t first_node, node = MEMF_get_node(memflags), req_node = node; + unsigned int i, j, zone, nodemask_retry; + nodeid_t first_node, node, req_node; unsigned long request = 1UL << order; struct page_info *pg; - nodemask_t nodemask = (d != NULL ) ? d->node_affinity : node_online_map; - bool_t need_tlbflush = 0; + nodemask_t nodemask; + bool_t need_scrub, need_tlbflush = 0, use_unscrubbed = 0; uint32_t tlbflush_timestamp = 0; /* Make sure there are enough bits in memflags for nodeID. */ BUILD_BUG_ON((_MEMF_bits - _MEMF_node) < (8 * sizeof(nodeid_t))); - if ( node == NUMA_NO_NODE ) - { - if ( d != NULL ) - { - node = next_node(d->last_alloc_node, nodemask); - if ( node >= MAX_NUMNODES ) - node = first_node(nodemask); - } - if ( node >= MAX_NUMNODES ) - node = cpu_to_node(smp_processor_id()); - } - first_node = node; - - ASSERT(node < MAX_NUMNODES); - ASSERT(zone_lo <= zone_hi); - ASSERT(zone_hi < NR_ZONES); - if ( unlikely(order > MAX_ORDER) ) return NULL; @@ -746,7 +729,10 @@ static struct page_info *alloc_heap_pages( total_avail_pages + tmem_freeable_pages()) && ((memflags & MEMF_no_refcount) || !d || d->outstanding_pages < request) ) - goto not_found; + { + spin_unlock(&heap_lock); + return NULL; + } /* * TMEM: When available memory is scarce due to tmem absorbing it, allow @@ -759,6 +745,28 @@ static struct page_info *alloc_heap_pages( tmem_freeable_pages() ) goto try_tmem; + again: + + nodemask_retry = 0; + nodemask = (d != NULL ) ? d->node_affinity : node_online_map; + node = req_node = MEMF_get_node(memflags); + if ( node == NUMA_NO_NODE ) + { + if ( d != NULL ) + { + node = next_node(d->last_alloc_node, nodemask); + if ( node >= MAX_NUMNODES ) + node = first_node(nodemask); + } + if ( node >= MAX_NUMNODES ) + node = cpu_to_node(smp_processor_id()); + } + first_node = node; + + ASSERT(node < MAX_NUMNODES); + ASSERT(zone_lo <= zone_hi); + ASSERT(zone_hi < NR_ZONES); + /* * Start with requested node, but exhaust all node memory in requested * zone before failing, only calc new node value if we fail to find memory @@ -774,8 +782,16 @@ static struct page_info *alloc_heap_pages( /* Find smallest order which can satisfy the request. */ for ( j = order; j <= MAX_ORDER; j++ ) + { if ( (pg = page_list_remove_head(&heap(node, zone, j))) ) - goto found; + { + if ( (order == 0) || use_unscrubbed || + !test_bit(_PGC_need_scrub, &pg->count_info) ) + goto found; + + page_list_add_tail(pg, &heap(node, zone, j)); + } + } } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */ if ( (memflags & MEMF_exact_node) && req_node != NUMA_NO_NODE ) @@ -814,18 +830,32 @@ static struct page_info *alloc_heap_pages( } not_found: + /* + * If we couldn't find clean page let's search again and this time + * take unscrubbed pages if available. + */ + if ( !use_unscrubbed ) + { + use_unscrubbed = 1; + goto again; + } + /* No suitable memory blocks. Fail the request. */ spin_unlock(&heap_lock); return NULL; found: + need_scrub = !!test_bit(_PGC_need_scrub, &pg->count_info); + /* We may have to halve the chunk a number of times. */ while ( j != order ) { PFN_ORDER(pg) = --j; - page_list_add(pg, &heap(node, zone, j)); + page_list_add_scrub(pg, node, zone, j, need_scrub); pg += 1 << j; } + if ( need_scrub ) + pg->count_info |= PGC_need_scrub; ASSERT(avail[node][zone] >= request); avail[node][zone] -= request; @@ -837,6 +867,15 @@ static struct page_info *alloc_heap_pages( if ( d != NULL ) d->last_alloc_node = node; + if ( need_scrub ) + { + for ( i = 0; i < (1 << order); i++ ) + scrub_one_page(&pg[i]); + pg->count_info &= ~PGC_need_scrub; + node_need_scrub[node] -= (1 << order); + } + + for ( i = 0; i < (1 << order); i++ ) { /* Reference count must continuously be zero for free pages. */