From patchwork Tue Aug 8 21:45:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9889135 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EF4FE60352 for ; Tue, 8 Aug 2017 21:46:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E1B7A2890F for ; Tue, 8 Aug 2017 21:46:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D670B2892F; Tue, 8 Aug 2017 21:46:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1B3AA28A2D for ; Tue, 8 Aug 2017 21:46:10 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dfCHC-00082f-5z; Tue, 08 Aug 2017 21:42:58 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dfCHA-00081g-Mg for xen-devel@lists.xen.org; Tue, 08 Aug 2017 21:42:56 +0000 Received: from [193.109.254.147] by server-11.bemta-6.messagelabs.com id A0/B5-03612-F503A895; Tue, 08 Aug 2017 21:42:55 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprAIsWRWlGSWpSXmKPExsUyZ7p8oG68QVe kwaWNghZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8aS43vYCn6pVPx+v4WpgbFDuouRi0NIYAKT xOKrKxkhnD+MEo0397FDOOsZJea+nMIM4fQwSmyc/hrI4eRgEzCSOHt0OiOILSIgLXHt82Wwd maBaUwSe7+9A0sIC7hL7G/4zAZiswioSsy5/gOsmVfAS2Lr924WEFtCQEFiysP3YHFOAW+JmR f+gsWFgGpmTJ7KDFFjKHH64TbGCYx8CxgZVjFqFKcWlaUW6Roa6CUVZaZnlOQmZuYAeWZ6uan FxYnpqTmJScV6yfm5mxiB4cIABDsYj7+PO8QoycGkJMq7SbszUogvKT+lMiOxOCO+qDQntfgQ owwHh5IEb51+V6SQYFFqempFWmYOMHBh0hIcPEoivO0gad7igsTc4sx0iNQpRmOODavXf2Hie DXh/zcmIZa8/LxUKXHeLJBSAZDSjNI8uEGwiLrEKCslzMsIdJoQT0FqUW5mCar8K0ZxDkYlYd 7ZIFN4MvNK4Pa9AjqFCeiUCN9OkFNKEhFSUg2MS94IHbv25laAlPDHxL5axdXsj1QTGc8sZmZ 1/ZQyW3rOIveUq+88XnMrLj1kan99odosw6WzWb69lbe72nlhNmsGd+vvSRXnDU/d1Gkt70zk YFC8P4W1SV5tqqThHKtXfturI3wbf9vedppa9MjhWYS/QXH6g/Sdvz2/cai/O/zEsMHx6pIiJ ZbijERDLeai4kQAFb9sCKMCAAA= X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-11.tower-27.messagelabs.com!1502228573!79116410!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 4397 invoked from network); 8 Aug 2017 21:42:55 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-11.tower-27.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 8 Aug 2017 21:42:55 -0000 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v78LggJb032338 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 Aug 2017 21:42:42 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id v78LgfnM007246 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 8 Aug 2017 21:42:42 GMT Received: from abhmp0003.oracle.com (abhmp0003.oracle.com [141.146.116.9]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v78LgfuC026076; Tue, 8 Aug 2017 21:42:41 GMT Received: from ovs104.us.oracle.com (/10.149.76.204) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 08 Aug 2017 14:42:41 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Tue, 8 Aug 2017 17:45:02 -0400 Message-Id: <1502228707-31883-5-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1502228707-31883-1-git-send-email-boris.ostrovsky@oracle.com> References: <1502228707-31883-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: userv0021.oracle.com [156.151.31.71] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, julien.grall@arm.com, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCH v7 4/9] mm: Scrub pages in alloc_heap_pages() if needed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When allocating pages in alloc_heap_pages() first look for clean pages. If none is found then retry, take pages marked as unscrubbed and scrub them. Note that we shouldn't find unscrubbed pages in alloc_heap_pages() yet. However, this will become possible when we stop scrubbing from free_heap_pages() and instead do it from idle loop. Since not all allocations require clean pages (such as xenheap allocations) introduce MEMF_no_scrub flag that callers can set if they are willing to consume unscrubbed pages. Signed-off-by: Boris Ostrovsky Reviewed-by: Jan Beulich --- xen/common/page_alloc.c | 33 +++++++++++++++++++++++++++++---- xen/include/xen/mm.h | 4 +++- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 5920e87..7fa8896 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -706,6 +706,7 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, nodemask_t nodemask = d ? d->node_affinity : node_online_map; unsigned int j, zone, nodemask_retry = 0; struct page_info *pg; + bool use_unscrubbed = (memflags & MEMF_no_scrub); if ( node == NUMA_NO_NODE ) { @@ -737,8 +738,20 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, /* Find smallest order which can satisfy the request. */ for ( j = order; j <= MAX_ORDER; j++ ) + { if ( (pg = page_list_remove_head(&heap(node, zone, j))) ) - return pg; + { + /* + * We grab single pages (order=0) even if they are + * unscrubbed. Given that scrubbing one page is fairly quick + * it is not worth breaking higher orders. + */ + if ( (order == 0) || use_unscrubbed || + pg->u.free.first_dirty == INVALID_DIRTY_IDX) + return pg; + page_list_add_tail(pg, &heap(node, zone, j)); + } + } } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */ if ( (memflags & MEMF_exact_node) && req_node != NUMA_NO_NODE ) @@ -822,6 +835,10 @@ static struct page_info *alloc_heap_pages( } pg = get_free_buddy(zone_lo, zone_hi, order, memflags, d); + /* Try getting a dirty buddy if we couldn't get a clean one. */ + if ( !pg && !(memflags & MEMF_no_scrub) ) + pg = get_free_buddy(zone_lo, zone_hi, order, + memflags | MEMF_no_scrub, d); if ( !pg ) { /* No suitable memory blocks. Fail the request. */ @@ -867,7 +884,15 @@ static struct page_info *alloc_heap_pages( for ( i = 0; i < (1 << order); i++ ) { /* Reference count must continuously be zero for free pages. */ - BUG_ON(pg[i].count_info != PGC_state_free); + BUG_ON((pg[i].count_info & ~PGC_need_scrub) != PGC_state_free); + + if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) + { + if ( !(memflags & MEMF_no_scrub) ) + scrub_one_page(&pg[i]); + node_need_scrub[node]--; + } + pg[i].count_info = PGC_state_inuse; if ( !(memflags & MEMF_no_tlbflush) ) @@ -1743,7 +1768,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) ASSERT(!in_irq()); pg = alloc_heap_pages(MEMZONE_XEN, MEMZONE_XEN, - order, memflags, NULL); + order, memflags | MEMF_no_scrub, NULL); if ( unlikely(pg == NULL) ) return NULL; @@ -1793,7 +1818,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) if ( !(memflags >> _MEMF_bits) ) memflags |= MEMF_bits(xenheap_bits); - pg = alloc_domheap_pages(NULL, order, memflags); + pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub); if ( unlikely(pg == NULL) ) return NULL; diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index 503b92e..e1f9c42 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -248,7 +248,9 @@ struct npfec { #define MEMF_no_tlbflush (1U<<_MEMF_no_tlbflush) #define _MEMF_no_icache_flush 7 #define MEMF_no_icache_flush (1U<<_MEMF_no_icache_flush) -#define _MEMF_node 8 +#define _MEMF_no_scrub 8 +#define MEMF_no_scrub (1U<<_MEMF_no_scrub) +#define _MEMF_node 16 #define MEMF_node_mask ((1U << (8 * sizeof(nodeid_t))) - 1) #define MEMF_node(n) ((((n) + 1) & MEMF_node_mask) << _MEMF_node) #define MEMF_get_node(f) ((((f) >> _MEMF_node) - 1) & MEMF_node_mask)