From patchwork Thu Jan 30 14:57:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Durrant X-Patchwork-Id: 11358207 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 94F1F924 for ; Thu, 30 Jan 2020 14:59:40 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 65BD1206F0 for ; Thu, 30 Jan 2020 14:59:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="toyc+Dg0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65BD1206F0 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ixBH3-0003vp-7w; Thu, 30 Jan 2020 14:58:29 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ixBH1-0003vX-J2 for xen-devel@lists.xenproject.org; Thu, 30 Jan 2020 14:58:27 +0000 X-Inumbo-ID: f82f2658-4370-11ea-8a92-12813bfff9fa Received: from smtp-fw-9101.amazon.com (unknown [207.171.184.25]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id f82f2658-4370-11ea-8a92-12813bfff9fa; Thu, 30 Jan 2020 14:58:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1580396307; x=1611932307; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4L/z/yQqPy+BWKnCL2PCLgYINPiu+gxFMPEXX/yEHyc=; b=toyc+Dg0RvoogD3E8L/b2Zy3tmEPa674LEKqTa+dJaOV3ESwcNCiEG1K RO2x7XzZwzX547jVsu3aIk0P3lvAT9dIzZK2LqCAX8DDYNP0oRPv2c2ja DSZp6/Vmm489Z2GQI4Pd2Ft2vF4BrzjTsx+Ng+yaLeMEytskYUQiofB1V s=; IronPort-SDR: 66C2zhC5hLWd1rgLXkEh4OV2zREpWFA2E9CLgmuDEO2Lb01CFS7mtbwQwPByxLKK7GU03KtV2I T6UsKjv/Arhw== X-IronPort-AV: E=Sophos;i="5.70,382,1574121600"; d="scan'208";a="13644135" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1e-62350142.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 30 Jan 2020 14:58:25 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-62350142.us-east-1.amazon.com (Postfix) with ESMTPS id 847B0A2663; Thu, 30 Jan 2020 14:58:21 +0000 (UTC) Received: from EX13D32EUB003.ant.amazon.com (10.43.166.165) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Thu, 30 Jan 2020 14:58:04 +0000 Received: from EX13MTAUWC001.ant.amazon.com (10.43.162.135) by EX13D32EUB003.ant.amazon.com (10.43.166.165) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Thu, 30 Jan 2020 14:58:03 +0000 Received: from u2f063a87eabd5f.cbg10.amazon.com (10.125.106.135) by mail-relay.amazon.com (10.43.162.232) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Thu, 30 Jan 2020 14:58:00 +0000 From: Paul Durrant To: Date: Thu, 30 Jan 2020 14:57:44 +0000 Message-ID: <20200130145745.1306-4-pdurrant@amazon.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200130145745.1306-1-pdurrant@amazon.com> References: <20200130145745.1306-1-pdurrant@amazon.com> MIME-Version: 1.0 Precedence: Bulk Subject: [Xen-devel] [PATCH v8 3/4] mm: make pages allocated with MEMF_no_refcount safe to assign X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Julien Grall , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Paul Durrant , Ian Jackson , Volodymyr Babchuk , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Currently it is unsafe to assign a domheap page allocated with MEMF_no_refcount to a domain because the domain't 'tot_pages' will not be incremented, but will be decrement when the page is freed (since free_domheap_pages() has no way of telling that the increment was skipped). This patch allocates a new 'count_info' bit for a PGC_extra flag which is then used to mark pages when alloc_domheap_pages() is called with MEMF_no_refcount. assign_pages() because it still needs to call domain_adjust_tot_pages() to make sure the domain is appropriately referenced. Hence it is modified to do that for PGC_extra pages even if it is passed MEMF_no_refount. The number of PGC_extra pages assigned to a domain is tracked in a new 'extra_pages' counter, which is then subtracted from 'total_pages' in the domain_tot_pages() helper. Thus 'normal' page assignments will still be appropriately checked against 'max_pages'. Signed-off-by: Paul Durrant Reviewed-by: Jan Beulich --- Cc: Andrew Cooper Cc: George Dunlap Cc: Ian Jackson Cc: Jan Beulich Cc: Julien Grall Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini Cc: Wei Liu Cc: Volodymyr Babchuk Cc: "Roger Pau Monné" v8: - Drop the idea of post-allocation assignment adding an error path to steal_page() if it encounters a PGC_extra page - Tighten up the ASSERTs in assign_pages() v7: - s/PGC_no_refcount/PGC_extra/g - Re-work allocation to account for 'extra' pages, also making it safe to assign PGC_extra pages post-allocation v6: - Add an extra ASSERT into assign_pages() that PGC_no_refcount is not set if MEMF_no_refcount is clear - ASSERT that count_info is 0 in alloc_domheap_pages() and set to PGC_no_refcount rather than ORing v5: - Make sure PGC_no_refcount is set before assign_pages() is called - Don't bother to clear PGC_no_refcount in free_domheap_pages() and drop ASSERT in free_heap_pages() - Don't latch count_info in free_heap_pages() v4: - New in v4 --- xen/arch/x86/mm.c | 3 +- xen/common/page_alloc.c | 63 +++++++++++++++++++++++++++++++--------- xen/include/asm-arm/mm.h | 5 +++- xen/include/asm-x86/mm.h | 7 +++-- xen/include/xen/sched.h | 5 +++- 5 files changed, 64 insertions(+), 19 deletions(-) diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c index 8bb66cf30c..2796161c1f 100644 --- a/xen/arch/x86/mm.c +++ b/xen/arch/x86/mm.c @@ -4217,7 +4217,8 @@ int steal_page( if ( !(owner = page_get_owner_and_reference(page)) ) goto fail; - if ( owner != d || is_xen_heap_page(page) ) + if ( owner != d || is_xen_heap_page(page) || + (page->count_info & PGC_extra) ) goto fail_put; /* diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index bbd3163909..1ac9d9c719 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -2267,7 +2267,29 @@ int assign_pages( goto out; } - if ( !(memflags & MEMF_no_refcount) ) +#ifndef NDEBUG + { + unsigned int extra_pages = 0; + + for ( i = 0; i < (1ul << order); i++ ) + { + ASSERT(!(pg[i].count_info & ~PGC_extra)); + if ( pg[i].count_info & PGC_extra ) + extra_pages++; + } + + ASSERT(!extra_pages || + ((memflags & MEMF_no_refcount) && + extra_pages == 1u << order)); + } +#endif + + if ( pg[0].count_info & PGC_extra ) + { + d->extra_pages += 1u << order; + memflags &= ~MEMF_no_refcount; + } + else if ( !(memflags & MEMF_no_refcount) ) { unsigned int tot_pages = domain_tot_pages(d) + (1 << order); @@ -2278,18 +2300,19 @@ int assign_pages( rc = -E2BIG; goto out; } + } - if ( unlikely(domain_adjust_tot_pages(d, 1 << order) == (1 << order)) ) + if ( !(memflags & MEMF_no_refcount) && + unlikely(domain_adjust_tot_pages(d, 1 << order) == (1 << order)) ) get_knownalive_domain(d); - } for ( i = 0; i < (1 << order); i++ ) { ASSERT(page_get_owner(&pg[i]) == NULL); - ASSERT(!pg[i].count_info); page_set_owner(&pg[i], d); smp_wmb(); /* Domain pointer must be visible before updating refcnt. */ - pg[i].count_info = PGC_allocated | 1; + pg[i].count_info = + (pg[i].count_info & PGC_extra) | PGC_allocated | 1; page_list_add_tail(&pg[i], &d->page_list); } @@ -2315,11 +2338,6 @@ struct page_info *alloc_domheap_pages( if ( memflags & MEMF_no_owner ) memflags |= MEMF_no_refcount; - else if ( (memflags & MEMF_no_refcount) && d ) - { - ASSERT(!(memflags & MEMF_no_refcount)); - return NULL; - } if ( !dma_bitsize ) memflags &= ~MEMF_no_dma; @@ -2332,11 +2350,23 @@ struct page_info *alloc_domheap_pages( memflags, d)) == NULL)) ) return NULL; - if ( d && !(memflags & MEMF_no_owner) && - assign_pages(d, pg, order, memflags) ) + if ( d && !(memflags & MEMF_no_owner) ) { - free_heap_pages(pg, order, memflags & MEMF_no_scrub); - return NULL; + if ( memflags & MEMF_no_refcount ) + { + unsigned long i; + + for ( i = 0; i < (1ul << order); i++ ) + { + ASSERT(!pg[i].count_info); + pg[i].count_info = PGC_extra; + } + } + if ( assign_pages(d, pg, order, memflags) ) + { + free_heap_pages(pg, order, memflags & MEMF_no_scrub); + return NULL; + } } return pg; @@ -2384,6 +2414,11 @@ void free_domheap_pages(struct page_info *pg, unsigned int order) BUG(); } arch_free_heap_page(d, &pg[i]); + if ( pg[i].count_info & PGC_extra ) + { + ASSERT(d->extra_pages); + d->extra_pages--; + } } drop_dom_ref = !domain_adjust_tot_pages(d, -(1 << order)); diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h index 333efd3a60..7df91280bc 100644 --- a/xen/include/asm-arm/mm.h +++ b/xen/include/asm-arm/mm.h @@ -119,9 +119,12 @@ struct page_info #define PGC_state_offlined PG_mask(2, 9) #define PGC_state_free PG_mask(3, 9) #define page_state_is(pg, st) (((pg)->count_info&PGC_state) == PGC_state_##st) +/* Page is not reference counted */ +#define _PGC_extra PG_shift(10) +#define PGC_extra PG_mask(1, 10) /* Count of references to this frame. */ -#define PGC_count_width PG_shift(9) +#define PGC_count_width PG_shift(10) #define PGC_count_mask ((1UL<count_info&PGC_state) == PGC_state_##st) +/* Page is not reference counted */ +#define _PGC_extra PG_shift(10) +#define PGC_extra PG_mask(1, 10) - /* Count of references to this frame. */ -#define PGC_count_width PG_shift(9) +/* Count of references to this frame. */ +#define PGC_count_width PG_shift(10) #define PGC_count_mask ((1UL<tot_pages; + ASSERT(d->extra_pages <= d->tot_pages); + + return d->tot_pages - d->extra_pages; } /* Protect updates/reads (resp.) of domain_list and domain_hash. */