From patchwork Fri Oct 2 15:44:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813887 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 330E113B2 for ; Fri, 2 Oct 2020 15:44:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 15C2B2085B for ; Fri, 2 Oct 2020 15:44:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="iPAuiy0N" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388082AbgJBPof (ORCPT ); Fri, 2 Oct 2020 11:44:35 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:64604 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388042AbgJBPo2 (ORCPT ); Fri, 2 Oct 2020 11:44:28 -0400 Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FgwPJ170905 for ; Fri, 2 Oct 2020 11:44:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=WZCOOR6Tso02f7vzcm7TNgr9Sxlc2Kv8yjQ1TOfgxzk=; b=iPAuiy0NWB3yq//sdPSVnvGBUYRVwt4CO4UqZ0PlJ6RTsd1dlrQ0DClmxFQ4nCuqk7fN 3lgpvzuhyi4+OU+3z/FwolXeDid8bd0qC448yMyWCd9SOrHwu5lWENYAOmoc1/tAqebO FEEYpUfWsuRZdMq8IWrvVaUdvq7MOk2e1iowYIs3GGuNcJHANGz1Xb+wdLSftPak5soa SVZ4/98OEEPtgFux+fQG9QlYseLP77TQxA/c+lG5x7H/VLbhmV14yZ9BeOWiHjOySJHl gXMfh01RLBPmdXLHDF8eSQwR42fWrb5ABKq34IRkT/gMQkYE6ArCx8vQmLzYiNAYIFJY qg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x73900ya-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:27 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FiM3n174298 for ; Fri, 2 Oct 2020 11:44:27 -0400 Received: from ppma01fra.de.ibm.com (46.49.7a9f.ip4.static.sl-reverse.com [159.122.73.70]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x73900x2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:27 -0400 Received: from pps.filterd (ppma01fra.de.ibm.com [127.0.0.1]) by ppma01fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092FiOwr003243; Fri, 2 Oct 2020 15:44:24 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma01fra.de.ibm.com with ESMTP id 33sw98bgyk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:24 +0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiLS432244042 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:22 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DC1AD42047; Fri, 2 Oct 2020 15:44:21 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7BF2942041; Fri, 2 Oct 2020 15:44:21 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:21 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 1/7] lib/list: Add double linked list management functions Date: Fri, 2 Oct 2020 17:44:14 +0200 Message-Id: <20201002154420.292134-2-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 mlxscore=0 bulkscore=0 phishscore=0 clxscore=1015 adultscore=0 mlxlogscore=944 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020119 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add simple double linked lists. Apart from the struct itself, there are functions to add and remove items, and check for emptyness. Signed-off-by: Claudio Imbrenda --- lib/list.h | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 lib/list.h diff --git a/lib/list.h b/lib/list.h new file mode 100644 index 0000000..702a78c --- /dev/null +++ b/lib/list.h @@ -0,0 +1,53 @@ +#ifndef LIST_H +#define LIST_H + +#include + +/* + * Simple double linked list. The pointer to the list is a list item itself, + * like in the kernel implementation. + */ +struct linked_list { + struct linked_list *prev; + struct linked_list *next; +}; + +/* + * An empty list is a list item whose prev and next both point to itself. + * Returns true if the list is empty. + */ +static inline bool is_list_empty(struct linked_list *p) +{ + return !p->next || !p->prev || p == p->next || p == p->prev; +} + +/* + * Remove the given element from the list, if the list is not already empty. + * The removed element is returned. + */ +static inline struct linked_list *list_remove(struct linked_list *l) +{ + if (is_list_empty(l)) + return NULL; + + l->prev->next = l->next; + l->next->prev = l->prev; + l->prev = l->next = NULL; + + return l; +} + +/* + * Add the given element after the given list head. + */ +static inline void list_add(struct linked_list *head, struct linked_list *li) +{ + assert(li); + assert(head); + li->prev = head; + li->next = head->next; + head->next->prev = li; + head->next = li; +} + +#endif From patchwork Fri Oct 2 15:44:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813895 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6DC8D92C for ; Fri, 2 Oct 2020 15:44:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4FB0320795 for ; Fri, 2 Oct 2020 15:44:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="OHzto9H0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388062AbgJBPod (ORCPT ); Fri, 2 Oct 2020 11:44:33 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:26278 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388040AbgJBPo2 (ORCPT ); Fri, 2 Oct 2020 11:44:28 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FWm4M142555 for ; Fri, 2 Oct 2020 11:44:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=XbOg4Q6+DrwFlErUmMzG4UbfnyVXPsTfI5UU2re81SM=; b=OHzto9H0s1U+KqqjKnVf52eL1H7zCoduu5indWGH7QGGuP6pwvd6QatgcW6sA9MrNmZr fMXTKhpoMzvCZTRHL9wmUSlvWUJ2RVSyU2lLEeE7serLEgg0eoJwweIIzwRZegJ1e8hq /2jT13CCFpOmuXiALnzvcSeMAwFVEi6g5nAHG4f2zv1+vCyFlhbH5SKQrAvHCDpqVZm0 S2eQqOq8L4AtX9WdhMpoYFptDnG5LUk3JwrgyMnKWtKkeVhGXoDmO9Pu7WSM2KBPduTb GLJAzHmYoN4ugRpzG3G43H+QlyyfgHGBVemUQKampd4xsAiEOi1fqMrjErbb0ZkbTYUU yA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x6sv0m3m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:27 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FXscf150831 for ; Fri, 2 Oct 2020 11:44:27 -0400 Received: from ppma06fra.de.ibm.com (48.49.7a9f.ip4.static.sl-reverse.com [159.122.73.72]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x6sv0m2r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:26 -0400 Received: from pps.filterd (ppma06fra.de.ibm.com [127.0.0.1]) by ppma06fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092FhF23011091; Fri, 2 Oct 2020 15:44:24 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma06fra.de.ibm.com with ESMTP id 33svwguhm5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:24 +0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiMfU33817004 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:22 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5433D4203F; Fri, 2 Oct 2020 15:44:22 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ED91442041; Fri, 2 Oct 2020 15:44:21 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:21 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 2/7] lib/vmalloc: vmalloc support for handling allocation metadata Date: Fri, 2 Oct 2020 17:44:15 +0200 Message-Id: <20201002154420.292134-3-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 mlxscore=0 clxscore=1015 mlxlogscore=999 adultscore=0 bulkscore=0 spamscore=0 malwarescore=0 phishscore=0 suspectscore=2 priorityscore=1501 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020119 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add allocation metadata handling to vmalloc. In upcoming patches, allocation metadata will have to be handled directly bt the lower level allocators, and will not be handled by the common wrapper. In this patch, the number of allocated pages plus a magic value are written immediately before the returned pointer. This means that multi page allocations will allocate one extra page (which is not worse than what the current allocator does). For small allocations there is an optimization: the returned address is intentionally not page-aligned. This signals that the allocation spanned one page only. In this case the metadata is only the magic value, and it is also saved immediately before the returned pointer. Since the pointer does not point to the begininng of the page, there is always space in the same page for the magic value. Signed-off-by: Claudio Imbrenda --- lib/vmalloc.c | 105 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 95 insertions(+), 10 deletions(-) diff --git a/lib/vmalloc.c b/lib/vmalloc.c index e0c7b6b..2f25734 100644 --- a/lib/vmalloc.c +++ b/lib/vmalloc.c @@ -15,6 +15,16 @@ #include #include "vmalloc.h" +#define VM_MAGIC 0x7E57C0DE + +#define GET_METADATA(x) (((struct metadata *)(x)) - 1) +#define GET_MAGIC(x) (*((unsigned long *)(x) - 1)) + +struct metadata { + unsigned long npages; + unsigned long magic; +}; + static struct spinlock lock; static void *vfree_top = 0; static void *page_root; @@ -25,8 +35,14 @@ static void *page_root; * * nr is the number of pages to allocate * alignment_pages is the alignment of the allocation *in pages* + * metadata indicates whether an extra (unaligned) page needs to be allocated + * right before the main (aligned) allocation. + * + * The return value points to the first allocated virtual page, which will + * be the (potentially unaligned) metadata page if the metadata flag is + * specified. */ -void *alloc_vpages_aligned(ulong nr, unsigned int align_order) +static void *do_alloc_vpages(ulong nr, unsigned int align_order, bool metadata) { uintptr_t ptr; @@ -34,6 +50,8 @@ void *alloc_vpages_aligned(ulong nr, unsigned int align_order) ptr = (uintptr_t)vfree_top; ptr -= PAGE_SIZE * nr; ptr &= GENMASK_ULL(63, PAGE_SHIFT + align_order); + if (metadata) + ptr -= PAGE_SIZE; vfree_top = (void *)ptr; spin_unlock(&lock); @@ -41,6 +59,11 @@ void *alloc_vpages_aligned(ulong nr, unsigned int align_order) return (void *)ptr; } +void *alloc_vpages_aligned(ulong nr, unsigned int align_order) +{ + return do_alloc_vpages(nr, align_order, false); +} + void *alloc_vpages(ulong nr) { return alloc_vpages_aligned(nr, 0); @@ -69,35 +92,97 @@ void *vmap(phys_addr_t phys, size_t size) return mem; } +/* + * Allocate one page, for an object with specified alignment. + * The resulting pointer will be aligned to the required alignment, but + * intentionally not page-aligned. + * The metadata for single pages allocation is just the magic value, + * which is placed right before the pointer, like for bigger allocations. + */ +static void *vm_alloc_one_page(size_t alignment) +{ + void *p; + + /* this guarantees that there will be space for the magic value */ + assert(alignment >= sizeof(uintptr_t)); + assert(alignment < PAGE_SIZE); + p = alloc_vpage(); + install_page(page_root, virt_to_phys(alloc_page()), p); + p = (void *)((uintptr_t)p + alignment); + /* write the magic value right before the returned address */ + GET_MAGIC(p) = VM_MAGIC; + return p; +} + /* * Allocate virtual memory, with the specified minimum alignment. + * If the allocation fits in one page, only one page is allocated. Otherwise + * enough pages are allocated for the object, plus one to keep metadata + * information about the allocation. */ static void *vm_memalign(size_t alignment, size_t size) { + struct metadata *m; phys_addr_t pa; - void *mem, *p; + uintptr_t p; + void *mem; + size_t i; + if (!size) + return NULL; assert(is_power_of_2(alignment)); + if (alignment < sizeof(uintptr_t)) + alignment = sizeof(uintptr_t); + /* it fits in one page, allocate only one page */ + if (alignment + size <= PAGE_SIZE) + return vm_alloc_one_page(alignment); size = PAGE_ALIGN(size) / PAGE_SIZE; alignment = get_order(PAGE_ALIGN(alignment) / PAGE_SIZE); - mem = p = alloc_vpages_aligned(size, alignment); - while (size--) { + mem = do_alloc_vpages(size, alignment, true); + p = (uintptr_t)mem; + /* skip the metadata page */ + mem = (void *)(p + PAGE_SIZE); + /* + * time to actually allocate the physical pages to back our virtual + * allocation; note that we need to allocate one extra page (for the + * metadata), hence the <= + */ + for (i = 0; i <= size; i++, p += PAGE_SIZE) { pa = virt_to_phys(alloc_page()); assert(pa); - install_page(page_root, pa, p); - p += PAGE_SIZE; + install_page(page_root, pa, (void *)p); } + m = GET_METADATA(mem); + m->npages = size; + m->magic = VM_MAGIC; return mem; } static void vm_free(void *mem, size_t size) { - while (size) { - free_page(phys_to_virt(virt_to_pte_phys(page_root, mem))); - mem += PAGE_SIZE; - size -= PAGE_SIZE; + struct metadata *m; + uintptr_t ptr, end; + + /* the pointer is not page-aligned, it was a single-page allocation */ + if (!IS_ALIGNED((uintptr_t)mem, PAGE_SIZE)) { + assert(GET_MAGIC(mem) == VM_MAGIC); + ptr = virt_to_pte_phys(page_root, mem) & PAGE_MASK; + free_page(phys_to_virt(ptr)); + return; } + + /* the pointer is page-aligned, it was a multi-page allocation */ + m = GET_METADATA(mem); + assert(m->magic == VM_MAGIC); + assert(m->npages > 0); + /* free all the pages including the metadata page */ + ptr = (uintptr_t)mem - PAGE_SIZE; + end = ptr + m->npages * PAGE_SIZE; + for ( ; ptr < end; ptr += PAGE_SIZE) + free_page(phys_to_virt(virt_to_pte_phys(page_root, (void *)ptr))); + /* free the last one separately to avoid overflow issues */ + free_page(phys_to_virt(virt_to_pte_phys(page_root, (void *)ptr))); } static struct alloc_ops vmalloc_ops = { From patchwork Fri Oct 2 15:44:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813889 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0ABE113B2 for ; Fri, 2 Oct 2020 15:44:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB1D1207DE for ; Fri, 2 Oct 2020 15:44:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="LvQKEGbh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388085AbgJBPog (ORCPT ); Fri, 2 Oct 2020 11:44:36 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:31926 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388046AbgJBPo2 (ORCPT ); Fri, 2 Oct 2020 11:44:28 -0400 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FgpHk143003 for ; Fri, 2 Oct 2020 11:44:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=WnwSEraaKWEae3aWVDe/6zBtOV1Oh9BhIGV/Eyaawx0=; b=LvQKEGbhJuvLwyldoPSfY3Xz5Rg78es93Bf1XGNLS67KEMTc38WQ492/kjInDqDn2kaL +4hVc2hvZizg5U17exXvnTFg3NxOztZ/zwoi1e5TEmoEByy/fyGyNQSkqjR8W74d1No7 VBMydpsZqw193onRUpv9bqHfR48FOuPmgRDIJibLAXgXr96u2T97UpEb39nsHwZi50qx ShtFErpKanSQEHTxnukOrUeo1FZjYpnlYmJczWZni2pGI6RfUCtYxSaR97lPXemvukUn SF0InzxU+JTlhCroXEpxed2R0Z/e6FxbLuH2ZSjlWILkEzal8WFOhhwWJO+gEFzR3sDN 7g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x73br0rq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:27 -0400 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FhVI1144150 for ; Fri, 2 Oct 2020 11:44:27 -0400 Received: from ppma04fra.de.ibm.com (6a.4a.5195.ip4.static.sl-reverse.com [149.81.74.106]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x73br0r5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:27 -0400 Received: from pps.filterd (ppma04fra.de.ibm.com [127.0.0.1]) by ppma04fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092FgrLo017369; Fri, 2 Oct 2020 15:44:25 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma04fra.de.ibm.com with ESMTP id 33wgcu0jxt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:25 +0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiMTd32637358 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:22 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C92724203F; Fri, 2 Oct 2020 15:44:22 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 686FC42041; Fri, 2 Oct 2020 15:44:22 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:22 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 3/7] lib/asm: Add definitions of memory areas Date: Fri, 2 Oct 2020 17:44:16 +0200 Message-Id: <20201002154420.292134-4-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 impostorscore=0 spamscore=0 suspectscore=2 bulkscore=0 phishscore=0 adultscore=0 mlxscore=0 clxscore=1015 mlxlogscore=800 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020120 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add definitions and boundaries of memory areas for some architectures. This is needed by the next patch. Most architectures only get one generic memory area, wherease x86 and s390x get some more attention: x86 gets * lowest area (24-bit addresses) * low area (32-bit addresses) * the rest s390x gets * low area (31-bit addresses) * the rest Notice that the number indicates the order in which the areas are scanned when more than one area is indicated. The default order tries to get allocations from higher address ranges before trying lower ones. This tries to keep the precious lower addresses as free as possible. Signed-off-by: Claudio Imbrenda --- lib/asm-generic/memory_areas.h | 11 +++++++++++ lib/arm/asm/memory_areas.h | 11 +++++++++++ lib/arm64/asm/memory_areas.h | 11 +++++++++++ lib/powerpc/asm/memory_areas.h | 11 +++++++++++ lib/ppc64/asm/memory_areas.h | 11 +++++++++++ lib/s390x/asm/memory_areas.h | 17 +++++++++++++++++ lib/x86/asm/memory_areas.h | 22 ++++++++++++++++++++++ 7 files changed, 94 insertions(+) create mode 100644 lib/asm-generic/memory_areas.h create mode 100644 lib/arm/asm/memory_areas.h create mode 100644 lib/arm64/asm/memory_areas.h create mode 100644 lib/powerpc/asm/memory_areas.h create mode 100644 lib/ppc64/asm/memory_areas.h create mode 100644 lib/s390x/asm/memory_areas.h create mode 100644 lib/x86/asm/memory_areas.h diff --git a/lib/asm-generic/memory_areas.h b/lib/asm-generic/memory_areas.h new file mode 100644 index 0000000..927baa7 --- /dev/null +++ b/lib/asm-generic/memory_areas.h @@ -0,0 +1,11 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN 0 +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#endif diff --git a/lib/arm/asm/memory_areas.h b/lib/arm/asm/memory_areas.h new file mode 100644 index 0000000..927baa7 --- /dev/null +++ b/lib/arm/asm/memory_areas.h @@ -0,0 +1,11 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN 0 +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#endif diff --git a/lib/arm64/asm/memory_areas.h b/lib/arm64/asm/memory_areas.h new file mode 100644 index 0000000..927baa7 --- /dev/null +++ b/lib/arm64/asm/memory_areas.h @@ -0,0 +1,11 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN 0 +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#endif diff --git a/lib/powerpc/asm/memory_areas.h b/lib/powerpc/asm/memory_areas.h new file mode 100644 index 0000000..927baa7 --- /dev/null +++ b/lib/powerpc/asm/memory_areas.h @@ -0,0 +1,11 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN 0 +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#endif diff --git a/lib/ppc64/asm/memory_areas.h b/lib/ppc64/asm/memory_areas.h new file mode 100644 index 0000000..927baa7 --- /dev/null +++ b/lib/ppc64/asm/memory_areas.h @@ -0,0 +1,11 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN 0 +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#endif diff --git a/lib/s390x/asm/memory_areas.h b/lib/s390x/asm/memory_areas.h new file mode 100644 index 0000000..4856a27 --- /dev/null +++ b/lib/s390x/asm/memory_areas.h @@ -0,0 +1,17 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN BIT(31-12) +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_LOW_PFN 0 +#define AREA_LOW_NUMBER 1 +#define AREA_LOW 2 + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#define AREA_DMA31 AREA_LOW + +#endif diff --git a/lib/x86/asm/memory_areas.h b/lib/x86/asm/memory_areas.h new file mode 100644 index 0000000..d704df3 --- /dev/null +++ b/lib/x86/asm/memory_areas.h @@ -0,0 +1,22 @@ +#ifndef MEMORY_AREAS_H +#define MEMORY_AREAS_H + +#define AREA_NORMAL_PFN BIT(32-12) +#define AREA_NORMAL_NUMBER 0 +#define AREA_NORMAL 1 + +#define AREA_LOW_PFN BIT(24-12) +#define AREA_LOW_NUMBER 1 +#define AREA_LOW 2 + +#define AREA_LOWEST_PFN 0 +#define AREA_LOWEST_NUMBER 2 +#define AREA_LOWEST 4 + +#define AREA_DMA24 AREA_LOWEST +#define AREA_DMA32 (AREA_LOWEST | AREA_LOW) + +#define AREA_ANY -1 +#define AREA_ANY_NUMBER 0xff + +#endif From patchwork Fri Oct 2 15:44:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813883 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1324B13B2 for ; Fri, 2 Oct 2020 15:44:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E0326207DE for ; Fri, 2 Oct 2020 15:44:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="r1SMCnRG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388072AbgJBPof (ORCPT ); Fri, 2 Oct 2020 11:44:35 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:24640 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388052AbgJBPob (ORCPT ); Fri, 2 Oct 2020 11:44:31 -0400 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FgpDO143006 for ; Fri, 2 Oct 2020 11:44:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=bRV+H/6IR7geygWeBivh99r4JvUGMj1FTV1s2OTa2TE=; b=r1SMCnRGqx80pn7kKriGkFgoCvEzPejKJXI/6rnnhv/6IotZt5UGeU0C0k7lbZ+kLRVv 8f3p8YCjAwtrjVVz7D6k503Y0u4iiVMzKtbe0O6c8u232pPMeuxir7PpR7w/fXY90W0v nm57dhiT/MI7+qFgmVvB91aqLiAZkAgmQ/3G6g2KrqSDDrGWw7FD22czeKV9xb5mf0vl jlWq6lrqSSWEF/8S077k6CL4ExsOtYsyKgFZKftpOE2olhfZe7gaKPxrIPdGDjrT+lzK MPqynvHoyscaQ4zwRw466ZGdT0dXEs9lKpInV/B5UdoEq5cd5veNSntAt1CP77xdrNLo GA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x73br0rv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:28 -0400 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FhpKL145010 for ; Fri, 2 Oct 2020 11:44:28 -0400 Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x73br0r8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:27 -0400 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092FfTXQ032314; Fri, 2 Oct 2020 15:44:25 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma03ams.nl.ibm.com with ESMTP id 33sw986qcj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:25 +0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiNHp16056802 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:23 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 59D304203F; Fri, 2 Oct 2020 15:44:23 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DE35A42042; Fri, 2 Oct 2020 15:44:22 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:22 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 4/7] lib/alloc_page: complete rewrite of the page allocator Date: Fri, 2 Oct 2020 17:44:17 +0200 Message-Id: <20201002154420.292134-5-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 impostorscore=0 spamscore=0 suspectscore=2 bulkscore=0 phishscore=0 adultscore=0 mlxscore=0 clxscore=1015 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020120 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This is a complete rewrite of the page allocator. This will bring a few improvements: * no need to specify the size when freeing * allocate small areas with a large alignment without wasting memory * ability to initialize and use multiple memory areas (e.g. DMA) * more sanity checks A few things have changed: * initialization cannot be done with free_pages like before, page_alloc_init_area has to be used instead Arch-specific changes: * s390x now uses the area below 2GiB for SMP lowcore initialization. Details: Each memory area has metadata at the very beginning. The metadata is a byte array with one entry per usable page (so, excluding the metadata itself). Each entry indicates if the page is special (unused for now), if it is allocated, and the order of the block. Both free and allocated pages are part of larger blocks. Some more fixed size metadata is present in a fixed-size static array. This metadata contains start and end page frame numbers, the pointer to the metadata array, and the array of freelists. The array of freelists has an entry for each possible order (indicated by the macro NLISTS, defined as BITS_PER_LONG - PAGE_SHIFT). On allocation, if the free list for the needed size is empty, larger blocks are split. When a small allocation with a large alignment is requested, an appropriately large block is split, to guarantee the alignment. When a block is freed, an attempt will be made to merge it into the neighbour, iterating the process as long as possible. Signed-off-by: Claudio Imbrenda --- lib/alloc_page.h | 63 +++++- lib/alloc_page.c | 484 +++++++++++++++++++++++++++++++++++++---------- lib/arm/setup.c | 2 +- lib/s390x/sclp.c | 6 +- lib/s390x/smp.c | 2 +- lib/vmalloc.c | 13 +- 6 files changed, 453 insertions(+), 117 deletions(-) diff --git a/lib/alloc_page.h b/lib/alloc_page.h index 88540d1..81847ae 100644 --- a/lib/alloc_page.h +++ b/lib/alloc_page.h @@ -8,12 +8,71 @@ #ifndef ALLOC_PAGE_H #define ALLOC_PAGE_H 1 +#include + +/* Returns true if the page allocator has been initialized */ bool page_alloc_initialized(void); + +/* + * Initializes a memory area. + * n is the number of the area to initialize + * base_pfn is the physical frame number of the start of the area to initialize + * top_pfn is the physical frame number of the first page immediately after + * the end of the area to initialize + */ +void page_alloc_init_area(u8 n, uintptr_t base_pfn, uintptr_t top_pfn); + +/* Enables the page allocator. At least one area must have been initialized */ void page_alloc_ops_enable(void); + +/* + * Allocate aligned memory from the specified areas. + * areas is a bitmap of allowed areas + * alignment must be a power of 2 + */ +void *memalign_pages_area(unsigned int areas, size_t alignment, size_t size); + +/* + * Allocate aligned memory from any area. + * Equivalent to memalign_pages_area(~0, alignment, size). + */ +void *memalign_pages(size_t alignment, size_t size); + +/* + * Allocate naturally aligned memory from the specified areas. + * Equivalent to memalign_pages_area(areas, 1ull << order, 1ull << order). + */ +void *alloc_pages_area(unsigned int areas, unsigned int order); + +/* + * Allocate one page from any area. + * Equivalent to alloc_pages(0); + */ void *alloc_page(void); + +/* + * Allocate naturally aligned memory from any area. + * Equivalent to alloc_pages_area(~0, order); + */ void *alloc_pages(unsigned int order); -void free_page(void *page); + +/* + * Frees a memory block allocated with any of the memalign_pages* or + * alloc_pages* functions. + * The pointer must point to the start of the block. + */ void free_pages(void *mem, size_t size); -void free_pages_by_order(void *mem, unsigned int order); + +/* For backwards compatibility */ +static inline void free_page(void *mem) +{ + return free_pages(mem, 1); +} + +/* For backwards compatibility */ +static inline void free_pages_by_order(void *mem, unsigned int order) +{ + free_pages(mem, 1ull << order); +} #endif diff --git a/lib/alloc_page.c b/lib/alloc_page.c index 74fe726..29d221f 100644 --- a/lib/alloc_page.c +++ b/lib/alloc_page.c @@ -9,169 +9,445 @@ #include "alloc_phys.h" #include "alloc_page.h" #include "bitops.h" +#include "list.h" #include #include #include +#include +#define IS_ALIGNED_ORDER(x,order) IS_ALIGNED((x),BIT_ULL(order)) +#define NLISTS ((BITS_PER_LONG) - (PAGE_SHIFT)) +#define PFN(x) ((uintptr_t)(x) >> PAGE_SHIFT) + +#define MAX_AREAS 6 + +#define ORDER_MASK 0x3f +#define ALLOC_MASK 0x40 + +struct mem_area { + /* Physical frame number of the first usable frame in the area */ + uintptr_t base; + /* Physical frame number of the first frame outside the area */ + uintptr_t top; + /* Combination ALLOC_MASK and order */ + u8 *page_states; + /* One freelist for each possible block size, up to NLISTS */ + struct linked_list freelists[NLISTS]; +}; + +static struct mem_area areas[MAX_AREAS]; +static unsigned int areas_mask; static struct spinlock lock; -static void *freelist = 0; bool page_alloc_initialized(void) { - return freelist != 0; + return areas_mask != 0; } -void free_pages(void *mem, size_t size) +static inline bool area_or_metadata_contains(struct mem_area *a, uintptr_t pfn) { - void *old_freelist; - void *end; + return (pfn >= PFN(a->page_states)) && (pfn < a->top); +} - assert_msg((unsigned long) mem % PAGE_SIZE == 0, - "mem not page aligned: %p", mem); +static inline bool area_contains(struct mem_area *a, uintptr_t pfn) +{ + return (pfn >= a->base) && (pfn < a->top); +} - assert_msg(size % PAGE_SIZE == 0, "size not page aligned: %#zx", size); +/* + * Splits the free block starting at addr into 2 blocks of half the size. + * + * The function depends on the following assumptions: + * - The allocator must have been initialized + * - the block must be within the memory area + * - all pages in the block must be free and not special + * - the pointer must point to the start of the block + * - all pages in the block must have the same block size. + * - the block size must be greater than 0 + * - the block size must be smaller than the maximum allowed + * - the block must be in a free list + * - the function is called with the lock held + */ +static void split(struct mem_area *a, void *addr) +{ + uintptr_t pfn = PFN(addr); + struct linked_list *p; + uintptr_t i, idx; + u8 order; - assert_msg(size == 0 || (uintptr_t)mem == -size || - (uintptr_t)mem + size > (uintptr_t)mem, - "mem + size overflow: %p + %#zx", mem, size); + assert(a && area_contains(a, pfn)); + idx = pfn - a->base; + order = a->page_states[idx]; + assert(!(order & ~ORDER_MASK) && order && (order < NLISTS)); + assert(IS_ALIGNED_ORDER(pfn, order)); + assert(area_contains(a, pfn + BIT(order) - 1)); - if (size == 0) { - freelist = NULL; - return; - } + /* Remove the block from its free list */ + p = list_remove(addr); + assert(p); - spin_lock(&lock); - old_freelist = freelist; - freelist = mem; - end = mem + size; - while (mem + PAGE_SIZE != end) { - *(void **)mem = (mem + PAGE_SIZE); - mem += PAGE_SIZE; + /* update the block size for each page in the block */ + for (i = 0; i < BIT(order); i++) { + assert(a->page_states[idx + i] == order); + a->page_states[idx + i] = order - 1; } - - *(void **)mem = old_freelist; - spin_unlock(&lock); + order--; + /* add the first half block to the appropriate free list */ + list_add(a->freelists + order, p); + /* add the second half block to the appropriate free list */ + list_add(a->freelists + order, (void *)((pfn + BIT(order)) * PAGE_SIZE)); } -void free_pages_by_order(void *mem, unsigned int order) +/* + * Returns a block whose alignment and size are at least the parameter values. + * If there is not enough free memory, NULL is returned. + * + * Both parameters must be not larger than the largest allowed order + */ +static void *page_memalign_order(struct mem_area *a, u8 al, u8 sz) { - free_pages(mem, 1ul << (order + PAGE_SHIFT)); + struct linked_list *p, *res = NULL; + u8 order; + + assert((al < NLISTS) && (sz < NLISTS)); + /* we need the bigger of the two as starting point */ + order = sz > al ? sz : al; + + /* search all free lists for some memory */ + for ( ; order < NLISTS; order++) { + p = a->freelists[order].next; + if (!is_list_empty(p)) + break; + } + /* out of memory */ + if (order >= NLISTS) + return NULL; + + /* + * the block is bigger than what we need because either there were + * no smaller blocks, or the smaller blocks were not aligned to our + * needs; therefore we split the block until we reach the needed size + */ + for (; order > sz; order--) + split(a, p); + + res = list_remove(p); + memset(a->page_states + (PFN(res) - a->base), ALLOC_MASK | order, BIT(order)); + return res; } -void *alloc_page() +/* + * Try to merge two blocks into a bigger one. + * Returns true in case of a successful merge. + * Merging will succeed only if both blocks have the same block size and are + * both free. + * + * The function depends on the following assumptions: + * - the first parameter is strictly smaller than the second + * - the parameters must point each to the start of their block + * - the two parameters point to adjacent blocks + * - the two blocks are both in a free list + * - all of the pages of the two blocks must be free + * - all of the pages of the two blocks must have the same block size + * - the function is called with the lock held + */ +static bool coalesce(struct mem_area *a, u8 order, uintptr_t pfn, uintptr_t pfn2) { - void *p; + uintptr_t first, second, i; + struct linked_list *li; - if (!freelist) - return 0; + assert(IS_ALIGNED_ORDER(pfn, order) && IS_ALIGNED_ORDER(pfn2, order)); + assert(pfn2 == pfn + BIT(order)); + assert(a); - spin_lock(&lock); - p = freelist; - freelist = *(void **)freelist; - spin_unlock(&lock); + /* attempting to coalesce two blocks that belong to different areas */ + if (!area_contains(a, pfn) || !area_contains(a, pfn2 + BIT(order) - 1)) + return false; + first = pfn - a->base; + second = pfn2 - a->base; + /* the two blocks have different sizes, cannot coalesce */ + if ((a->page_states[first] != order) || (a->page_states[second] != order)) + return false; - if (p) - memset(p, 0, PAGE_SIZE); - return p; + /* we can coalesce, remove both blocks from their freelists */ + li = list_remove((void *)(pfn2 << PAGE_SHIFT)); + assert(li); + li = list_remove((void *)(pfn << PAGE_SHIFT)); + assert(li); + /* check the metadata entries and update with the new size */ + for (i = 0; i < (2ull << order); i++) { + assert(a->page_states[first + i] == order); + a->page_states[first + i] = order + 1; + } + /* finally add the newly coalesced block to the appropriate freelist */ + list_add(a->freelists + order + 1, li); + return true; } /* - * Allocates (1 << order) physically contiguous and naturally aligned pages. - * Returns NULL if there's no memory left. + * Free a block of memory. + * The parameter can be NULL, in which case nothing happens. + * + * The function depends on the following assumptions: + * - the parameter is page aligned + * - the parameter belongs to an existing memory area + * - the parameter points to the beginning of the block + * - the size of the block is less than the maximum allowed + * - the block is completely contained in its memory area + * - all pages in the block have the same block size + * - no pages in the memory block were already free + * - no pages in the memory block are special */ -void *alloc_pages(unsigned int order) +static void _free_pages(void *mem) { - /* Generic list traversal. */ - void *prev; - void *curr = NULL; - void *next = freelist; - - /* Looking for a run of length (1 << order). */ - unsigned long run = 0; - const unsigned long n = 1ul << order; - const unsigned long align_mask = (n << PAGE_SHIFT) - 1; - void *run_start = NULL; - void *run_prev = NULL; - unsigned long run_next_pa = 0; - unsigned long pa; + uintptr_t pfn2, pfn = PFN(mem); + struct mem_area *a = NULL; + uintptr_t i, p; + u8 order; - assert(order < sizeof(unsigned long) * 8); + if (!mem) + return; + assert(IS_ALIGNED((uintptr_t)mem, PAGE_SIZE)); - spin_lock(&lock); - for (;;) { - prev = curr; - curr = next; + /* find which area this pointer belongs to*/ + for (i = 0; !a && (i < MAX_AREAS); i++) { + if ((areas_mask & BIT(i)) && area_contains(areas + i, pfn)) + a = areas + i; + } + assert_msg(a, "memory does not belong to any area: %p", mem); - if (!curr) { - run_start = NULL; - break; - } + p = pfn - a->base; + order = a->page_states[p] & ORDER_MASK; - next = *((void **) curr); - pa = virt_to_phys(curr); - - if (run == 0) { - if (!(pa & align_mask)) { - run_start = curr; - run_prev = prev; - run_next_pa = pa + PAGE_SIZE; - run = 1; - } - } else if (pa == run_next_pa) { - run_next_pa += PAGE_SIZE; - run += 1; - } else { - run = 0; - } + /* ensure that the first page is allocated and not special */ + assert(a->page_states[p] == (order | ALLOC_MASK)); + /* ensure that the order has a sane value */ + assert(order < NLISTS); + /* ensure that the block is aligned properly for its size */ + assert(IS_ALIGNED_ORDER(pfn, order)); + /* ensure that the area can contain the whole block */ + assert(area_contains(a, pfn + BIT(order) - 1)); - if (run == n) { - if (run_prev) - *((void **) run_prev) = next; - else - freelist = next; - break; - } + for (i = 0; i < BIT(order); i++) { + /* check that all pages of the block have consistent metadata */ + assert(a->page_states[p + i] == (ALLOC_MASK | order)); + /* set the page as free */ + a->page_states[p + i] &= ~ALLOC_MASK; } - spin_unlock(&lock); - if (run_start) - memset(run_start, 0, n * PAGE_SIZE); - return run_start; + /* provisionally add the block to the appropriate free list */ + list_add(a->freelists + order, mem); + /* try to coalesce the block with neighbouring blocks if possible */ + do { + /* + * get the order again since it might have changed after + * coalescing in a previous iteration + */ + order = a->page_states[p] & ORDER_MASK; + /* + * let's consider this block and the next one if this block + * is aligned to the next size, otherwise let's consider the + * previous block and this one + */ + if (!IS_ALIGNED_ORDER(pfn, order + 1)) + pfn = pfn - BIT(order); + pfn2 = pfn + BIT(order); + /* repeat as long as we manage to coalesce something */ + } while (coalesce(a, order, pfn, pfn2)); } +void free_pages(void *mem, size_t size) +{ + spin_lock(&lock); + _free_pages(mem); + spin_unlock(&lock); +} -void free_page(void *page) +static void *page_memalign_order_area(unsigned area, u8 ord, u8 al) { + void *res = NULL; + int i; + spin_lock(&lock); - *(void **)page = freelist; - freelist = page; + area &= areas_mask; + for (i = 0; !res && (i < MAX_AREAS); i++) + if (area & BIT(i)) + res = page_memalign_order(areas + i, ord, al); spin_unlock(&lock); + return res; } -static void *page_memalign(size_t alignment, size_t size) +/* + * Allocates (1 << order) physically contiguous and naturally aligned pages. + * Returns NULL if the allocation was not possible. + */ +void *alloc_pages_area(unsigned int area, unsigned int order) { - unsigned long n = ALIGN(size, PAGE_SIZE) >> PAGE_SHIFT; - unsigned int order; + return page_memalign_order_area(area, order, order); +} - if (!size) - return NULL; +void *alloc_pages(unsigned int order) +{ + return alloc_pages_area(AREA_ANY, order); +} - order = get_order(n); +/* + * Allocates (1 << order) physically contiguous aligned pages. + * Returns NULL if the allocation was not possible. + */ +void *memalign_pages_area(unsigned int area, size_t alignment, size_t size) +{ + assert(is_power_of_2(alignment)); + alignment = get_order(PAGE_ALIGN(alignment) >> PAGE_SHIFT); + size = get_order(PAGE_ALIGN(size) >> PAGE_SHIFT); + assert(alignment < NLISTS); + assert(size < NLISTS); + return page_memalign_order_area(area, size, alignment); +} - return alloc_pages(order); +void *memalign_pages(size_t alignment, size_t size) +{ + return memalign_pages_area(AREA_ANY, alignment, size); } -static void page_free(void *mem, size_t size) +/* + * Allocates one page + */ +void *alloc_page() { - free_pages(mem, size); + return alloc_pages(0); } static struct alloc_ops page_alloc_ops = { - .memalign = page_memalign, - .free = page_free, + .memalign = memalign_pages, + .free = free_pages, .align_min = PAGE_SIZE, }; +/* + * Enables the page allocator. + * + * Prerequisites: + * - at least one memory area has been initialized + */ void page_alloc_ops_enable(void) { + spin_lock(&lock); + assert(page_alloc_initialized()); alloc_ops = &page_alloc_ops; + spin_unlock(&lock); +} + +/* + * Adds a new memory area to the pool of available memory. + * + * Prerequisites: + * - the lock is held + * - start and top are page frame numbers + * - start is smaller than top + * - top does not fall outside of addressable memory + * - there is at least one more slot free for memory areas + * - if a specific memory area number has been indicated, it needs to be free + * - the memory area to add does not overlap with existing areas + * - the memory area to add has at least 5 pages available + */ +static void _page_alloc_init_area(u8 n, uintptr_t start_pfn, uintptr_t top_pfn) +{ + size_t table_size, npages, i; + struct mem_area *a; + u8 order = 0; + + /* the number must be within the allowed range */ + assert(n < MAX_AREAS); + /* the new area number must be unused */ + assert(!(areas_mask & BIT(n))); + + /* other basic sanity checks */ + assert(top_pfn > start_pfn); + assert(top_pfn - start_pfn > 4); + assert(top_pfn < BIT_ULL(sizeof(void *) * 8 - PAGE_SHIFT)); + + /* calculate the size of the metadata table in pages */ + table_size = (top_pfn - start_pfn + PAGE_SIZE) / (PAGE_SIZE + 1); + + /* fill in the values of the new area */ + a = areas + n; + a->page_states = (void *)(start_pfn << PAGE_SHIFT); + a->base = start_pfn + table_size; + a->top = top_pfn; + npages = top_pfn - a->base; + assert((a->base - start_pfn) * PAGE_SIZE >= npages); + + /* check that the new area does not overlap with any existing areas */ + for (i = 0; i < MAX_AREAS; i++) { + if (!(areas_mask & BIT(i))) + continue; + assert(!area_or_metadata_contains(areas + i, start_pfn)); + assert(!area_or_metadata_contains(areas + i, top_pfn - 1)); + assert(!area_or_metadata_contains(a, PFN(areas[i].page_states))); + assert(!area_or_metadata_contains(a, areas[i].top - 1)); + } + /* initialize all freelists for the new area */ + for (i = 0; i < NLISTS; i++) + a->freelists[i].next = a->freelists[i].prev = a->freelists + i; + + /* initialize the metadata for the available memory */ + for (i = a->base; i < a->top; i += 1ull << order) { + /* search which order to start from */ + while (i + BIT(order) > a->top) { + assert(order); + order--; + } + /* + * we need both loops, one for the start and the other for + * the end of the block, in case it spans a power of two + * boundary + */ + while (IS_ALIGNED_ORDER(i, order + 1) && (i + BIT(order + 1) <= a->top)) + order++; + assert(order < NLISTS); + /* initialize the metadata and add to the freelist */ + memset(a->page_states + (i - a->base), order, BIT(order)); + list_add(a->freelists + order, (void *)(i << PAGE_SHIFT)); + } + /* finally mark the area as present */ + areas_mask |= BIT(n); +} + +static void __page_alloc_init_area(u8 n, uintptr_t cutoff, uintptr_t base_pfn, uintptr_t *top_pfn) +{ + if (*top_pfn > cutoff) { + spin_lock(&lock); + if (base_pfn >= cutoff) { + _page_alloc_init_area(n, base_pfn, *top_pfn); + *top_pfn = 0; + } else { + _page_alloc_init_area(n, cutoff, *top_pfn); + *top_pfn = cutoff; + } + spin_unlock(&lock); + } +} + +/* + * Adds a new memory area to the pool of available memory. + * + * Prerequisites: + * see _page_alloc_init_area + */ +void page_alloc_init_area(u8 n, uintptr_t base_pfn, uintptr_t top_pfn) +{ + if (n != AREA_ANY_NUMBER) { + __page_alloc_init_area(n, 0, base_pfn, &top_pfn); + return; + } +#ifdef AREA_HIGH_PFN + __page_alloc_init_area(AREA_HIGH_NUMBER, AREA_HIGH_PFN), base_pfn, &top_pfn); +#endif + __page_alloc_init_area(AREA_NORMAL_NUMBER, AREA_NORMAL_PFN, base_pfn, &top_pfn); +#ifdef AREA_LOW_PFN + __page_alloc_init_area(AREA_LOW_NUMBER, AREA_LOW_PFN, base_pfn, &top_pfn); +#endif +#ifdef AREA_LOWEST_PFN + __page_alloc_init_area(AREA_LOWEST_NUMBER, AREA_LOWEST_PFN, base_pfn, &top_pfn); +#endif } diff --git a/lib/arm/setup.c b/lib/arm/setup.c index 78562e4..3f03ca6 100644 --- a/lib/arm/setup.c +++ b/lib/arm/setup.c @@ -155,7 +155,7 @@ static void mem_init(phys_addr_t freemem_start) assert(sizeof(long) == 8 || !(base >> 32)); if (sizeof(long) != 8 && (top >> 32) != 0) top = ((uint64_t)1 << 32); - free_pages((void *)(unsigned long)base, top - base); + page_alloc_init_area(0, base >> PAGE_SHIFT, top >> PAGE_SHIFT); page_alloc_ops_enable(); } diff --git a/lib/s390x/sclp.c b/lib/s390x/sclp.c index 4054d0e..4e2ac18 100644 --- a/lib/s390x/sclp.c +++ b/lib/s390x/sclp.c @@ -37,11 +37,11 @@ static void mem_init(phys_addr_t mem_end) phys_alloc_init(freemem_start, mem_end - freemem_start); phys_alloc_get_unused(&base, &top); - base = (base + PAGE_SIZE - 1) & -PAGE_SIZE; - top = top & -PAGE_SIZE; + base = PAGE_ALIGN(base) >> PAGE_SHIFT; + top = top >> PAGE_SHIFT; /* Make the pages available to the physical allocator */ - free_pages((void *)(unsigned long)base, top - base); + page_alloc_init_area(AREA_ANY_NUMBER, base, top); page_alloc_ops_enable(); } diff --git a/lib/s390x/smp.c b/lib/s390x/smp.c index 2860e9c..ea93329 100644 --- a/lib/s390x/smp.c +++ b/lib/s390x/smp.c @@ -190,7 +190,7 @@ int smp_cpu_setup(uint16_t addr, struct psw psw) sigp_retry(cpu->addr, SIGP_INITIAL_CPU_RESET, 0, NULL); - lc = alloc_pages(1); + lc = alloc_pages_area(AREA_DMA31, 1); cpu->lowcore = lc; memset(lc, 0, PAGE_SIZE * 2); sigp_retry(cpu->addr, SIGP_SET_PREFIX, (unsigned long )lc, NULL); diff --git a/lib/vmalloc.c b/lib/vmalloc.c index 2f25734..3aec5ac 100644 --- a/lib/vmalloc.c +++ b/lib/vmalloc.c @@ -217,18 +217,19 @@ void setup_vm() * so that it can be used to allocate page tables. */ if (!page_alloc_initialized()) { - base = PAGE_ALIGN(base); - top = top & -PAGE_SIZE; - free_pages(phys_to_virt(base), top - base); + base = PAGE_ALIGN(base) >> PAGE_SHIFT; + top = top >> PAGE_SHIFT; + page_alloc_init_area(AREA_ANY_NUMBER, base, top); + page_alloc_ops_enable(); } find_highmem(); phys_alloc_get_unused(&base, &top); page_root = setup_mmu(top); if (base != top) { - base = PAGE_ALIGN(base); - top = top & -PAGE_SIZE; - free_pages(phys_to_virt(base), top - base); + base = PAGE_ALIGN(base) >> PAGE_SHIFT; + top = top >> PAGE_SHIFT; + page_alloc_init_area(AREA_ANY_NUMBER, base, top); } spin_lock(&lock); From patchwork Fri Oct 2 15:44:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813893 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2832F112E for ; Fri, 2 Oct 2020 15:44:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 05084207DE for ; Fri, 2 Oct 2020 15:44:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="nh3puc7v" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388065AbgJBPod (ORCPT ); Fri, 2 Oct 2020 11:44:33 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:10324 "EHLO mx0b-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388053AbgJBPo3 (ORCPT ); Fri, 2 Oct 2020 11:44:29 -0400 Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FVbUa035607 for ; Fri, 2 Oct 2020 11:44:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=4rnYkq1xA25KtIa/ibM5hIb56rez3PjIOa7BmxeeNeU=; b=nh3puc7vf0mEybRPToAAI38iJwA5AiHAxbtKOx8fV1KnKDSZHQpbDFk/JYLIsbbpl5Hh HrONRzvzJqDGfCq8FhGm0jv6OwR6dpSIqVAjO4bgQqxHnoXRgC1Pbcp/+RSYakUokudQ isx+el0rZWUmsNUEiE4VKLqIEeOkwMdZOyPoIBM9Tt2TNjdzqQZYtYnXQDsmoT1PvlT6 tvkJE2/h/+KSQPGlomZlRjK5lXyXBlnCNa3pFZQBiG9ROGxZOecR9UVZTcN7xB+GR72E Kh9aljjoQatxfVyjuqQkfDoUY57At66NQwHpzAmn3nvJcSsD9vwwR7TJI0hqUkMXKijF lw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x6wg8c3j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:28 -0400 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FW7Ad037856 for ; Fri, 2 Oct 2020 11:44:28 -0400 Received: from ppma03fra.de.ibm.com (6b.4a.5195.ip4.static.sl-reverse.com [149.81.74.107]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x6wg8c2y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:28 -0400 Received: from pps.filterd (ppma03fra.de.ibm.com [127.0.0.1]) by ppma03fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092FhGI0015509; Fri, 2 Oct 2020 15:44:26 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma03fra.de.ibm.com with ESMTP id 33v5kg1rrp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:25 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiNxL32637354 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:23 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C7EE44203F; Fri, 2 Oct 2020 15:44:23 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6BB1042041; Fri, 2 Oct 2020 15:44:23 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:23 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 5/7] lib/alloc: simplify free and malloc Date: Fri, 2 Oct 2020 17:44:18 +0200 Message-Id: <20201002154420.292134-6-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 priorityscore=1501 phishscore=0 bulkscore=0 suspectscore=2 impostorscore=0 clxscore=1015 mlxlogscore=999 spamscore=0 adultscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020119 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Remove the size parameter from the various free functions Since the backends can handle the allocation sizes on their own, simplify the generic malloc wrappers. Signed-off-by: Claudio Imbrenda --- lib/alloc.h | 2 +- lib/alloc_page.h | 6 +++--- lib/alloc.c | 42 +++++------------------------------------- lib/alloc_page.c | 2 +- lib/s390x/smp.c | 4 ++-- lib/vmalloc.c | 2 +- s390x/smp.c | 4 ++-- 7 files changed, 15 insertions(+), 47 deletions(-) diff --git a/lib/alloc.h b/lib/alloc.h index c44d459..9b4b634 100644 --- a/lib/alloc.h +++ b/lib/alloc.h @@ -24,7 +24,7 @@ struct alloc_ops { void *(*memalign)(size_t alignment, size_t size); - void (*free)(void *ptr, size_t size); + void (*free)(void *ptr); size_t align_min; }; diff --git a/lib/alloc_page.h b/lib/alloc_page.h index 81847ae..6c23018 100644 --- a/lib/alloc_page.h +++ b/lib/alloc_page.h @@ -61,18 +61,18 @@ void *alloc_pages(unsigned int order); * alloc_pages* functions. * The pointer must point to the start of the block. */ -void free_pages(void *mem, size_t size); +void free_pages(void *mem); /* For backwards compatibility */ static inline void free_page(void *mem) { - return free_pages(mem, 1); + return free_pages(mem); } /* For backwards compatibility */ static inline void free_pages_by_order(void *mem, unsigned int order) { - free_pages(mem, 1ull << order); + free_pages(mem); } #endif diff --git a/lib/alloc.c b/lib/alloc.c index 9d89d24..a46f464 100644 --- a/lib/alloc.c +++ b/lib/alloc.c @@ -50,56 +50,24 @@ void *calloc(size_t nmemb, size_t size) return ptr; } -#define METADATA_EXTRA (2 * sizeof(uintptr_t)) -#define OFS_SLACK (-2 * sizeof(uintptr_t)) -#define OFS_SIZE (-sizeof(uintptr_t)) - -static inline void *block_begin(void *mem) -{ - uintptr_t slack = *(uintptr_t *)(mem + OFS_SLACK); - return mem - slack; -} - -static inline uintptr_t block_size(void *mem) -{ - return *(uintptr_t *)(mem + OFS_SIZE); -} - void free(void *ptr) { - if (!alloc_ops->free) - return; - - void *base = block_begin(ptr); - uintptr_t sz = block_size(ptr); - - alloc_ops->free(base, sz); + if (alloc_ops->free) + alloc_ops->free(ptr); } void *memalign(size_t alignment, size_t size) { void *p; - uintptr_t blkalign; - uintptr_t mem; if (!size) return NULL; - assert(alignment >= sizeof(void *) && is_power_of_2(alignment)); + assert(is_power_of_2(alignment)); assert(alloc_ops && alloc_ops->memalign); - size += alignment - 1; - blkalign = MAX(alignment, alloc_ops->align_min); - size = ALIGN(size + METADATA_EXTRA, alloc_ops->align_min); - p = alloc_ops->memalign(blkalign, size); + p = alloc_ops->memalign(alignment, size); assert(p); - /* Leave room for metadata before aligning the result. */ - mem = (uintptr_t)p + METADATA_EXTRA; - mem = ALIGN(mem, alignment); - - /* Write the metadata */ - *(uintptr_t *)(mem + OFS_SLACK) = mem - (uintptr_t)p; - *(uintptr_t *)(mem + OFS_SIZE) = size; - return (void *)mem; + return (void *)p; } diff --git a/lib/alloc_page.c b/lib/alloc_page.c index 29d221f..046082a 100644 --- a/lib/alloc_page.c +++ b/lib/alloc_page.c @@ -255,7 +255,7 @@ static void _free_pages(void *mem) } while (coalesce(a, order, pfn, pfn2)); } -void free_pages(void *mem, size_t size) +void free_pages(void *mem) { spin_lock(&lock); _free_pages(mem); diff --git a/lib/s390x/smp.c b/lib/s390x/smp.c index ea93329..77d80ca 100644 --- a/lib/s390x/smp.c +++ b/lib/s390x/smp.c @@ -163,8 +163,8 @@ int smp_cpu_destroy(uint16_t addr) rc = smp_cpu_stop_nolock(addr, false); if (!rc) { cpu = smp_cpu_from_addr(addr); - free_pages(cpu->lowcore, 2 * PAGE_SIZE); - free_pages(cpu->stack, 4 * PAGE_SIZE); + free_pages(cpu->lowcore); + free_pages(cpu->stack); cpu->lowcore = (void *)-1UL; cpu->stack = (void *)-1UL; } diff --git a/lib/vmalloc.c b/lib/vmalloc.c index 3aec5ac..986a34c 100644 --- a/lib/vmalloc.c +++ b/lib/vmalloc.c @@ -159,7 +159,7 @@ static void *vm_memalign(size_t alignment, size_t size) return mem; } -static void vm_free(void *mem, size_t size) +static void vm_free(void *mem) { struct metadata *m; uintptr_t ptr, end; diff --git a/s390x/smp.c b/s390x/smp.c index ad30e3c..4ca1dce 100644 --- a/s390x/smp.c +++ b/s390x/smp.c @@ -143,7 +143,7 @@ static void test_store_status(void) sigp(1, SIGP_STORE_STATUS_AT_ADDRESS, (uintptr_t)status, NULL); while (!status->prefix) { mb(); } report(1, "status written"); - free_pages(status, PAGE_SIZE * 2); + free_pages(status); report_prefix_pop(); smp_cpu_stop(1); @@ -276,7 +276,7 @@ static void test_reset_initial(void) report_prefix_pop(); report(smp_cpu_stopped(1), "cpu stopped"); - free_pages(status, PAGE_SIZE); + free_pages(status); report_prefix_pop(); } From patchwork Fri Oct 2 15:44:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813885 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A66D992C for ; Fri, 2 Oct 2020 15:44:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8A1CB207DE for ; Fri, 2 Oct 2020 15:44:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Jde5UiiM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388077AbgJBPof (ORCPT ); Fri, 2 Oct 2020 11:44:35 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:46934 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388054AbgJBPoa (ORCPT ); Fri, 2 Oct 2020 11:44:30 -0400 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FXB2Y080576 for ; Fri, 2 Oct 2020 11:44:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=FHprWJ6NbTNrYJGfaHPl3hQhyT1VPi+z1BRjKXkm7CU=; b=Jde5UiiMKnwVtckiKqnNqwTlLfqWnDRe4qOhVZaB+cMkGfxW/+7ghlgCtBLT61CQJ4tv fwH/UcV5VvJ0vHFJh0jmWwAZv177Vudi8CxIpwScE1DgdCknkUuTfWRhMAKWtBLEtObu qteeAML0LR8IAAsdlUjA5XsT+vu/IdOZ4joBY+CG9DQZoC+CxrxyNlkzn63wdLjjJ2os tjE2e4xUUUM/QjsbP4yxDlXDtAV4ibAC15xRZlGr1ss4pVx9J9BtftZXhBgaRpJ1ddPR TCrrY0eM8gi0O3xVOoe8ezD2kmU1iEq8trWIHRLvFxiaKSZMELWiiU/LKHJACAYup8Ur gQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x6m7907b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:29 -0400 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FXFrm080945 for ; Fri, 2 Oct 2020 11:44:29 -0400 Received: from ppma02fra.de.ibm.com (47.49.7a9f.ip4.static.sl-reverse.com [159.122.73.71]) by mx0a-001b2d01.pphosted.com with ESMTP id 33x6m7906p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:29 -0400 Received: from pps.filterd (ppma02fra.de.ibm.com [127.0.0.1]) by ppma02fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092FiQCI027332; Fri, 2 Oct 2020 15:44:26 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma02fra.de.ibm.com with ESMTP id 33sw983ha3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:26 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiO4Z27918668 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:24 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 49A5142042; Fri, 2 Oct 2020 15:44:24 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DE99B42041; Fri, 2 Oct 2020 15:44:23 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:23 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 6/7] lib/alloc.h: remove align_min from struct alloc_ops Date: Fri, 2 Oct 2020 17:44:19 +0200 Message-Id: <20201002154420.292134-7-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 clxscore=1015 adultscore=0 bulkscore=0 spamscore=0 malwarescore=0 priorityscore=1501 lowpriorityscore=0 suspectscore=2 mlxlogscore=949 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020120 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Remove align_min from struct alloc_ops. Signed-off-by: Claudio Imbrenda --- lib/alloc.h | 1 - lib/alloc_page.c | 1 - lib/alloc_phys.c | 9 +++++---- lib/vmalloc.c | 1 - 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/lib/alloc.h b/lib/alloc.h index 9b4b634..db90b01 100644 --- a/lib/alloc.h +++ b/lib/alloc.h @@ -25,7 +25,6 @@ struct alloc_ops { void *(*memalign)(size_t alignment, size_t size); void (*free)(void *ptr); - size_t align_min; }; extern struct alloc_ops *alloc_ops; diff --git a/lib/alloc_page.c b/lib/alloc_page.c index 046082a..3c6c4ee 100644 --- a/lib/alloc_page.c +++ b/lib/alloc_page.c @@ -320,7 +320,6 @@ void *alloc_page() static struct alloc_ops page_alloc_ops = { .memalign = memalign_pages, .free = free_pages, - .align_min = PAGE_SIZE, }; /* diff --git a/lib/alloc_phys.c b/lib/alloc_phys.c index 72e20f7..a4d2bf2 100644 --- a/lib/alloc_phys.c +++ b/lib/alloc_phys.c @@ -29,8 +29,8 @@ static phys_addr_t base, top; static void *early_memalign(size_t alignment, size_t size); static struct alloc_ops early_alloc_ops = { .memalign = early_memalign, - .align_min = DEFAULT_MINIMUM_ALIGNMENT }; +static size_t align_min; struct alloc_ops *alloc_ops = &early_alloc_ops; @@ -39,8 +39,7 @@ void phys_alloc_show(void) int i; spin_lock(&lock); - printf("phys_alloc minimum alignment: %#" PRIx64 "\n", - (u64)early_alloc_ops.align_min); + printf("phys_alloc minimum alignment: %#" PRIx64 "\n", (u64)align_min); for (i = 0; i < nr_regions; ++i) printf("%016" PRIx64 "-%016" PRIx64 " [%s]\n", (u64)regions[i].base, @@ -64,7 +63,7 @@ void phys_alloc_set_minimum_alignment(phys_addr_t align) { assert(align && !(align & (align - 1))); spin_lock(&lock); - early_alloc_ops.align_min = align; + align_min = align; spin_unlock(&lock); } @@ -83,6 +82,8 @@ static phys_addr_t phys_alloc_aligned_safe(phys_addr_t size, top_safe = MIN(top_safe, 1ULL << 32); assert(base < top_safe); + if (align < align_min) + align = align_min; addr = ALIGN(base, align); size += addr - base; diff --git a/lib/vmalloc.c b/lib/vmalloc.c index 986a34c..b28a390 100644 --- a/lib/vmalloc.c +++ b/lib/vmalloc.c @@ -188,7 +188,6 @@ static void vm_free(void *mem) static struct alloc_ops vmalloc_ops = { .memalign = vm_memalign, .free = vm_free, - .align_min = PAGE_SIZE, }; void __attribute__((__weak__)) find_highmem(void) From patchwork Fri Oct 2 15:44:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Claudio Imbrenda X-Patchwork-Id: 11813881 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A35E9112E for ; Fri, 2 Oct 2020 15:44:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54F482074B for ; Fri, 2 Oct 2020 15:44:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="GkC8QdDc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388058AbgJBPoe (ORCPT ); Fri, 2 Oct 2020 11:44:34 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:3370 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2387777AbgJBPoa (ORCPT ); Fri, 2 Oct 2020 11:44:30 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 092FVTw6109057 for ; Fri, 2 Oct 2020 11:44:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=99VxAMhjIlbO8ommuZ8vYILz5NGD6QmMGy+OtreF4NE=; b=GkC8QdDcbpq8D9LSkMiB/EjdmhlWlibe37T0jpmBIYKLaMmr4xhJlBFXWKGZKBLPnV9+ giRgkVb9+vkYdoe1IvFKjVBFm8q0bjmKlcj8Fd1RtqsnZXhEThbzdXUI+MNvGeuDw60z flh5EfA4ei8gQtVh+4YJCDjEwGHIa2M/xhvxrd0vpz2oe138WHjHO/68eFl1s3kdHYk6 cECkazd3Hd0tgOVLsMRgjhrIgA1bOXCQoU2g43kPJDlGSFCb6MWnHNfnTzbn2WfNjzwX FSYZ9r1DVozD5tYQKO6gS1qXW+V0lJrj2ptQdQU/3EZJ5RwBfZ9FPimU/vz0ZPuFibFj JA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 33x6cbshqu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 02 Oct 2020 11:44:29 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 092FiS5a144801 for ; Fri, 2 Oct 2020 11:44:28 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0b-001b2d01.pphosted.com with ESMTP id 33x6cbshq8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 11:44:28 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 092Fh2GG031064; Fri, 2 Oct 2020 15:44:27 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma04ams.nl.ibm.com with ESMTP id 33sw97xqj4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 02 Oct 2020 15:44:26 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 092FiOAX31981926 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 2 Oct 2020 15:44:24 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BD7B742047; Fri, 2 Oct 2020 15:44:24 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5BCF042041; Fri, 2 Oct 2020 15:44:24 +0000 (GMT) Received: from ibm-vm.ibmuc.com (unknown [9.145.14.90]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 2 Oct 2020 15:44:24 +0000 (GMT) From: Claudio Imbrenda To: kvm@vger.kernel.org, pbonzini@redhat.com Cc: frankja@linux.ibm.com, david@redhat.com, thuth@redhat.com, cohuck@redhat.com, lvivier@redhat.com Subject: [kvm-unit-tests PATCH v2 7/7] lib/alloc_page: allow reserving arbitrary memory ranges Date: Fri, 2 Oct 2020 17:44:20 +0200 Message-Id: <20201002154420.292134-8-imbrenda@linux.ibm.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201002154420.292134-1-imbrenda@linux.ibm.com> References: <20201002154420.292134-1-imbrenda@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-10-02_10:2020-10-02,2020-10-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 phishscore=0 spamscore=0 clxscore=1015 priorityscore=1501 lowpriorityscore=0 adultscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0 suspectscore=2 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2010020119 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Two new functions are introduced, that allow specific memory ranges to be reserved and freed. This is useful when a testcase needs memory at very specific addresses, with the guarantee that the page allocator will not touch those pages. Signed-off-by: Claudio Imbrenda --- lib/alloc_page.h | 15 ++++++++++ lib/alloc_page.c | 78 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 88 insertions(+), 5 deletions(-) diff --git a/lib/alloc_page.h b/lib/alloc_page.h index 6c23018..816ff5d 100644 --- a/lib/alloc_page.h +++ b/lib/alloc_page.h @@ -75,4 +75,19 @@ static inline void free_pages_by_order(void *mem, unsigned int order) free_pages(mem); } +/* + * Allocates and reserves the specified memory range if possible. + * Returns NULL in case of failure. + */ +void *alloc_pages_special(uintptr_t addr, size_t npages); + +/* + * Frees a reserved memory range that had been reserved with + * alloc_pages_special. + * The memory range does not need to match a previous allocation + * exactly, it can also be a subset, in which case only the specified + * pages will be freed and unreserved. + */ +void free_pages_special(uintptr_t addr, size_t npages); + #endif diff --git a/lib/alloc_page.c b/lib/alloc_page.c index 3c6c4ee..d9665a4 100644 --- a/lib/alloc_page.c +++ b/lib/alloc_page.c @@ -23,13 +23,14 @@ #define ORDER_MASK 0x3f #define ALLOC_MASK 0x40 +#define SPECIAL_MASK 0x80 struct mem_area { /* Physical frame number of the first usable frame in the area */ uintptr_t base; /* Physical frame number of the first frame outside the area */ uintptr_t top; - /* Combination ALLOC_MASK and order */ + /* Combination of SPECIAL_MASK, ALLOC_MASK, and order */ u8 *page_states; /* One freelist for each possible block size, up to NLISTS */ struct linked_list freelists[NLISTS]; @@ -136,6 +137,16 @@ static void *page_memalign_order(struct mem_area *a, u8 al, u8 sz) return res; } +static struct mem_area *get_area(uintptr_t pfn) +{ + uintptr_t i; + + for (i = 0; i < MAX_AREAS; i++) + if ((areas_mask & BIT(i)) && area_contains(areas + i, pfn)) + return areas + i; + return NULL; +} + /* * Try to merge two blocks into a bigger one. * Returns true in case of a successful merge. @@ -210,10 +221,7 @@ static void _free_pages(void *mem) assert(IS_ALIGNED((uintptr_t)mem, PAGE_SIZE)); /* find which area this pointer belongs to*/ - for (i = 0; !a && (i < MAX_AREAS); i++) { - if ((areas_mask & BIT(i)) && area_contains(areas + i, pfn)) - a = areas + i; - } + a = get_area(pfn); assert_msg(a, "memory does not belong to any area: %p", mem); p = pfn - a->base; @@ -262,6 +270,66 @@ void free_pages(void *mem) spin_unlock(&lock); } +static void *_alloc_page_special(uintptr_t addr) +{ + struct mem_area *a; + uintptr_t mask, i; + + a = get_area(PFN(addr)); + assert(a); + i = PFN(addr) - a->base; + if (a->page_states[i] & (ALLOC_MASK | SPECIAL_MASK)) + return NULL; + while (a->page_states[i]) { + mask = GENMASK_ULL(63, PAGE_SHIFT + a->page_states[i]); + split(a, (void *)(addr & mask)); + } + a->page_states[i] = SPECIAL_MASK; + return (void *)addr; +} + +static void _free_page_special(uintptr_t addr) +{ + struct mem_area *a; + uintptr_t i; + + a = get_area(PFN(addr)); + assert(a); + i = PFN(addr) - a->base; + assert(a->page_states[i] == SPECIAL_MASK); + a->page_states[i] = ALLOC_MASK; + _free_pages((void *)addr); +} + +void *alloc_pages_special(uintptr_t addr, size_t n) +{ + uintptr_t i; + + assert(IS_ALIGNED(addr, PAGE_SIZE)); + spin_lock(&lock); + for (i = 0; i < n; i++) + if (!_alloc_page_special(addr + i * PAGE_SIZE)) + break; + if (i < n) { + for (n = 0 ; n < i; n++) + _free_page_special(addr + n * PAGE_SIZE); + addr = 0; + } + spin_unlock(&lock); + return (void *)addr; +} + +void free_pages_special(uintptr_t addr, size_t n) +{ + uintptr_t i; + + assert(IS_ALIGNED(addr, PAGE_SIZE)); + spin_lock(&lock); + for (i = 0; i < n; i++) + _free_page_special(addr + i * PAGE_SIZE); + spin_unlock(&lock); +} + static void *page_memalign_order_area(unsigned area, u8 ord, u8 al) { void *res = NULL;