From patchwork Wed May 11 15:17:23 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 776192 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.3) with ESMTP id p4BFlOI7029890 for ; Wed, 11 May 2011 15:47:24 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755039Ab1EKPq2 (ORCPT ); Wed, 11 May 2011 11:46:28 -0400 Received: from mail-ww0-f44.google.com ([74.125.82.44]:56059 "EHLO mail-ww0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755004Ab1EKPqY (ORCPT ); Wed, 11 May 2011 11:46:24 -0400 Received: by wwa36 with SMTP id 36so756256wwa.1 for ; Wed, 11 May 2011 08:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:cc:subject:date:message-id:x-mailer; bh=fXgzVIjR86kPes7llEB8W5/Q/m6yuOCsq/lJaNETmHw=; b=uDfEvREDsodmVuf0SKRfQdUktN6ACwBdq85wfjXUZoy+4WnXStuQoQDMcQTXQGEo8V KRVulSNHAR0qJxELLYAfI5cAuAea4OzygA3Nr8emDEVeMsMXN0xekEW4eMbEX7RLwFGx K7A4TjZmk2o7NCeUSgo3fPhnpmuT86wgMxKWI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:date:message-id:x-mailer; b=pMx4O73XVdtq0Tt7tI7K38JtE9xdImMGWE0fAkyChNzf+6JmnKPPpUkd/BW0gquB87 ZaOb55zUVvf/ot9Xw2rIbDEZvODxOFC12CH8LryjlIcZcBc/kXWFNLuA6A3vXLRHEEMj ueqCDtY5bnWJKAn7folr3uWNtmS1fCNzU0SHU= Received: by 10.227.182.136 with SMTP id cc8mr9797384wbb.77.1305127060046; Wed, 11 May 2011 08:17:40 -0700 (PDT) Received: from localhost.localdomain ([188.120.134.217]) by mx.google.com with ESMTPS id w25sm170262wbd.22.2011.05.11.08.17.37 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 11 May 2011 08:17:39 -0700 (PDT) From: levinsasha928@gmail.com To: penberg@kernel.org Cc: mingo@elte.hu, asias.hejun@gmail.com, prasadjoshi124@gmail.com, avi@redhat.com, gorcunov@gmail.com, kvm@vger.kernel.org, Sasha Levin Subject: [PATCH 1/3 V4] kvm tools: Add memory gap for larger RAM sizes Date: Wed, 11 May 2011 18:17:23 +0300 Message-Id: <4dcaa893.5925e30a.0f9e.125c@mx.google.com> X-Mailer: git-send-email 1.7.5.rc3 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Wed, 11 May 2011 15:47:25 +0000 (UTC) From: Sasha Levin e820 is expected to leave a memory gap within the low 32 bits of RAM space. From the documentation of e820_setup_gap(): /* * Search for the biggest gap in the low 32 bits of the e820 * memory space. We pass this space to PCI to assign MMIO resources * for hotplug or unconfigured devices in. * Hopefully the BIOS let enough space left. */ Not leaving such gap causes errors and hangs during the boot process. This patch adds a memory gap between 0xe0000000 and 0x100000000 when using more than 0xe0000000 bytes for guest RAM. This patch updates the e820 table, slot allocations used for KVM_SET_USER_MEMORY_REGION. Changes in V2: - Allocate RAM with the gap to avoid altering the translation code. - New patch description. Changes in V3: - Remove unnecessary casts. Changes in V4: - Rewrite kvm__init_ram(). - Document the 64bit gap within the code. Signed-off-by: Sasha Levin --- tools/kvm/bios.c | 27 ++++++++++++---- tools/kvm/include/kvm/e820.h | 2 +- tools/kvm/include/kvm/kvm.h | 2 + tools/kvm/kvm.c | 66 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 82 insertions(+), 15 deletions(-) diff --git a/tools/kvm/bios.c b/tools/kvm/bios.c index 2199c0c..3cd9b24 100644 --- a/tools/kvm/bios.c +++ b/tools/kvm/bios.c @@ -61,8 +61,6 @@ static void e820_setup(struct kvm *kvm) size = guest_flat_to_host(kvm, E820_MAP_SIZE); mem_map = guest_flat_to_host(kvm, E820_MAP_START); - *size = E820_MEM_AREAS; - mem_map[i++] = (struct e820_entry) { .addr = REAL_MODE_IVT_BEGIN, .size = EBDA_START - REAL_MODE_IVT_BEGIN, @@ -78,13 +76,28 @@ static void e820_setup(struct kvm *kvm) .size = MB_BIOS_END - MB_BIOS_BEGIN, .type = E820_MEM_RESERVED, }; - mem_map[i++] = (struct e820_entry) { - .addr = BZ_KERNEL_START, - .size = kvm->ram_size - BZ_KERNEL_START, - .type = E820_MEM_USABLE, - }; + if (kvm->ram_size < KVM_32BIT_GAP_START) { + mem_map[i++] = (struct e820_entry) { + .addr = BZ_KERNEL_START, + .size = kvm->ram_size - BZ_KERNEL_START, + .type = E820_MEM_USABLE, + }; + } else { + mem_map[i++] = (struct e820_entry) { + .addr = BZ_KERNEL_START, + .size = KVM_32BIT_GAP_START - BZ_KERNEL_START, + .type = E820_MEM_USABLE, + }; + mem_map[i++] = (struct e820_entry) { + .addr = 0x100000000ULL, + .size = kvm->ram_size - KVM_32BIT_GAP_START, + .type = E820_MEM_USABLE, + }; + } BUILD_BUG_ON(i > E820_MEM_AREAS); + + *size = i; } /** diff --git a/tools/kvm/include/kvm/e820.h b/tools/kvm/include/kvm/e820.h index 252ae1f..e0f5f2a 100644 --- a/tools/kvm/include/kvm/e820.h +++ b/tools/kvm/include/kvm/e820.h @@ -8,7 +8,7 @@ #define E820_MEM_USABLE 1 #define E820_MEM_RESERVED 2 -#define E820_MEM_AREAS 4 +#define E820_MEM_AREAS 5 struct e820_entry { u64 addr; /* start of memory segment */ diff --git a/tools/kvm/include/kvm/kvm.h b/tools/kvm/include/kvm/kvm.h index 3dab78d..5e2e64c 100644 --- a/tools/kvm/include/kvm/kvm.h +++ b/tools/kvm/include/kvm/kvm.h @@ -8,6 +8,8 @@ #include #define KVM_NR_CPUS (255) +#define KVM_32BIT_GAP_SIZE (512 << 20) +#define KVM_32BIT_GAP_START ((1ULL << 32) - KVM_32BIT_GAP_SIZE) struct kvm { int sys_fd; /* For system ioctls(), i.e. /dev/kvm */ diff --git a/tools/kvm/kvm.c b/tools/kvm/kvm.c index 65793f2..a3d3dd8 100644 --- a/tools/kvm/kvm.c +++ b/tools/kvm/kvm.c @@ -153,23 +153,64 @@ static bool kvm__cpu_supports_vm(void) return regs.ecx & (1 << feature); } -void kvm__init_ram(struct kvm *self) +static void kvm_register_mem_slot(struct kvm *kvm, u32 slot, u64 guest_phys, u64 size, void *userspace_addr) { struct kvm_userspace_memory_region mem; int ret; mem = (struct kvm_userspace_memory_region) { - .slot = 0, - .guest_phys_addr = 0x0UL, - .memory_size = self->ram_size, - .userspace_addr = (unsigned long) self->ram_start, + .slot = slot, + .guest_phys_addr = guest_phys, + .memory_size = size, + .userspace_addr = (u64)userspace_addr, }; - ret = ioctl(self->vm_fd, KVM_SET_USER_MEMORY_REGION, &mem); + ret = ioctl(kvm->vm_fd, KVM_SET_USER_MEMORY_REGION, &mem); if (ret < 0) die_perror("KVM_SET_USER_MEMORY_REGION ioctl"); } +/* + * Allocating RAM size bigger than 4GB requires us to leave a gap + * in the RAM which is used for PCI MMIO, hotplug, and unconfigured + * devices (see documentation of e820_setup_gap() for details). + * + * If we're required to initialize RAM bigger than 4GB, we will create + * a gap between 0xe0000000 and 0x100000000 in the guest virtual mem space. + */ + +void kvm__init_ram(struct kvm *self) +{ + u64 phys_start, phys_size; + void *host_mem; + + if (self->ram_size < KVM_32BIT_GAP_START) { + /* Use a single block of RAM for 32bit RAM */ + + phys_start = 0; + phys_size = self->ram_size; + host_mem = self->ram_start; + + kvm_register_mem_slot(self, 0, 0, self->ram_size, self->ram_start); + } else { + /* First RAM range from zero to the PCI gap: */ + + phys_start = 0; + phys_size = KVM_32BIT_GAP_START; + host_mem = self->ram_start; + + kvm_register_mem_slot(self, 0, phys_start, phys_size, host_mem); + + /* Second RAM range from 4GB to the end of RAM: */ + + phys_start = 0x100000000ULL; + phys_size = self->ram_size - phys_size; + host_mem = self->ram_start + phys_start; + + kvm_register_mem_slot(self, 1, phys_start, phys_size, host_mem); + } +} + int kvm__max_cpus(struct kvm *self) { int ret; @@ -225,7 +266,18 @@ struct kvm *kvm__init(const char *kvm_dev, unsigned long ram_size) self->ram_size = ram_size; - self->ram_start = mmap(NULL, ram_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (self->ram_size < KVM_32BIT_GAP_START) { + self->ram_start = mmap(NULL, ram_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + } else { + self->ram_start = mmap(NULL, ram_size + KVM_32BIT_GAP_SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0); + if (self->ram_start != MAP_FAILED) { + /* + * We mprotect the gap (see kvm__init_ram() for details) PROT_NONE so that + * if we accidently write to it, we will know. + */ + mprotect(self->ram_start + KVM_32BIT_GAP_START, KVM_32BIT_GAP_SIZE, PROT_NONE); + } + } if (self->ram_start == MAP_FAILED) die("out of memory");