From patchwork Thu May 5 10:24:30 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 756422 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter1.kernel.org (8.14.4/8.14.3) with ESMTP id p45AP62X010782 for ; Thu, 5 May 2011 10:25:06 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753792Ab1EEKZB (ORCPT ); Thu, 5 May 2011 06:25:01 -0400 Received: from mail-wy0-f174.google.com ([74.125.82.174]:40073 "EHLO mail-wy0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753715Ab1EEKZA (ORCPT ); Thu, 5 May 2011 06:25:00 -0400 Received: by mail-wy0-f174.google.com with SMTP id 21so1485243wya.19 for ; Thu, 05 May 2011 03:24:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references; bh=JdJjP9BmZG67qn2IZfbNlNkh+TRoGtIH8skfidRHlZo=; b=HQa0Q3LNudSwlcMvYeTYl5mtCYtzeA2uHRWf4qEGs0kugrQlIyN+YpUW1PzIWYvsYy 3FSRkpq8g1eTd+92hUJg3sQV5sCRY6QDTqi3bGrObNm/GVjDqHzFaIx/BD+Z4LJsXIX1 aE1RysC3zeNYgUy6mRxcy4vSZpoKZKZ4Nly2o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references; b=eciz/G7Be2qlBvSsfQIgPAsmuxebHpibfWS6gy+LFrWPuAVHFf0IXgOLK/kvYi785O o8TMwio9Gv5nWBoI/rdg/IpKa83uKGZphSbfkF/pEWxcnZ829y09C2bSHgiqGWv6VPIj fAUCRXDcMhacGfhdI/oeZErqZDNXK9dcIGEEY= Received: by 10.216.143.88 with SMTP id k66mr6769487wej.15.1304591098526; Thu, 05 May 2011 03:24:58 -0700 (PDT) Received: from localhost.localdomain (bzq-79-183-199-149.red.bezeqint.net [79.183.199.149]) by mx.google.com with ESMTPS id n2sm1009192wej.22.2011.05.05.03.24.56 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 05 May 2011 03:24:58 -0700 (PDT) From: Sasha Levin To: penberg@kernel.org Cc: mingo@elte.hu, asias.hejun@gmail.com, gorcunov@gmail.com, prasadjoshi124@gmail.com, kvm@vger.kernel.org, Sasha Levin Subject: [PATCH 2/3 V3] kvm tools: Add support for multiple virtio-blk Date: Thu, 5 May 2011 13:24:30 +0300 Message-Id: <1304591071-27074-2-git-send-email-levinsasha928@gmail.com> X-Mailer: git-send-email 1.7.5.rc3 In-Reply-To: <1304591071-27074-1-git-send-email-levinsasha928@gmail.com> References: <1304591071-27074-1-git-send-email-levinsasha928@gmail.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter1.kernel.org [140.211.167.41]); Thu, 05 May 2011 10:25:06 +0000 (UTC) Add support for multiple blk_devices by un-globalizing the current blk_device and allow multiple blk_devices. Signed-off-by: Sasha Levin --- tools/kvm/include/kvm/ioport.h | 2 +- tools/kvm/mptable.c | 56 ++++++------ tools/kvm/virtio-blk.c | 193 +++++++++++++++++++++++++--------------- 3 files changed, 154 insertions(+), 97 deletions(-) diff --git a/tools/kvm/include/kvm/ioport.h b/tools/kvm/include/kvm/ioport.h index 6020124..98a880f 100644 --- a/tools/kvm/include/kvm/ioport.h +++ b/tools/kvm/include/kvm/ioport.h @@ -7,7 +7,7 @@ /* some ports we reserve for own use */ #define IOPORT_DBG 0xe0 #define IOPORT_VIRTIO_BLK 0xc200 /* Virtio block device */ -#define IOPORT_VIRTIO_BLK_SIZE 256 +#define IOPORT_VIRTIO_BLK_SIZE 0x200 #define IOPORT_VIRTIO_CONSOLE 0xd200 /* Virtio console device */ #define IOPORT_VIRTIO_CONSOLE_SIZE 256 #define IOPORT_VIRTIO_NET 0xe200 /* Virtio network device */ diff --git a/tools/kvm/mptable.c b/tools/kvm/mptable.c index b74c491..4fc378e 100644 --- a/tools/kvm/mptable.c +++ b/tools/kvm/mptable.c @@ -16,6 +16,7 @@ #endif #include +#include /* * FIXME: please make sure the addresses borrowed @@ -59,6 +60,21 @@ static unsigned int gen_cpu_flag(unsigned int cpu, unsigned int ncpu) */ #define MPTABLE_MAX_CPUS 255 +static void mptable_add_irq_src(struct mpc_intsrc *mpc_intsrc, + u16 srcbusid, u16 srcbusirq, + u16 dstapic, u16 dstirq) +{ + *mpc_intsrc = (struct mpc_intsrc) { + .type = MP_INTSRC, + .irqtype = mp_INT, + .irqflag = MP_IRQDIR_DEFAULT, + .srcbus = srcbusid, + .srcbusirq = srcbusirq, + .dstapic = dstapic, + .dstirq = dstirq + }; +} + /** * mptable_setup - create mptable and fill guest memory with it */ @@ -171,38 +187,26 @@ void mptable_setup(struct kvm *kvm, unsigned int ncpus) * Also note we use PCI irqs here, no for ISA bus yet. */ mpc_intsrc = last_addr; - mpc_intsrc->type = MP_INTSRC; - mpc_intsrc->irqtype = mp_INT; - mpc_intsrc->irqflag = MP_IRQDIR_DEFAULT; - mpc_intsrc->srcbus = pcibusid; - mpc_intsrc->srcbusirq = 2; /* virtio console irq pin */ - mpc_intsrc->dstapic = ioapicid; - mpc_intsrc->dstirq = 13; /* VIRTIO_CONSOLE_IRQ */ - - last_addr = (void *)&mpc_intsrc[1]; - nentries++; - - mpc_intsrc = last_addr; - mpc_intsrc->type = MP_INTSRC; - mpc_intsrc->irqtype = mp_INT; - mpc_intsrc->irqflag = MP_IRQDIR_DEFAULT; - mpc_intsrc->srcbus = pcibusid; - mpc_intsrc->srcbusirq = 1; /* virtio block irq pin */ - mpc_intsrc->dstapic = ioapicid; - mpc_intsrc->dstirq = 15; /* VIRTIO_BLK_IRQ */ + /* src irq = virtio console irq pin, dst irq = virtio console irq */ + mptable_add_irq_src(mpc_intsrc, pcibusid, 2, ioapicid, 13); last_addr = (void *)&mpc_intsrc[1]; nentries++; + /* Currently we define 4 possible virtio-blk devices */ + for (i = 0; i < 4; i++) { + mpc_intsrc = last_addr; + + /* src irq = virtio blk irq pin, dst irq = virtio blk irq */ + mptable_add_irq_src(mpc_intsrc, pcibusid, 1, ioapicid, 9 + i); + last_addr = (void *)&mpc_intsrc[1]; + nentries++; + } + mpc_intsrc = last_addr; - mpc_intsrc->type = MP_INTSRC; - mpc_intsrc->irqtype = mp_INT; - mpc_intsrc->irqflag = MP_IRQDIR_DEFAULT; - mpc_intsrc->srcbus = pcibusid; - mpc_intsrc->srcbusirq = 3; /* virtio net irq pin */ - mpc_intsrc->dstapic = ioapicid; - mpc_intsrc->dstirq = 14; /* VIRTIO_NET_IRQ */ + /* src irq = virtio net irq pin, dst irq = virtio net irq */ + mptable_add_irq_src(mpc_intsrc, pcibusid, 3, ioapicid, 14); last_addr = (void *)&mpc_intsrc[1]; nentries++; diff --git a/tools/kvm/virtio-blk.c b/tools/kvm/virtio-blk.c index d7bda5f..16c9658 100644 --- a/tools/kvm/virtio-blk.c +++ b/tools/kvm/virtio-blk.c @@ -17,51 +17,47 @@ #include #include -#define VIRTIO_BLK_IRQ 15 +#define VIRTIO_BLK_IRQ 9 #define VIRTIO_BLK_PIN 1 - +#define VIRTIO_BLK_MAX_DEV 4 #define NUM_VIRT_QUEUES 1 #define VIRTIO_BLK_QUEUE_SIZE 128 +struct blk_device_job { + struct virt_queue *vq; + struct blk_device *blk_device; + void *job_id; +}; + struct blk_device { pthread_mutex_t mutex; struct virtio_blk_config blk_config; - struct disk_image *disk; + struct disk_image *disk; uint32_t host_features; uint32_t guest_features; uint16_t config_vector; uint8_t status; + u8 idx; /* virtio queue */ uint16_t queue_selector; struct virt_queue vqs[NUM_VIRT_QUEUES]; - - void *jobs[NUM_VIRT_QUEUES]; + struct blk_device_job jobs[NUM_VIRT_QUEUES]; + struct pci_device_header pci_device; }; -#define DISK_SEG_MAX 126 - -static struct blk_device blk_device = { - .mutex = PTHREAD_MUTEX_INITIALIZER, - - .blk_config = (struct virtio_blk_config) { - /* VIRTIO_BLK_F_SEG_MAX */ - .seg_max = DISK_SEG_MAX, - }, - /* - * Note we don't set VIRTIO_BLK_F_GEOMETRY here so the - * node kernel will compute disk geometry by own, the - * same applies to VIRTIO_BLK_F_BLK_SIZE - */ - .host_features = (1UL << VIRTIO_BLK_F_SEG_MAX), -}; +static struct blk_device *blk_devices[VIRTIO_BLK_MAX_DEV]; -static bool virtio_blk_pci_io_device_specific_in(void *data, unsigned long offset, int size, uint32_t count) +static bool virtio_blk_pci_io_device_specific_in(struct blk_device *blk_device, + void *data, + unsigned long offset, + int size, + uint32_t count) { - uint8_t *config_space = (uint8_t *) &blk_device.blk_config; + uint8_t *config_space = (uint8_t *) &blk_device->blk_config; if (size != 1 || count != 1) return false; @@ -71,24 +67,38 @@ static bool virtio_blk_pci_io_device_specific_in(void *data, unsigned long offse return true; } +/* Translate port into device id + offset in that device addr space */ +static void virtio_blk_port2dev(u16 port, + u16 base, + u16 size, + u16 *dev_idx, + u16 *offset) +{ + *dev_idx = (port - base) / size; + *offset = port - (base + *dev_idx * size); +} static bool virtio_blk_pci_io_in(struct kvm *self, uint16_t port, void *data, int size, uint32_t count) { - unsigned long offset; + u16 offset, dev_idx; bool ret = true; + struct blk_device *blk_device; + + virtio_blk_port2dev(port, IOPORT_VIRTIO_BLK, IOPORT_VIRTIO_BLK_SIZE, + &dev_idx, &offset); - mutex_lock(&blk_device.mutex); + blk_device = blk_devices[dev_idx]; - offset = port - IOPORT_VIRTIO_BLK; + mutex_lock(&blk_device->mutex); switch (offset) { case VIRTIO_PCI_HOST_FEATURES: - ioport__write32(data, blk_device.host_features); + ioport__write32(data, blk_device->host_features); break; case VIRTIO_PCI_GUEST_FEATURES: ret = false; break; case VIRTIO_PCI_QUEUE_PFN: - ioport__write32(data, blk_device.vqs[blk_device.queue_selector].pfn); + ioport__write32(data, blk_device->vqs[blk_device->queue_selector].pfn); break; case VIRTIO_PCI_QUEUE_NUM: ioport__write16(data, VIRTIO_BLK_QUEUE_SIZE); @@ -98,25 +108,27 @@ static bool virtio_blk_pci_io_in(struct kvm *self, uint16_t port, void *data, in ret = false; break; case VIRTIO_PCI_STATUS: - ioport__write8(data, blk_device.status); + ioport__write8(data, blk_device->status); break; case VIRTIO_PCI_ISR: ioport__write8(data, 0x1); - kvm__irq_line(self, VIRTIO_BLK_IRQ, 0); + kvm__irq_line(self, VIRTIO_BLK_IRQ + blk_device->idx, 0); break; case VIRTIO_MSI_CONFIG_VECTOR: - ioport__write16(data, blk_device.config_vector); + ioport__write16(data, blk_device->config_vector); break; default: - ret = virtio_blk_pci_io_device_specific_in(data, offset, size, count); + ret = virtio_blk_pci_io_device_specific_in(blk_device, data, offset, size, count); }; - mutex_unlock(&blk_device.mutex); + mutex_unlock(&blk_device->mutex); return ret; } -static bool virtio_blk_do_io_request(struct kvm *self, struct virt_queue *queue) +static bool virtio_blk_do_io_request(struct kvm *self, + struct blk_device *blk_device, + struct virt_queue *queue) { struct iovec iov[VIRTIO_BLK_QUEUE_SIZE]; struct virtio_blk_outhdr *req; @@ -131,11 +143,11 @@ static bool virtio_blk_do_io_request(struct kvm *self, struct virt_queue *queue) switch (req->type) { case VIRTIO_BLK_T_IN: - block_cnt = disk_image__read_sector_iov(blk_device.disk, req->sector, iov + 1, in + out - 2); + block_cnt = disk_image__read_sector_iov(blk_device->disk, req->sector, iov + 1, in + out - 2); break; case VIRTIO_BLK_T_OUT: - block_cnt = disk_image__write_sector_iov(blk_device.disk, req->sector, iov + 1, in + out - 2); + block_cnt = disk_image__write_sector_iov(blk_device->disk, req->sector, iov + 1, in + out - 2); break; @@ -155,58 +167,69 @@ static bool virtio_blk_do_io_request(struct kvm *self, struct virt_queue *queue) static void virtio_blk_do_io(struct kvm *kvm, void *param) { - struct virt_queue *vq = param; + struct blk_device_job *job = param; + struct virt_queue *vq = job->vq; + struct blk_device *blk_device = job->blk_device; while (virt_queue__available(vq)) - virtio_blk_do_io_request(kvm, vq); + virtio_blk_do_io_request(kvm, blk_device, vq); - kvm__irq_line(kvm, VIRTIO_BLK_IRQ, 1); + kvm__irq_line(kvm, VIRTIO_BLK_IRQ + blk_device->idx, 1); } static bool virtio_blk_pci_io_out(struct kvm *self, uint16_t port, void *data, int size, uint32_t count) { - unsigned long offset; + u16 offset, dev_idx; bool ret = true; + struct blk_device *blk_device; - mutex_lock(&blk_device.mutex); + virtio_blk_port2dev(port, IOPORT_VIRTIO_BLK, IOPORT_VIRTIO_BLK_SIZE, + &dev_idx, &offset); - offset = port - IOPORT_VIRTIO_BLK; + blk_device = blk_devices[dev_idx]; + + mutex_lock(&blk_device->mutex); switch (offset) { case VIRTIO_PCI_GUEST_FEATURES: - blk_device.guest_features = ioport__read32(data); + blk_device->guest_features = ioport__read32(data); break; case VIRTIO_PCI_QUEUE_PFN: { struct virt_queue *queue; + struct blk_device_job *job; void *p; - queue = &blk_device.vqs[blk_device.queue_selector]; + job = &blk_device->jobs[blk_device->queue_selector]; + queue = &blk_device->vqs[blk_device->queue_selector]; queue->pfn = ioport__read32(data); - p = guest_flat_to_host(self, queue->pfn << 12); vring_init(&queue->vring, VIRTIO_BLK_QUEUE_SIZE, p, 4096); - blk_device.jobs[blk_device.queue_selector] = - thread_pool__add_job(self, virtio_blk_do_io, queue); + *job = (struct blk_device_job) { + .vq = queue, + .blk_device = blk_device, + }; + + job->job_id = thread_pool__add_job(self, virtio_blk_do_io, job); break; } case VIRTIO_PCI_QUEUE_SEL: - blk_device.queue_selector = ioport__read16(data); + blk_device->queue_selector = ioport__read16(data); break; case VIRTIO_PCI_QUEUE_NOTIFY: { uint16_t queue_index; queue_index = ioport__read16(data); - thread_pool__do_job(blk_device.jobs[queue_index]); + thread_pool__do_job(blk_device->jobs[queue_index].job_id); break; } case VIRTIO_PCI_STATUS: - blk_device.status = ioport__read8(data); + blk_device->status = ioport__read8(data); break; case VIRTIO_MSI_CONFIG_VECTOR: - blk_device.config_vector = VIRTIO_MSI_NO_VECTOR; + blk_device->config_vector = VIRTIO_MSI_NO_VECTOR; break; case VIRTIO_MSI_QUEUE_VECTOR: break; @@ -214,7 +237,7 @@ static bool virtio_blk_pci_io_out(struct kvm *self, uint16_t port, void *data, i ret = false; }; - mutex_unlock(&blk_device.mutex); + mutex_unlock(&blk_device->mutex); return ret; } @@ -228,32 +251,62 @@ static struct ioport_operations virtio_blk_io_ops = { #define PCI_DEVICE_ID_VIRTIO_BLK 0x1001 #define PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET 0x1af4 #define PCI_SUBSYSTEM_ID_VIRTIO_BLK 0x0002 +#define PCI_VIRTIO_BLK_DEVNUM 10 -static struct pci_device_header virtio_blk_pci_device = { - .vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET, - .device_id = PCI_DEVICE_ID_VIRTIO_BLK, - .header_type = PCI_HEADER_TYPE_NORMAL, - .revision_id = 0, - .class = 0x010000, - .subsys_vendor_id = PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET, - .subsys_id = PCI_SUBSYSTEM_ID_VIRTIO_BLK, - .bar[0] = IOPORT_VIRTIO_BLK | PCI_BASE_ADDRESS_SPACE_IO, - .irq_pin = VIRTIO_BLK_PIN, - .irq_line = VIRTIO_BLK_IRQ, -}; +static int virtio_blk_find_empty_dev(void) +{ + int i; -#define PCI_VIRTIO_BLK_DEVNUM 1 + for (i = 0; i < VIRTIO_BLK_MAX_DEV; i++) { + if (blk_devices[i] == NULL) + return i; + } + + return -1; +} void virtio_blk__init(struct kvm *self, struct disk_image *disk) { + int new_dev_idx; + u16 blk_dev_base_addr; + struct blk_device *blk_device; + if (!disk) return; - blk_device.disk = disk; - - blk_device.blk_config.capacity = disk->size / SECTOR_SIZE; + new_dev_idx = virtio_blk_find_empty_dev(); + if (new_dev_idx < 0) + die("Could not find an empty block device slot"); + + blk_devices[new_dev_idx] = calloc(1, sizeof(struct blk_device)); + if (blk_devices[new_dev_idx] == NULL) + die("Failed allocating blk_device"); + + blk_device = blk_devices[new_dev_idx]; + blk_dev_base_addr = IOPORT_VIRTIO_BLK + new_dev_idx * IOPORT_VIRTIO_BLK_SIZE; + + *blk_device = (struct blk_device) { + .mutex = PTHREAD_MUTEX_INITIALIZER, + .disk = disk, + .idx = new_dev_idx, + .blk_config = (struct virtio_blk_config) { + .capacity = disk->size / SECTOR_SIZE, + }, + .pci_device = (struct pci_device_header) { + .vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET, + .device_id = PCI_DEVICE_ID_VIRTIO_BLK, + .header_type = PCI_HEADER_TYPE_NORMAL, + .revision_id = 0, + .class = 0x010000, + .subsys_vendor_id = PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET, + .subsys_id = PCI_SUBSYSTEM_ID_VIRTIO_BLK, + .bar[0] = blk_dev_base_addr | PCI_BASE_ADDRESS_SPACE_IO, + .irq_pin = VIRTIO_BLK_PIN, + .irq_line = VIRTIO_BLK_IRQ + new_dev_idx, + }, + }; - pci__register(&virtio_blk_pci_device, PCI_VIRTIO_BLK_DEVNUM); + pci__register(&blk_device->pci_device, PCI_VIRTIO_BLK_DEVNUM + new_dev_idx); - ioport__register(IOPORT_VIRTIO_BLK, &virtio_blk_io_ops, IOPORT_VIRTIO_BLK_SIZE); + ioport__register(blk_dev_base_addr, &virtio_blk_io_ops, IOPORT_VIRTIO_BLK_SIZE); }