From patchwork Wed May 4 13:45:16 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 753802 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.3) with ESMTP id p44DjdoL019712 for ; Wed, 4 May 2011 13:45:39 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753882Ab1EDNpe (ORCPT ); Wed, 4 May 2011 09:45:34 -0400 Received: from mail-wy0-f174.google.com ([74.125.82.174]:48851 "EHLO mail-wy0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753860Ab1EDNpc (ORCPT ); Wed, 4 May 2011 09:45:32 -0400 Received: by mail-wy0-f174.google.com with SMTP id 21so843669wya.19 for ; Wed, 04 May 2011 06:45:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:to:cc:subject:date:message-id:x-mailer :in-reply-to:references; bh=y2xclyA+kluucUbGo9HDGpYLrqBkPXajGYPVL4OgawY=; b=a7Vk18hsGYl2x5Smn+DZCM8aLP00MlMZufCT0RzKVMocD7Ue0HwMA9Qk/JOgcxA0Sd AaVtc0fX1JPSYHnuTeomZ73ngntEsp46KQMoaCl4tS/ffZozyqrMLEi0fyzrlPiXITDa vfStNOGrpjOOlBAtuDu8rZxChN9qGPPaqKtp8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references; b=AE0EovR5/XKT1uSjmpOp5yZF5IvpKHH9LaNZrOKRjenzpMU8reGLyy8VYxPID+947Y BHCahZIBGPIoy7dnKFVu4sVg4O9GQdeUZ4AIkSLEmbG8BvGufyzFoCzyUURx5sx4FeDz j6C0CHpTZgTLvhxBPK2jacW8bV4RehEurpCwg= Received: by 10.227.207.196 with SMTP id fz4mr1215159wbb.42.1304516731532; Wed, 04 May 2011 06:45:31 -0700 (PDT) Received: from localhost.localdomain ([77.124.181.142]) by mx.google.com with ESMTPS id x13sm226068wby.42.2011.05.04.06.45.29 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 04 May 2011 06:45:31 -0700 (PDT) From: Sasha Levin To: penberg@kernel.org, kvm@vger.kernel.org Cc: mingo@elte.hu, asias.hejun@gmail.com, gorcunov@gmail.com, prasadjoshi124@gmail.com, Sasha Levin Subject: [PATCH 2/3 V2] kvm tools: Add support for multiple virtio-blk Date: Wed, 4 May 2011 16:45:16 +0300 Message-Id: <1304516717-24512-3-git-send-email-levinsasha928@gmail.com> X-Mailer: git-send-email 1.7.5.rc3 In-Reply-To: <1304516717-24512-1-git-send-email-levinsasha928@gmail.com> References: <1304516717-24512-1-git-send-email-levinsasha928@gmail.com> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Wed, 04 May 2011 13:45:39 +0000 (UTC) Add support for multiple blk_devices by un-globalizing the current blk_device and allow multiple blk_devices. Signed-off-by: Sasha Levin --- tools/kvm/include/kvm/ioport.h | 2 +- tools/kvm/mptable.c | 39 ++++++++- tools/kvm/virtio-blk.c | 193 +++++++++++++++++++++++++--------------- 3 files changed, 162 insertions(+), 72 deletions(-) diff --git a/tools/kvm/include/kvm/ioport.h b/tools/kvm/include/kvm/ioport.h index 6020124..98a880f 100644 --- a/tools/kvm/include/kvm/ioport.h +++ b/tools/kvm/include/kvm/ioport.h @@ -7,7 +7,7 @@ /* some ports we reserve for own use */ #define IOPORT_DBG 0xe0 #define IOPORT_VIRTIO_BLK 0xc200 /* Virtio block device */ -#define IOPORT_VIRTIO_BLK_SIZE 256 +#define IOPORT_VIRTIO_BLK_SIZE 0x200 #define IOPORT_VIRTIO_CONSOLE 0xd200 /* Virtio console device */ #define IOPORT_VIRTIO_CONSOLE_SIZE 256 #define IOPORT_VIRTIO_NET 0xe200 /* Virtio network device */ diff --git a/tools/kvm/mptable.c b/tools/kvm/mptable.c index 5bbe7ea..e777f64 100644 --- a/tools/kvm/mptable.c +++ b/tools/kvm/mptable.c @@ -181,6 +181,7 @@ void mptable_setup(struct kvm *kvm, unsigned int ncpus) last_addr = (void *)&mpc_intsrc[1]; nentries++; + /* Currently we define 4 possible virtio-blk devices */ mpc_intsrc = last_addr; mpc_intsrc->type = MP_INTSRC; mpc_intsrc->irqtype = mp_INT; @@ -188,7 +189,43 @@ void mptable_setup(struct kvm *kvm, unsigned int ncpus) mpc_intsrc->srcbus = pcibusid; mpc_intsrc->srcbusirq = 1; /* virtio block irq pin */ mpc_intsrc->dstapic = ioapicid; - mpc_intsrc->dstirq = 15; /* VIRTIO_BLK_IRQ */ + mpc_intsrc->dstirq = 9; /* VIRTIO_BLK_IRQ */ + + last_addr = (void *)&mpc_intsrc[1]; + nentries++; + + mpc_intsrc = last_addr; + mpc_intsrc->type = MP_INTSRC; + mpc_intsrc->irqtype = mp_INT; + mpc_intsrc->irqflag = MP_IRQDIR_DEFAULT; + mpc_intsrc->srcbus = pcibusid; + mpc_intsrc->srcbusirq = 1; /* virtio block irq pin */ + mpc_intsrc->dstapic = ioapicid; + mpc_intsrc->dstirq = 10; /* VIRTIO_BLK_IRQ */ + + last_addr = (void *)&mpc_intsrc[1]; + nentries++; + + mpc_intsrc = last_addr; + mpc_intsrc->type = MP_INTSRC; + mpc_intsrc->irqtype = mp_INT; + mpc_intsrc->irqflag = MP_IRQDIR_DEFAULT; + mpc_intsrc->srcbus = pcibusid; + mpc_intsrc->srcbusirq = 1; /* virtio block irq pin */ + mpc_intsrc->dstapic = ioapicid; + mpc_intsrc->dstirq = 11; /* VIRTIO_BLK_IRQ */ + + last_addr = (void *)&mpc_intsrc[1]; + nentries++; + + mpc_intsrc = last_addr; + mpc_intsrc->type = MP_INTSRC; + mpc_intsrc->irqtype = mp_INT; + mpc_intsrc->irqflag = MP_IRQDIR_DEFAULT; + mpc_intsrc->srcbus = pcibusid; + mpc_intsrc->srcbusirq = 1; /* virtio block irq pin */ + mpc_intsrc->dstapic = ioapicid; + mpc_intsrc->dstirq = 12; /* VIRTIO_BLK_IRQ */ last_addr = (void *)&mpc_intsrc[1]; nentries++; diff --git a/tools/kvm/virtio-blk.c b/tools/kvm/virtio-blk.c index d7bda5f..cf6416d 100644 --- a/tools/kvm/virtio-blk.c +++ b/tools/kvm/virtio-blk.c @@ -17,51 +17,47 @@ #include #include -#define VIRTIO_BLK_IRQ 15 +#define VIRTIO_BLK_IRQ 9 #define VIRTIO_BLK_PIN 1 - +#define VIRTIO_BLK_MAX_DEV 4 #define NUM_VIRT_QUEUES 1 #define VIRTIO_BLK_QUEUE_SIZE 128 +struct blk_device_job { + struct virt_queue *vq; + struct blk_device *blk_device; + void *job_id; +}; + struct blk_device { pthread_mutex_t mutex; struct virtio_blk_config blk_config; - struct disk_image *disk; + struct disk_image *disk; uint32_t host_features; uint32_t guest_features; uint16_t config_vector; uint8_t status; + u8 idx; /* virtio queue */ uint16_t queue_selector; struct virt_queue vqs[NUM_VIRT_QUEUES]; - - void *jobs[NUM_VIRT_QUEUES]; + struct blk_device_job jobs[NUM_VIRT_QUEUES]; + struct pci_device_header pci_device; }; -#define DISK_SEG_MAX 126 - -static struct blk_device blk_device = { - .mutex = PTHREAD_MUTEX_INITIALIZER, - - .blk_config = (struct virtio_blk_config) { - /* VIRTIO_BLK_F_SEG_MAX */ - .seg_max = DISK_SEG_MAX, - }, - /* - * Note we don't set VIRTIO_BLK_F_GEOMETRY here so the - * node kernel will compute disk geometry by own, the - * same applies to VIRTIO_BLK_F_BLK_SIZE - */ - .host_features = (1UL << VIRTIO_BLK_F_SEG_MAX), -}; +static struct blk_device *blk_devices[VIRTIO_BLK_MAX_DEV]; -static bool virtio_blk_pci_io_device_specific_in(void *data, unsigned long offset, int size, uint32_t count) +static bool virtio_blk_pci_io_device_specific_in(struct blk_device *blk_device, + void *data, + unsigned long offset, + int size, + uint32_t count) { - uint8_t *config_space = (uint8_t *) &blk_device.blk_config; + uint8_t *config_space = (uint8_t *) &blk_device->blk_config; if (size != 1 || count != 1) return false; @@ -71,24 +67,38 @@ static bool virtio_blk_pci_io_device_specific_in(void *data, unsigned long offse return true; } +/* Translate port into device id + offset in that device addr space */ +static void virtio_blk_port2dev(u16 port, + u16 base, + u16 size, + u16 *dev_idx, + u16 *offset) +{ + *dev_idx = (port - base) / size; + *offset = port - (base + *dev_idx * size); +} static bool virtio_blk_pci_io_in(struct kvm *self, uint16_t port, void *data, int size, uint32_t count) { - unsigned long offset; + u16 offset, dev_idx; bool ret = true; + struct blk_device *blk_device; + + virtio_blk_port2dev(port, IOPORT_VIRTIO_BLK, IOPORT_VIRTIO_BLK_SIZE, + &dev_idx, &offset); - mutex_lock(&blk_device.mutex); + blk_device = blk_devices[dev_idx]; - offset = port - IOPORT_VIRTIO_BLK; + mutex_lock(&blk_device->mutex); switch (offset) { case VIRTIO_PCI_HOST_FEATURES: - ioport__write32(data, blk_device.host_features); + ioport__write32(data, blk_device->host_features); break; case VIRTIO_PCI_GUEST_FEATURES: ret = false; break; case VIRTIO_PCI_QUEUE_PFN: - ioport__write32(data, blk_device.vqs[blk_device.queue_selector].pfn); + ioport__write32(data, blk_device->vqs[blk_device->queue_selector].pfn); break; case VIRTIO_PCI_QUEUE_NUM: ioport__write16(data, VIRTIO_BLK_QUEUE_SIZE); @@ -98,25 +108,27 @@ static bool virtio_blk_pci_io_in(struct kvm *self, uint16_t port, void *data, in ret = false; break; case VIRTIO_PCI_STATUS: - ioport__write8(data, blk_device.status); + ioport__write8(data, blk_device->status); break; case VIRTIO_PCI_ISR: ioport__write8(data, 0x1); - kvm__irq_line(self, VIRTIO_BLK_IRQ, 0); + kvm__irq_line(self, VIRTIO_BLK_IRQ + blk_device->idx, 0); break; case VIRTIO_MSI_CONFIG_VECTOR: - ioport__write16(data, blk_device.config_vector); + ioport__write16(data, blk_device->config_vector); break; default: - ret = virtio_blk_pci_io_device_specific_in(data, offset, size, count); + ret = virtio_blk_pci_io_device_specific_in(blk_device, data, offset, size, count); }; - mutex_unlock(&blk_device.mutex); + mutex_unlock(&blk_device->mutex); return ret; } -static bool virtio_blk_do_io_request(struct kvm *self, struct virt_queue *queue) +static bool virtio_blk_do_io_request(struct kvm *self, + struct blk_device *blk_device, + struct virt_queue *queue) { struct iovec iov[VIRTIO_BLK_QUEUE_SIZE]; struct virtio_blk_outhdr *req; @@ -131,11 +143,11 @@ static bool virtio_blk_do_io_request(struct kvm *self, struct virt_queue *queue) switch (req->type) { case VIRTIO_BLK_T_IN: - block_cnt = disk_image__read_sector_iov(blk_device.disk, req->sector, iov + 1, in + out - 2); + block_cnt = disk_image__read_sector_iov(blk_device->disk, req->sector, iov + 1, in + out - 2); break; case VIRTIO_BLK_T_OUT: - block_cnt = disk_image__write_sector_iov(blk_device.disk, req->sector, iov + 1, in + out - 2); + block_cnt = disk_image__write_sector_iov(blk_device->disk, req->sector, iov + 1, in + out - 2); break; @@ -155,58 +167,69 @@ static bool virtio_blk_do_io_request(struct kvm *self, struct virt_queue *queue) static void virtio_blk_do_io(struct kvm *kvm, void *param) { - struct virt_queue *vq = param; + struct blk_device_job *job = param; + struct virt_queue *vq = job->vq; + struct blk_device *blk_device = job->blk_device; while (virt_queue__available(vq)) - virtio_blk_do_io_request(kvm, vq); + virtio_blk_do_io_request(kvm, blk_device, vq); - kvm__irq_line(kvm, VIRTIO_BLK_IRQ, 1); + kvm__irq_line(kvm, VIRTIO_BLK_IRQ + blk_device->idx, 1); } static bool virtio_blk_pci_io_out(struct kvm *self, uint16_t port, void *data, int size, uint32_t count) { - unsigned long offset; + u16 offset, dev_idx; bool ret = true; + struct blk_device *blk_device; - mutex_lock(&blk_device.mutex); + virtio_blk_port2dev(port, IOPORT_VIRTIO_BLK, IOPORT_VIRTIO_BLK_SIZE, + &dev_idx, &offset); - offset = port - IOPORT_VIRTIO_BLK; + blk_device = blk_devices[dev_idx]; + + mutex_lock(&blk_device->mutex); switch (offset) { case VIRTIO_PCI_GUEST_FEATURES: - blk_device.guest_features = ioport__read32(data); + blk_device->guest_features = ioport__read32(data); break; case VIRTIO_PCI_QUEUE_PFN: { struct virt_queue *queue; + struct blk_device_job *job; void *p; - queue = &blk_device.vqs[blk_device.queue_selector]; + job = &blk_device->jobs[blk_device->queue_selector]; + queue = &blk_device->vqs[blk_device->queue_selector]; queue->pfn = ioport__read32(data); - p = guest_flat_to_host(self, queue->pfn << 12); vring_init(&queue->vring, VIRTIO_BLK_QUEUE_SIZE, p, 4096); - blk_device.jobs[blk_device.queue_selector] = - thread_pool__add_job(self, virtio_blk_do_io, queue); + *job = (struct blk_device_job) { + .vq = queue, + .blk_device = blk_device, + }; + + job->job_id = thread_pool__add_job(self, virtio_blk_do_io, job); break; } case VIRTIO_PCI_QUEUE_SEL: - blk_device.queue_selector = ioport__read16(data); + blk_device->queue_selector = ioport__read16(data); break; case VIRTIO_PCI_QUEUE_NOTIFY: { uint16_t queue_index; queue_index = ioport__read16(data); - thread_pool__do_job(blk_device.jobs[queue_index]); + thread_pool__do_job(blk_device->jobs[queue_index].job_id); break; } case VIRTIO_PCI_STATUS: - blk_device.status = ioport__read8(data); + blk_device->status = ioport__read8(data); break; case VIRTIO_MSI_CONFIG_VECTOR: - blk_device.config_vector = VIRTIO_MSI_NO_VECTOR; + blk_device->config_vector = VIRTIO_MSI_NO_VECTOR; break; case VIRTIO_MSI_QUEUE_VECTOR: break; @@ -214,7 +237,7 @@ static bool virtio_blk_pci_io_out(struct kvm *self, uint16_t port, void *data, i ret = false; }; - mutex_unlock(&blk_device.mutex); + mutex_unlock(&blk_device->mutex); return ret; } @@ -228,32 +251,62 @@ static struct ioport_operations virtio_blk_io_ops = { #define PCI_DEVICE_ID_VIRTIO_BLK 0x1001 #define PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET 0x1af4 #define PCI_SUBSYSTEM_ID_VIRTIO_BLK 0x0002 +#define PCI_VIRTIO_BLK_DEVNUM 10 -static struct pci_device_header virtio_blk_pci_device = { - .vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET, - .device_id = PCI_DEVICE_ID_VIRTIO_BLK, - .header_type = PCI_HEADER_TYPE_NORMAL, - .revision_id = 0, - .class = 0x010000, - .subsys_vendor_id = PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET, - .subsys_id = PCI_SUBSYSTEM_ID_VIRTIO_BLK, - .bar[0] = IOPORT_VIRTIO_BLK | PCI_BASE_ADDRESS_SPACE_IO, - .irq_pin = VIRTIO_BLK_PIN, - .irq_line = VIRTIO_BLK_IRQ, -}; +static int virtio_blk_find_empty_dev(void) +{ + int i; -#define PCI_VIRTIO_BLK_DEVNUM 1 + for (i = 0; i < VIRTIO_BLK_MAX_DEV; i++) { + if (blk_devices[i] == NULL) + return i; + } + + return -1; +} void virtio_blk__init(struct kvm *self, struct disk_image *disk) { + int new_dev_idx; + u16 blk_dev_base_addr; + struct blk_device *blk_device; + if (!disk) return; - blk_device.disk = disk; - - blk_device.blk_config.capacity = disk->size / SECTOR_SIZE; + new_dev_idx = virtio_blk_find_empty_dev(); + if (new_dev_idx < 0) + die("Could not find an empty block device slot"); + + blk_devices[new_dev_idx] = calloc(1, sizeof(struct blk_device)); + if (blk_devices[new_dev_idx] == NULL) + die("Failed allocating blk_device"); + + blk_device = blk_devices[new_dev_idx]; + blk_dev_base_addr = IOPORT_VIRTIO_BLK + new_dev_idx * IOPORT_VIRTIO_BLK_SIZE; + + *blk_device = (struct blk_device) { + .mutex = PTHREAD_MUTEX_INITIALIZER, + .disk = disk, + .idx = new_dev_idx, + .blk_config = (struct virtio_blk_config) { + .capacity = disk->size / SECTOR_SIZE, + }, + .pci_device = (struct pci_device_header) { + .vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET, + .device_id = PCI_DEVICE_ID_VIRTIO_BLK, + .header_type = PCI_HEADER_TYPE_NORMAL, + .revision_id = 0, + .class = 0x010000, + .subsys_vendor_id = PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET, + .subsys_id = PCI_SUBSYSTEM_ID_VIRTIO_BLK, + .bar[0] = blk_dev_base_addr | PCI_BASE_ADDRESS_SPACE_IO, + .irq_pin = VIRTIO_BLK_PIN, + .irq_line = VIRTIO_BLK_IRQ + new_dev_idx, + }, + }; - pci__register(&virtio_blk_pci_device, PCI_VIRTIO_BLK_DEVNUM); + pci__register(&blk_device->pci_device, PCI_VIRTIO_BLK_DEVNUM + new_dev_idx); - ioport__register(IOPORT_VIRTIO_BLK, &virtio_blk_io_ops, IOPORT_VIRTIO_BLK_SIZE); + ioport__register(blk_dev_base_addr, &virtio_blk_io_ops, IOPORT_VIRTIO_BLK_SIZE); }