From patchwork Tue Jul 24 07:42:04 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nicholas A. Bellinger" X-Patchwork-Id: 1230441 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 7A2D940071 for ; Tue, 24 Jul 2012 07:42:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755523Ab2GXHmg (ORCPT ); Tue, 24 Jul 2012 03:42:36 -0400 Received: from mail.linux-iscsi.org ([67.23.28.174]:40842 "EHLO linux-iscsi.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755434Ab2GXHm1 (ORCPT ); Tue, 24 Jul 2012 03:42:27 -0400 Received: from [192.168.0.113] (c-24-130-178-18.hsd1.ca.comcast.net [24.130.178.18]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: nab) by linux-iscsi.org (Postfix) with ESMTPSA id 8402822D9D7; Tue, 24 Jul 2012 07:39:24 +0000 (UTC) Subject: vhost-scsi port to v1.1.0 + MSI-X performance regression From: "Nicholas A. Bellinger" To: Anthony Liguori Cc: qemu-devel , kvm-devel , target-devel , Stefan Hajnoczi , Paolo Bonzini , Jan Kiszka , Hannes Reinecke , "Michael S. Tsirkin" , Christoph Hellwig , Avi Kivity Date: Tue, 24 Jul 2012 00:42:04 -0700 Message-ID: <1343115724.1813.69.camel@haakon2.linux-iscsi.org> Mime-Version: 1.0 X-Mailer: Evolution 2.30.3 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hi Anthony, Stefan & QEMU folks, So during the process of separating out the patches from Zhi's vhost-scsi tree this evening, I managed to squash everything down to nine nicely reviewable patches that apply against the current qemu.git/master: Nicholas Bellinger (1): virtio-scsi: Set max_target=0 during vhost-scsi operation Stefan Hajnoczi (8): notifier: add validity check and notify function virtio-pci: support host notifiers in TCG mode virtio-pci: check that event notification worked vhost: Pass device path to vhost_dev_init() virtio-scsi: Add wwpn and tgpt properties virtio-scsi: Open and initialize /dev/vhost-scsi virtio-scsi: Start/stop vhost vhost-scsi: add -vhost-scsi host device However, thus far I've not been able to get virtio-scsi <-> tcm_vhost I/O to actually work against the latest qemu.git/master.. So while doing a (manual) bisection w/ this series to track down the issue with qemu/master, I managed to run across something else.. With the vhost-scsi series applied, everything is working as expected up until the following commit: commit 1523ed9e1d46b0b54540049d491475ccac7e6421 Author: Jan Kiszka Date: Thu May 17 10:32:39 2012 -0300 virtio/vhost: Add support for KVM in-kernel MSI injection This commit ends up triggering the following assert immediately after starting qemu with virtio-scsi <-> tcm_vhost: qemu-system-x86_64: /usr/src/qemu.git/hw/msix.c:515: msix_unset_vector_notifiers: Assertion `dev->msix_vector_use_notifier && dev->msix_vector_release_notifier' failed. OK, so adding the following hack allows me to boot: --- and virtio-scsi is then able to load + detect tcm_vhost LUNs as expected. However the random I/O performance with commit 1523ed9e1d46b is off by a couple of orders of magnitude, ~6K IOPs compared to ~60K IOPs on raw block flash using just the previous commit bdd00bdc64ba in Jan's series. So AFAICT there appears to be a serious performance regression that is easily reproducible with that patch, which is about as far along as I've been able to diagnose yet. Interestingly enough, virtio-scsi-raw performance does not seem to be effected AFAICT by this regression, and is still able to go ~20K IOPs with the same workload using commit 1523ed9e1d46b. (Roughly the same as before) Does anyone have any idea why commit 1523ed9e1d46b would be killing vhost / tcm_vhost performance so terribly, or is there something else that vhost / vhost-scsi should be doing with new code..? Thanks! --nab -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/hw/msix.c b/hw/msix.c index 59c7a83..6036909 100644 --- a/hw/msix.c +++ b/hw/msix.c @@ -511,6 +511,11 @@ void msix_unset_vector_notifiers(PCIDevice *dev) { int vector; + if (!dev->msix_vector_use_notifier && !dev->msix_vector_release_notifier) { + printf("Hit NULL msix_unset_vector_notifiers for: %s\n", dev->name); + return; + } + assert(dev->msix_vector_use_notifier && dev->msix_vector_release_notifier);