From patchwork Tue Feb 18 21:00:59 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nicholas A. Bellinger" X-Patchwork-Id: 3674831 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 01A6E9F35F for ; Tue, 18 Feb 2014 20:58:04 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id BEB7D20163 for ; Tue, 18 Feb 2014 20:58:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 990942010F for ; Tue, 18 Feb 2014 20:58:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752379AbaBRU55 (ORCPT ); Tue, 18 Feb 2014 15:57:57 -0500 Received: from mail.linux-iscsi.org ([67.23.28.174]:50520 "EHLO linux-iscsi.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751675AbaBRU5z (ORCPT ); Tue, 18 Feb 2014 15:57:55 -0500 Received: from [172.18.0.146] (unknown [157.22.22.146]) (using SSLv3 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: nab) by linux-iscsi.org (Postfix) with ESMTPSA id B800422D9DE; Tue, 18 Feb 2014 20:36:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=linux-iscsi.org; s=default.private; t=1392755783; bh=H5XccyENeFfYdo3IEX7xk8LPlZuQcZ4 TlJo3WaKlir0=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To: References:Content-Type:Mime-Version:Content-Transfer-Encoding; b=VCov0iNplAQbwKM3qngX8X18VLuGSlQcvHf+leq72MtrrbvDhlq6yyermuHl1bcPt F3LaD9pFO6duZZhPOiUAuRr9jVx5o7MYOZF03sTCT8hDPEN5yBPgUvisfJwWY4SKEnI u6NSNQgKCWXD8/+gXWsjlXMRPUIG1EXRELNzZzY= Message-ID: <1392757259.28946.46.camel@haakon3.risingtidesystems.com> Subject: Re: MSI interrupt support with vioscsi.c miniport driver From: "Nicholas A. Bellinger" To: Vadim Rozenfeld Cc: Yan Vugenfirer , "Michael S. Tsirkin" , Paolo Bonzini , kvm-devel , qemu-devel , target-devel , dnk@daterainc.com Date: Tue, 18 Feb 2014 13:00:59 -0800 In-Reply-To: <1392059126.17867.12.camel@haakon3.risingtidesystems.com> References: <1391804085.1155.60.camel@haakon3.risingtidesystems.com> <693A9682-53BF-4A61-86AB-1F19F1B91B52@redhat.com> <1391945757.27748.32.camel@w7-rhel> <1392059126.17867.12.camel@haakon3.risingtidesystems.com> X-Mailer: Evolution 3.4.4-1 Mime-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-7.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Mon, 2014-02-10 at 11:05 -0800, Nicholas A. Bellinger wrote: > > > > Hi Yan, > > > > > > > > So recently I've been doing some KVM guest performance comparisons > > > > between the scsi-mq prototype using virtio-scsi + vhost-scsi, and > > > > Windows Server 2012 with vioscsi.sys (virtio-win-0.1-74.iso) + > > > > vhost-scsi using PCIe flash backend devices. > > > > > > > > I've noticed that small block random performance for the MSFT guest is > > > > at around ~80K IOPs with multiple vioscsi LUNs + adapters, which ends up > > > > being well below what the Linux guest with scsi-mq + virtio-scsi is > > > > capable of (~500K). > > > > > > > > After searching through the various vioscsi registry settings, it > > > > appears that MSIEnabled is being explicitly disabled (0x00000000), that > > > > is different from what vioscsi.inx is currently defining: > > > > > > > > [pnpsafe_pci_addreg_msix] > > > > HKR, "Interrupt Management",, 0x00000010 > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 > > > > HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 > > > > > > > > Looking deeper at vioscsi.c code, I've noticed that MSI_SUPPORTED=0 is > > > > explicitly disabled at build time in SOURCES + vioscsi.vcxproj, as well > > > > as VioScsiFindAdapter() code always ends setting msix_enabled = FALSE > > > > here, regardless of MSI_SUPPORTED: > > > > > > > > https://github.com/YanVugenfirer/kvm-guest-drivers-windows/blob/master/vioscsi/vioscsi.c#L340 > > > > > > > > Also looking at virtio_stor.c for the raw block driver, MSI_SUPPORTED=1 > > > > appears to be the default setting for the driver included in the offical > > > > virtio-win iso builds, right..? > > > > > > > > Sooo, I'd like to try enabling MSI_SUPPORTED=1 in a test vioscsi.sys > > > > build of my own, but before going down the WDK development rabbit whole, > > > > I'd like to better understand why you've explicitly disabled this logic > > > > within vioscsi.c code to start..? > > > > > > > > Is there anything that needs to be addressed / carried over from > > > > virtio_stor.c in order to get MSI_SUPPORTED=1 to work with vioscsi.c > > > > miniport code..? > > > > Hi Nicholas, > > > > I was thinking about enabling MSI in RHEL 6.6 (build 74) but for some > > reasons decided to keep it disabled until adding mq support. > > > > > > You definitely should be able to turn on MSI_SUPPORTED, rebuild the > > driver, and switch MSISupported to 1 to make vioscsi driver working in > > MSI mode. > > > > Thanks for the quick response. We'll give MSI_SUPPORTED=1 a shot over > the next days with a test build on Server 2012 / Server 2008 R2 and see > how things go.. > Just a quick update on progress. I've been able to successfully build + load a unsigned vioscsi.sys driver on Server 2012 with WDK 8.0. Running with MSI_SUPPORTED=1 against vhost-scsi results in a significant performance and efficiency gain, on the order of 100K to 225K IOPs for 4K block random I/O workload, depending on read/write mix. Below is a simple patch to enable MSI operation by default. Any chance to apply this separate from future mq efforts..? Thanks, --nab From 89adb6d5800386d44b36737d1587e0ffc09c4902 Mon Sep 17 00:00:00 2001 From: Nicholas Bellinger Date: Fri, 14 Feb 2014 10:26:04 -0800 Subject: [PATCH] vioscsi: Set MSI_SUPPORTED=1 by default Signed-off-by: Nicholas Bellinger --- vioscsi/SOURCES | 2 +- vioscsi/vioscsi.c | 2 -- vioscsi/vioscsi.inx | 2 +- vioscsi/vioscsi.vcxproj | 6 +++--- 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/vioscsi/SOURCES b/vioscsi/SOURCES index f2083de..f631bd2 100644 --- a/vioscsi/SOURCES +++ b/vioscsi/SOURCES @@ -6,7 +6,7 @@ C_DEFINES = -D_MINORVERSION_=$(_BUILD_MINOR_VERSION_) $(C_DEFINES) C_DEFINES = -D_NT_TARGET_MAJ=$(_NT_TARGET_MAJ) $(C_DEFINES) C_DEFINES = -D_NT_TARGET_MIN=$(_RHEL_RELEASE_VERSION_) $(C_DEFINES) -C_DEFINES = -DMSI_SUPPORTED=0 $(C_DEFINES) +C_DEFINES = -DMSI_SUPPORTED=1 $(C_DEFINES) C_DEFINES = -DINDIRECT_SUPPORTED=1 $(C_DEFINES) TARGETLIBS=$(SDK_LIB_PATH)\storport.lib ..\VirtIO\$(O)\virtiolib.lib diff --git a/vioscsi/vioscsi.c b/vioscsi/vioscsi.c index 77c0e46..70b9bb4 100644 --- a/vioscsi/vioscsi.c +++ b/vioscsi/vioscsi.c @@ -337,8 +337,6 @@ ENTER_FN(); adaptExt->queue_depth = pageNum / ConfigInfo->NumberOfPhysicalBreaks - 1; } - adaptExt->msix_enabled = FALSE; - RhelDbgPrint(TRACE_LEVEL_ERROR, ("breaks_number = %x queue_depth = %x\n", ConfigInfo->NumberOfPhysicalBreaks, adaptExt->queue_depth)); diff --git a/vioscsi/vioscsi.inx b/vioscsi/vioscsi.inx index cc94b7c..ec717c6 100644 --- a/vioscsi/vioscsi.inx +++ b/vioscsi/vioscsi.inx @@ -79,7 +79,7 @@ HKR, "Parameters", "BusType", %REG_DWORD%, 0x00000001 [pnpsafe_pci_addreg_msix] HKR, "Interrupt Management",, 0x00000010 HKR, "Interrupt Management\MessageSignaledInterruptProperties",, 0x00000010 -HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 0 +HKR, "Interrupt Management\MessageSignaledInterruptProperties", MSISupported, 0x00010001, 1 HKR, "Interrupt Management\MessageSignaledInterruptProperties", MessageNumberLimit, 0x00010001, 4 HKR, "Interrupt Management\Affinity Policy",, 0x00000010 HKR, "Interrupt Management\Affinity Policy", DevicePolicy, 0x00010001, 5 diff --git a/vioscsi/vioscsi.vcxproj b/vioscsi/vioscsi.vcxproj index 889e513..68d4e85 100644 --- a/vioscsi/vioscsi.vcxproj +++ b/vioscsi/vioscsi.vcxproj @@ -66,7 +66,7 @@ Level3 - _X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions) + _X86_=1;i386=1;STD_CALL;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions) %(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib @@ -95,7 +95,7 @@ Level3 - _WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=0;%(PreprocessorDefinitions) + _WIN64;_AMD64_;AMD64;INDIRECT_SUPPORTED=1;MSI_SUPPORTED=1;%(PreprocessorDefinitions) %(AdditionalDependencies);$(KernelBufferOverflowLib);$(DDK_LIB_PATH)\ntoskrnl.lib;$(DDK_LIB_PATH)\storport.lib;$(DDK_LIB_PATH)\wdm.lib;virtiolib.lib @@ -140,4 +140,4 @@ - \ No newline at end of file +