From patchwork Sat Dec 1 18:52:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dongli Zhang X-Patchwork-Id: 10707761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 468CA13AF for ; Sat, 1 Dec 2018 18:53:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 314E52BF80 for ; Sat, 1 Dec 2018 18:53:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 201D92BFA0; Sat, 1 Dec 2018 18:53:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A1C9A2BF80 for ; Sat, 1 Dec 2018 18:53:14 +0000 (UTC) Received: from localhost ([::1]:42213 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gTAO9-00070w-Cb for patchwork-qemu-devel@patchwork.kernel.org; Sat, 01 Dec 2018 13:53:13 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gTANT-0005Wv-2A for qemu-devel@nongnu.org; Sat, 01 Dec 2018 13:52:31 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gTANO-0007Ur-13 for qemu-devel@nongnu.org; Sat, 01 Dec 2018 13:52:31 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:44074) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gTANN-0007Tm-Od for qemu-devel@nongnu.org; Sat, 01 Dec 2018 13:52:25 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id wB1InSkF099726; Sat, 1 Dec 2018 18:52:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=mime-version : message-id : date : from : to : cc : subject : content-type : content-transfer-encoding; s=corp-2018-07-02; bh=4DDOKaMkhqGk+chccGq8ozYBFpjIJC9OpB042CKxALk=; b=IRQKVFkDQw3mgL6uBPH+6GBFDhTxLdM2ALqgZPvfdV+Rbd1Rwssu235f4BNqP8/0GTHF C5CdaMuGiPZs0HckBIE6FRvi6p4rSSMNc8WucAVLfah9e2PBGraLbBFMEVybXSpOKsmB y0xAbzl0Z12SvQSDcOCgDfnZ2gHF+R/KsOiIjF17TFsE6GU3CdndP/uBrad/9sHPu1L6 etDcTBQS2XFKqxVWARcq2xBje00X8SKZ10ofM8nMDjLWymmpycrCH9K9etLLh42n5eRS pokLS02pSyyLnUNe12BRhTl+lLFqLH9IYAhhIiW2RIE+hkopChJaCcVB6S9Up6vbAWPQ OQ== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2p3jxr18hp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 01 Dec 2018 18:52:22 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id wB1IqMGN017652 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 1 Dec 2018 18:52:22 GMT Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id wB1IqLoq009290; Sat, 1 Dec 2018 18:52:21 GMT MIME-Version: 1.0 Message-ID: <099db937-3fa3-465e-9a23-a900df9adb7c@default> Date: Sat, 1 Dec 2018 10:52:21 -0800 (PST) From: Dongli Zhang To: X-Mailer: Zimbra on Oracle Beehive Content-Disposition: inline X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9094 signatures=668686 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812010175 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] [fuzzy] X-Received-From: 156.151.31.85 Subject: [Qemu-devel] vfio failure with intel 760p 128GB nvme X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: keith.busch@intel.com, alex.williamson@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Hi, I obtained below error when assigning an intel 760p 128GB nvme to guest via vfio on my desktop: qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0: vfio 0000:01:00.0: failed to add PCI capability 0x11[0x50]@0xb0: table & pba overlap, or they don't fit in BARs, or don't align This is because the msix table is overlapping with pba. According to below 'lspci -vv' from host, the distance between msix table offset and pba offset is only 0x100, although there are 22 entries supported (22 entries need 0x160). Looks qemu supports at most 0x800. # sudo lspci -vv ... ... 01:00.0 Non-Volatile memory controller: Intel Corporation Device f1a6 (rev 03) (prog-if 02 [NVM Express]) Subsystem: Intel Corporation Device 390b ... ... Capabilities: [b0] MSI-X: Enable- Count=22 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=0 offset=00002100 A patch below could workaround the issue and passthrough nvme successfully. Would you please help confirm if this can be regarded as bug in qemu, or issue with nvme hardware? Should we fix thin in qemu, or we should never use such buggy hardware with vfio? Thank you very much! Dongli Zhang diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5c7bd96..54fc25e 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1510,6 +1510,11 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) msix->pba_offset = pba & ~PCI_MSIX_FLAGS_BIRMASK; msix->entries = (ctrl & PCI_MSIX_FLAGS_QSIZE) + 1; + if (msix->table_bar == msix->pba_bar && + msix->table_offset + msix->entries * PCI_MSIX_ENTRY_SIZE > msix->pba_offset) { + msix->entries = (msix->pba_offset - msix->table_offset) / PCI_MSIX_ENTRY_SIZE; + } + /* * Test the size of the pba_offset variable and catch if it extends outside * of the specified BAR. If it is the case, we need to apply a hardware