From patchwork Thu Oct 24 06:52:31 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Niilona X-Patchwork-Id: 3090201 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Original-To: patchwork-linux-pci@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 54D3A9F2B7 for ; Thu, 24 Oct 2013 06:53:03 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5C49220398 for ; Thu, 24 Oct 2013 06:53:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6CF2D20387 for ; Thu, 24 Oct 2013 06:53:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753895Ab3JXGwy (ORCPT ); Thu, 24 Oct 2013 02:52:54 -0400 Received: from mail-la0-f41.google.com ([209.85.215.41]:42394 "EHLO mail-la0-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753894Ab3JXGwx (ORCPT ); Thu, 24 Oct 2013 02:52:53 -0400 Received: by mail-la0-f41.google.com with SMTP id el20so1542995lab.28 for ; Wed, 23 Oct 2013 23:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=WCUsgGxmswS7FEIPBbX+LkbmNDh18a79CSOrgLXXg/w=; b=A36ONs/TGOJEaa9N/BsMm9Bhw1niSIbg5iua+YYX6E2XyUlx+KSmfDL857R2UluAPX OO7vSc9Ew3FqBWcxMB1HpfDkQu91jMM78nD3SonvgZGr/RoPSqMt13gntUnPsSt+JGf/ q/VKmLzXa6h3ZoKX0Z1Z6H9QUYgKtYVTEMc0qzPwj+TKbnTDGQ2IJsz5s9EyObCr8tap N1vaVw2a75DJQMEKhyZlyIrGjLLjZ6oZZ+115/CU+wH+Ngov59a4uukY0t3/WVi49orM zjx+t3yN3nMys0sxicc7ABm0pts1wdfhZRQSCKgjpXp5N3iWS5BivXcDqqt1bjRqaZsZ HikA== X-Received: by 10.112.59.134 with SMTP id z6mr996609lbq.28.1382597572103; Wed, 23 Oct 2013 23:52:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.114.172.171 with HTTP; Wed, 23 Oct 2013 23:52:31 -0700 (PDT) From: Niilona Date: Thu, 24 Oct 2013 09:52:31 +0300 X-Google-Sender-Auth: edLSJL24OOAhgOCoFcDEkOB1VF0 Message-ID: Subject: PATCH: pci-subsystem: ixgbe: SR-IOV: kernel way to order driver's virtfn -entries is odd causing libvirt failures. To: linux-pci@vger.kernel.org, libvir-list Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Spam-Status: No, score=-7.2 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi. As Bjorn Helgaas recommend, this might be the item to discuss in the wider area. ----------------------------------- There is a behavior effecting virtfn -entries in sysfs, when amount of them increases over 10. Run VM's through LIBVIRT -> QEMU/KVM, this causes : - MAC address setting by LIBVIRT disordered ie. setting targeted to wrong VF. - VLAN setting by LIBVIRT overall failed Basics of this are in "/libvirt-x.x.x/src/util/virpci.c" ; in function below, which don't order virtfn entries correctly. /* * Returns virtual functions of a physical function */ int virPCIGetVirtualFunctions(const char *sysfs_path, virPCIDeviceAddressPtr **virtual_functions, unsigned int *num_virtual_functions) { But I let you to decide which is best way to fix this, as if every application reads "virtfn" entries from PF's directory, they all need to sort entries in alphabet. order to avoid this influence. So personally I did get over this by adding pre-zeroes to names to have them in sorted order in PF's directory. ---------------------------- Tested : - kernel 3.8.0-21 - libvirt 1.0.2 & libvirt 1.1.3 - ixgbe 3.11.33-k, ixgbe 3.17-3 & ixgbe 3.18.7 After applying this MAC/VLAN seen to work and VM's get contacted using MAC's & VLAN's, which been checked using tcpdump. P.S. Tested with 3.8.0-21, but patch constructed against "https://github.com/torvalds/linux.git", so there is possibilty this works in latest kernel, but haven't checked it. At least ordering scheme seen in "iov.c" in same form as before. ----------------------------- From fac886b86099dca1937730026da24c34857c4077 Mon Sep 17 00:00:00 2001 From: Niilo Minkkinen Date: Wed, 23 Oct 2013 14:50:07 +0300 Subject: [PATCH 1/1] Fixes ordering of virtfn -entries on drivers with more than 10 entries ( virtfn0...virtfn9 ). This effects on drivers like Intel 82599 "ixgbe", which have max 31 VF's (Virtual Function) on one PF (Physical Function). Currently, ordering is "diffused", which in case causes MAC/VLAN -settings on libvirt based to PCI -addressing failed. Signed-off-by: Niilo Minkkinen Signed-off-by: Tommy Varre --- drivers/pci/iov.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) goto failed1; @@ -149,7 +149,7 @@ static void virtfn_remove(struct pci_dev *dev, int id, int reset) __pci_reset_function(virtfn); } - sprintf(buf, "virtfn%u", id); + sprintf(buf, "virtfn%02u", id); sysfs_remove_link(&dev->dev.kobj, buf); /* * pci_stop_dev() could have been called for this virtfn already, diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index 21a7182..fbd1209 100644 --- a/drivers/pci/iov.c +++ b/drivers/pci/iov.c @@ -106,7 +106,7 @@ static int virtfn_add(struct pci_dev *dev, int id, int reset) mutex_unlock(&iov->dev->sriov->lock); rc = pci_bus_add_device(virtfn); - sprintf(buf, "virtfn%u", id); + sprintf(buf, "virtfn%02u", id); rc = sysfs_create_link(&dev->dev.kobj, &virtfn->dev.kobj, buf); if (rc)