From patchwork Wed Oct 11 23:01:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brett Creeley X-Patchwork-Id: 13418047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C7CDCDB465 for ; Wed, 11 Oct 2023 23:01:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376551AbjJKXBi (ORCPT ); Wed, 11 Oct 2023 19:01:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376516AbjJKXBf (ORCPT ); Wed, 11 Oct 2023 19:01:35 -0400 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2053.outbound.protection.outlook.com [40.107.93.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1B1F9D; Wed, 11 Oct 2023 16:01:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d8KWL5/DhdtExv4nD4D+YrsDl0PxKzsuHfsxhW7a+/KjmWXhq12w68WGxxKlEgM+M4xK8tz85wRKDdRSjKRFSd1a8pZPdCURnQjCzXa+XB6Ktgdyyso4Y9sgOsc+pAMl1lAsL1qyKDRJq8aX7k/dm/sNgd6gx2n05roH/R88EkT6bxYJ69seRPSvjhfeU74Yxh7dvIUDGCYlQdDo1egACDlBv1Iu3+kYHokEnOH8BOy5hhs6EYsIohdoj1k2Al23ZYRRHPl9UwYVvlCtC5Pd/L9mgJs51Cec+lmKBF3Ioa44WcgBimWDKiuDapFcif04laM9+ekZ8LFwF6mvcJBWmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=K43OMtLi3SL2eBNzFpi18AFLkQEj3+mlb2T7C6faEBM=; b=dOAH31c6/llOlrHRS+VdgPX+RnUwJNqgGiBsxeZb5MnQBnqqg9+eziWuawWgMnWDnInIj9VbsKar+quanbJIbetShL1CebdNq04Li55VMFWZ0nmwM+8wDsppkqb+uhhbR280ZmvewsM5dX2gcf9vXx8hcJ1duq9MYOBQSoX4kkQ4sSopkwIL72VxFVT5iVEQc7BUfsjXZaKZ9lw9SpTL237GZvZW8nRV9zIjjJ6XWCHbpGmNyppXgcm6rYGuXCplIYGkjVgd6FOksBULEj9SJl0MiHHBykx6Iuh2hHHfmx+oAvcN6WULh1Fb2CnYALWJ3Hgd0CyaE1BMsUDAs20Aww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K43OMtLi3SL2eBNzFpi18AFLkQEj3+mlb2T7C6faEBM=; b=T3gNIIb5vz1dTQ7Y4C9R2fetLzCOl/cTN1s4+49aJEg0joICEEDeCsNxs6TtSL1KdHUbaeZFv6pKKUC6aVAnw542dtad4mAINkraYbg6ogN0zUiXmWBf7uJqOeGL6jWMAKFT4n+SElIPVfL/cBx97w3SRLPSSWKq0wek4SHotME= Received: from CY5PR15CA0139.namprd15.prod.outlook.com (2603:10b6:930:67::7) by CH0PR12MB8487.namprd12.prod.outlook.com (2603:10b6:610:18c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38; Wed, 11 Oct 2023 23:01:29 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:930:67:cafe::12) by CY5PR15CA0139.outlook.office365.com (2603:10b6:930:67::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38 via Frontend Transport; Wed, 11 Oct 2023 23:01:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.208) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6838.22 via Frontend Transport; Wed, 11 Oct 2023 23:01:29 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Wed, 11 Oct 2023 18:01:28 -0500 From: Brett Creeley To: , , , , , CC: , , , Subject: [PATCH v2 vfio 1/3] pds/vfio: Fix spinlock bad magic BUG Date: Wed, 11 Oct 2023 16:01:13 -0700 Message-ID: <20231011230115.35719-2-brett.creeley@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231011230115.35719-1-brett.creeley@amd.com> References: <20231011230115.35719-1-brett.creeley@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|CH0PR12MB8487:EE_ X-MS-Office365-Filtering-Correlation-Id: c9150eea-63ed-4dea-039d-08dbcaae014b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Yssbilc0eP1bR31IebWZbTWXwLj4KXogyjsLwZH4atxjhlNZSIJbtQEGWAZG0X6gyrp/E078vbBp0dFGZTwtRazOsdPbj2oIU/+gi6Uw0hW8MqnoVX4+j+U5V1uYrilQkJOz2Sqiht4obp1K5XkJ3vqYz1I77Nx65XROLI1qtUGqchjn0lvnC/mV7mHOA0nNkbnxL/noj+5Fhqm+WTIxStYzDgkpu8h13VAM6ynTQcZ/wP0GkvbPd3Swr2JIW/2CJEKc74SSp9jaiB16jh++atPdauv5ygXJ8kALpMlhuYaP9oO2zqWNxO7Ct0FUCCEQIDD4bEkIqlYB23U6jrtFYZ3vUqLNem9bBFNGJRoZNmLGjiDjFNpSSJ/02dCTErcw9sVr5MZql2UVObNoKtgpZWn5Fz9q3om+27jdASqPqw6PR6F0D5PhZVjNbNpr8LNNR858d2X8rpAS8NDMdujlNNYfYZ2RbZXRrW6E6E/cjNi/YYpns1ShO8njj4GwN3MhZnas3suXZnWAjoqI9Lwne3MSka5dfjhVsmrypvy1x+U7lIpVuh+LV0w9HQvLiFDE2F2SV5dwweDWU7rYM2lD82vAhEtHx1MpVtgO5Xq52z6qCb273iW90tYjLyo9xCrMmprqQMC7b1IxNxU7/OP32ZxbPQwKz3nIdcELBNl67CUpUGqvIvMrCay+jCm0tOpQdSITYMHqFmcs7c5tkajfFnR7FgdsNI9o86siKbarXeDEdFrwOJ08FeHEBKTTJwa9/R38HpcXUvX30BUKSGqf1g== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(346002)(376002)(396003)(39860400002)(136003)(230922051799003)(64100799003)(451199024)(82310400011)(1800799009)(186009)(46966006)(36840700001)(40470700004)(36756003)(86362001)(81166007)(356005)(41300700001)(478600001)(40480700001)(82740400003)(8936002)(44832011)(4326008)(2906002)(5660300002)(8676002)(6666004)(1076003)(336012)(83380400001)(426003)(47076005)(40460700003)(70206006)(54906003)(70586007)(316002)(26005)(36860700001)(2616005)(16526019)(110136005)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2023 23:01:29.6776 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c9150eea-63ed-4dea-039d-08dbcaae014b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB8487 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The following BUG was found when running on a kernel with CONFIG_DEBUG_SPINLOCK=y set: BUG: spinlock bad magic on CPU#2, bash/2481 lock: 0xffff8d6052a88f50, .magic: 00000000, .owner: /-1, .owner_cpu: 0 Call Trace: dump_stack_lvl+0x36/0x50 do_raw_spin_lock+0x79/0xc0 pds_vfio_reset+0x1d/0x60 [pds_vfio_pci] pci_reset_function+0x4b/0x70 reset_store+0x5b/0xa0 kernfs_fop_write_iter+0x137/0x1d0 vfs_write+0x2de/0x410 ksys_write+0x5d/0xd0 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 As shown, the .magic: 00000000, does not match the expected value. This is because spin_lock_init() is never called for the reset_lock. Fix this by calling spin_lock_init(&pds_vfio->reset_lock) when initializing the device. Signed-off-by: Brett Creeley Reviewed-by: Shannon Nelson Reviewed-by: Shameer Kolothum --- drivers/vfio/pci/pds/vfio_dev.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c index 649b18ee394b..c351f588fa13 100644 --- a/drivers/vfio/pci/pds/vfio_dev.c +++ b/drivers/vfio/pci/pds/vfio_dev.c @@ -155,6 +155,8 @@ static int pds_vfio_init_device(struct vfio_device *vdev) pds_vfio->vf_id = vf_id; + spin_lock_init(&pds_vfio->reset_lock); + vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P; vdev->mig_ops = &pds_vfio_lm_ops; vdev->log_ops = &pds_vfio_log_ops; From patchwork Wed Oct 11 23:01:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brett Creeley X-Patchwork-Id: 13418048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63B35C46CA1 for ; Wed, 11 Oct 2023 23:01:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376600AbjJKXBl (ORCPT ); Wed, 11 Oct 2023 19:01:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376524AbjJKXBf (ORCPT ); Wed, 11 Oct 2023 19:01:35 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2081.outbound.protection.outlook.com [40.107.94.81]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CBFBA9; Wed, 11 Oct 2023 16:01:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jxoFiwV3QjqF+C8cD4UKUKnY6gzwDvofHdXjZBsLuUCTByarJYdLFhEaF3R5wqJCEDGi0OpxVlQa7L+vH4M/IAXIfNnFeY6dXvdDs9AdI8QgWjcCq8P6TrQBauzjzVuamtXBs56d2PfdpgQ1U1Ra3+9uMChXmzCJAuzmBILRBepM50iLwRqHtcfMu4bRQDD/Z5KJhZiW6nAKDlYyM7JoJl93zEdDC76lHQDAO8Yz9uA3mNZW7hl7G8WaqVF98j+cDS29cazkSd+xu4NfBT3MY7ac6ozjaPaQw7tNVIcgj2yykvYCiEa9MvJOLmA63sSlhtfmWFpPhne51ZdmxkyMZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/lV4lyDxKEyVexv4om1bfCyS3zqujBEjXmB9A1ox1lQ=; b=cAzRzQ+qPUjMn42Op7Ak4wzJtWO0KQPOV0EhPoEIZ+HDvfxNYECxaDWQthyL/jXf8km65uSN3YeF9ZMBPAv/gotl+94MGCj3RgUPUAGHKv98YMLKx8qYZTF//ZdavxmW1sPD00O8NUJPXWknHvgyL4VvgQgpHo+8V+YD2ExMUCCVLjlx+kMzLkkqoCcu2vbsojxqX0+/V6HrgblrP4V5LC1tqZWJRs9u75bSGJYh41LH+4XB6BEu81/zArfkPYoQN/HSjsQsHssHhkJLVgJ2zYwHFQ+oszspoYCkVCnmNlKLrDjnHblt2gT9tE/f8+KfMj2eE401X5KNYVry9LxaTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/lV4lyDxKEyVexv4om1bfCyS3zqujBEjXmB9A1ox1lQ=; b=K7aLhVjk17MNJGKHsYGacIpe7i9xOV51kJNwi8+FHu3iWFWuJvmTJen+v+ijk666JD4iCW1bKwXniDgL0TBIzgNJbMgbwCKOB1j7lB6a3oNYsjAXAxXoZOmKKz1SDWZc/h6yxGHo9TK1eB6EBJ2jO39obJsMVLQkY51CQ7yN2l8= Received: from CY5PR15CA0147.namprd15.prod.outlook.com (2603:10b6:930:67::16) by MW6PR12MB8865.namprd12.prod.outlook.com (2603:10b6:303:23b::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38; Wed, 11 Oct 2023 23:01:32 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:930:67:cafe::ed) by CY5PR15CA0147.outlook.office365.com (2603:10b6:930:67::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.27 via Frontend Transport; Wed, 11 Oct 2023 23:01:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.208) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6838.22 via Frontend Transport; Wed, 11 Oct 2023 23:01:31 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Wed, 11 Oct 2023 18:01:30 -0500 From: Brett Creeley To: , , , , , CC: , , , Subject: [PATCH v2 vfio 2/3] pds/vfio: Fix mutex lock->magic != lock warning Date: Wed, 11 Oct 2023 16:01:14 -0700 Message-ID: <20231011230115.35719-3-brett.creeley@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231011230115.35719-1-brett.creeley@amd.com> References: <20231011230115.35719-1-brett.creeley@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|MW6PR12MB8865:EE_ X-MS-Office365-Filtering-Correlation-Id: d59c567b-0b2c-4863-f610-08dbcaae0280 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +h0ozZcY2FQTxbMXUy0v3UwYVCyj8QrkXp/PEJyJBqfK7TkFXcz86AwwrPhEY4y4wZ2obS/9hry2HHX8eac7/8mi5bwD04yd94jIIXdke8yVvv9bHzrQAMbhh7yeDr1mSjRdgSwhEtIaUc2foZGhA5sEiQkkjkovqC6GeJdeuJ0PSUADzMZgvq4gvmmOh+qZxXEHCoAn4TZ3e0QQYIAXj6dz9LWHhHIWSGv/KrdLhKTOUBIGOKqK7GKooCAerIJxvGBsY1eW+L5EcmcIogoAvaOG5CY1iBcwC9/Cc0mFCsGmQg9zYa42JmUGdnULq/Jt59Oc1LtwwtUg2PYAWRArGAQumyqEwylVSIbH/NJbcjPwnfJopZepKKtCDEFtoGpdOrnjMYpVvvE7Skwr6kKGfhg6j6QAeVQdX18SHd4xMX7zQDRWoyo6U/qNybzP+Dpq4TnETFD1gmh8f2TougKutaAj7sTLjKnXtTIj96XirbGh3WwJECW/rNay6f8cN+GEK2UdvAtDLw9JI/bBNxq5JqYYGVFp8yZOcr5TProBRo+my0CRSdfxCW1WQ7C3Vf9KIR9htK332dt5tEO/ZtKdf2ZrRTV5hfrmPI2JBGb9Hy/EtEbynOJg0vRaWpG+xECunv8e+sMFSneHrWS5Y+CbVXn5Q+u18BcOkFETbZ7xHgosHofeAPrPEkST+mJZfJVY61GJXSOPR8byAZh99WxaXlAgtK0KhUoiSlAhrjEHr4F92RELfHWoLD9m/lcXLNJysIU50Vg+1T7qXozr2dp1RA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(396003)(346002)(376002)(136003)(39860400002)(230922051799003)(82310400011)(451199024)(1800799009)(64100799003)(186009)(36840700001)(40470700004)(46966006)(4326008)(8936002)(2906002)(8676002)(40460700003)(356005)(26005)(336012)(426003)(2616005)(1076003)(5660300002)(81166007)(478600001)(16526019)(44832011)(40480700001)(82740400003)(70586007)(110136005)(36756003)(70206006)(54906003)(316002)(86362001)(47076005)(6666004)(36860700001)(41300700001)(83380400001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2023 23:01:31.7089 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d59c567b-0b2c-4863-f610-08dbcaae0280 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8865 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The following BUG was found when running on a kernel with CONFIG_DEBUG_MUTEXES=y set: DEBUG_LOCKS_WARN_ON(lock->magic != lock) RIP: 0010:mutex_trylock+0x10d/0x120 Call Trace: ? __warn+0x85/0x140 ? mutex_trylock+0x10d/0x120 ? report_bug+0xfc/0x1e0 ? handle_bug+0x3f/0x70 ? exc_invalid_op+0x17/0x70 ? asm_exc_invalid_op+0x1a/0x20 ? mutex_trylock+0x10d/0x120 ? mutex_trylock+0x10d/0x120 pds_vfio_reset+0x3a/0x60 [pds_vfio_pci] pci_reset_function+0x4b/0x70 reset_store+0x5b/0xa0 kernfs_fop_write_iter+0x137/0x1d0 vfs_write+0x2de/0x410 ksys_write+0x5d/0xd0 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 As shown, lock->magic != lock. This is because mutex_init(&pds_vfio->state_mutex) is called in the VFIO open path. So, if a reset is initiated before the VFIO device is opened the mutex will have never been initialized. Fix this by calling mutex_init(&pds_vfio->state_mutex) in the VFIO init path. Also, don't destroy the mutex on close because the device may be re-opened, which would cause mutex to be uninitialized. Fix this by implementing a driver specific vfio_device_ops.release callback that destroys the mutex before calling vfio_pci_core_release_dev(). Signed-off-by: Brett Creeley Reviewed-by: Shannon Nelson Reviewed-by: Shameer Kolothum --- drivers/vfio/pci/pds/vfio_dev.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c index c351f588fa13..306b1c25f016 100644 --- a/drivers/vfio/pci/pds/vfio_dev.c +++ b/drivers/vfio/pci/pds/vfio_dev.c @@ -155,6 +155,7 @@ static int pds_vfio_init_device(struct vfio_device *vdev) pds_vfio->vf_id = vf_id; + mutex_init(&pds_vfio->state_mutex); spin_lock_init(&pds_vfio->reset_lock); vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P; @@ -170,6 +171,16 @@ static int pds_vfio_init_device(struct vfio_device *vdev) return 0; } +static void pds_vfio_release_device(struct vfio_device *vdev) +{ + struct pds_vfio_pci_device *pds_vfio = + container_of(vdev, struct pds_vfio_pci_device, + vfio_coredev.vdev); + + mutex_destroy(&pds_vfio->state_mutex); + vfio_pci_core_release_dev(vdev); +} + static int pds_vfio_open_device(struct vfio_device *vdev) { struct pds_vfio_pci_device *pds_vfio = @@ -181,7 +192,6 @@ static int pds_vfio_open_device(struct vfio_device *vdev) if (err) return err; - mutex_init(&pds_vfio->state_mutex); pds_vfio->state = VFIO_DEVICE_STATE_RUNNING; pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING; @@ -201,14 +211,13 @@ static void pds_vfio_close_device(struct vfio_device *vdev) pds_vfio_put_save_file(pds_vfio); pds_vfio_dirty_disable(pds_vfio, true); mutex_unlock(&pds_vfio->state_mutex); - mutex_destroy(&pds_vfio->state_mutex); vfio_pci_core_close_device(vdev); } static const struct vfio_device_ops pds_vfio_ops = { .name = "pds-vfio", .init = pds_vfio_init_device, - .release = vfio_pci_core_release_dev, + .release = pds_vfio_release_device, .open_device = pds_vfio_open_device, .close_device = pds_vfio_close_device, .ioctl = vfio_pci_core_ioctl, From patchwork Wed Oct 11 23:01:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brett Creeley X-Patchwork-Id: 13418049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 363C1CDB465 for ; Wed, 11 Oct 2023 23:01:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376620AbjJKXBr (ORCPT ); Wed, 11 Oct 2023 19:01:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376558AbjJKXBj (ORCPT ); Wed, 11 Oct 2023 19:01:39 -0400 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5444A9; Wed, 11 Oct 2023 16:01:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LpB2p9JEvmiCADDVJ4nRAve4vcWXfCfHK63YV5rqJet8Gyra4KlmeuFnvisCoKQOt+ZJ1SoIHQ9KGFAoHakCZk5wcSbOHcd71COsC2co8ZyRu+JBEirHXZlUvSOD52+hhiPJnvcmeHB2UiRZNrWloS44uiipMCmuPFX4SaweJuPV93nhqD2iBvldf7gswh/yEVG7+F2xuQDsmu5ofGvv/Yz73/opS7xt9EaN0dQ5P829x2QgqgkrFSag3aLvb9F8NeMeXOCCr2IXvFyEIvhEuwVHyZ3eKMO/7S4hqHcGNIeD2O4jk3mX19ueyU0y+qYOZNWilVZ+oZcSxsIiAfpEjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0sEdZtOgpDIhG7UGxS07MqFGSIENgdo5lxNiz2tjrT8=; b=ijq20lZLsrvehxlIVFkMcU+hJk/dpHM/9/hqq6VPRtTadwxj52z9ORUR8+O9KOfWzlz0F9PXYIXsfuNvLxomwJ8UVFSjbArnO/UxSIwvx7FYpKOaA7x6RklZqFwflxOMTn8O23ylYfRadIX54WaiBMahvsrR56+fQB9K49LgU1L8MNxBDC0ejk58Kgr8jx1KNNWsGX5f83K9W4d0CyZ9K4cvFF6kZV6J9VyAMA5G9QwoFIpc43/sOQpUwU2wqSDBWVcfnheJkITjCeifJZfG9MWssdiQq/r5ECj6IZw3KMNzPxwKGIkd54UEM4MRrmUu9wqiM1a7JlVSf/neii3CxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=ziepe.ca smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0sEdZtOgpDIhG7UGxS07MqFGSIENgdo5lxNiz2tjrT8=; b=EVsVQk3kNN64gXprlwR8iUEJLL5EmgwQlbHPaE7XFazEHMkcht1ZqzpitsF6tQmikXsc6FecS+1R9zI8kT4Bi+Dyc3XaL4Cu2cMMs4kGcyXLQnKVy9mYrpaNxCaaFzo0xKSOCsdMBM+VZz8NMaha7vzW3ujEr6evkcBdkD6qrm8= Received: from CY5PR15CA0141.namprd15.prod.outlook.com (2603:10b6:930:67::6) by SJ0PR12MB6831.namprd12.prod.outlook.com (2603:10b6:a03:47d::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36; Wed, 11 Oct 2023 23:01:34 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:930:67:cafe::ed) by CY5PR15CA0141.outlook.office365.com (2603:10b6:930:67::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38 via Frontend Transport; Wed, 11 Oct 2023 23:01:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.208) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6838.22 via Frontend Transport; Wed, 11 Oct 2023 23:01:34 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Wed, 11 Oct 2023 18:01:32 -0500 From: Brett Creeley To: , , , , , CC: , , , Subject: [PATCH v2 vfio 3/3] pds/vfio: Fix possible sleep while in atomic context Date: Wed, 11 Oct 2023 16:01:15 -0700 Message-ID: <20231011230115.35719-4-brett.creeley@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231011230115.35719-1-brett.creeley@amd.com> References: <20231011230115.35719-1-brett.creeley@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|SJ0PR12MB6831:EE_ X-MS-Office365-Filtering-Correlation-Id: ba32ae97-7624-4a19-934f-08dbcaae0443 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 52mkIW8+L1MAgU9YOmKp5eTTHGwgcHyd6/YSPcIKR0lJ6gf91wg4oUGLuhgzmch9tZXKqsS4Sg4UWTnLXDlwAVUIO4sTl4uTEJdFSoaWW9qU4PR8P88iLJbQaFQM/CXlYiFgk/Cnty0mR35j59z3FrH2EBpNHwO87NkttUjqhSeKhbYfgYC3NbVN3sJLmjELRHIgbiT36cU61jdeJzg27+JndkpzGYsSUyaWsy97oy/VXHcx70ys4Sxgl2UF/CBK3CHSbZ65i11Ih5NcSzbItOiu2k8Rpr/HPCwzpDhkN9D9KsZlQMuHqvU79a0O0DKtW0NMzrFZm5xsTsl+vB0qkVBSNNeAvu0bmDIqriQdFaYBxuP9wGJ26EESIARl6B2zsRArM8iPnb89/0Eh3wdMsoV1DW2XItAhGe9NKuqpIhQEJ2va7VHmaJA6ucU6GIy5rculVj8NqDEl6vafQDCK4i/C8dopSZIcyejDCfpifuCq2G1mJkPFyF4SVboch485CuDQCmXspnp0JKCsi6ZIEpKHlJyXDbcXIYgws026bTJ7oMJsgTR7dbdTXADwztjAgIk2CATzcJgz/8oMiO3fRIAHopR48anXUl3Qaj/UrHnKC/aKU7Cw6eKjki5vHnDreRPzysnqUAdd+5j5nb02pAHIDvnGeFZ3ltcnNyVvssK4tiAOmyio9KezbEt3l7y0+OwR/9I+2WMrh+o1epouEunuE8GZc6/juPzJscbJfTRvWZlX6qCIeZlCVYm5rnaduAX66NC5M0btgxTFV/j8QA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(346002)(136003)(376002)(396003)(230922051799003)(451199024)(64100799003)(186009)(82310400011)(1800799009)(40470700004)(36840700001)(46966006)(40480700001)(478600001)(2906002)(47076005)(2616005)(336012)(1076003)(40460700003)(426003)(16526019)(83380400001)(36860700001)(26005)(110136005)(70206006)(316002)(54906003)(70586007)(4326008)(5660300002)(8676002)(966005)(6666004)(41300700001)(44832011)(8936002)(82740400003)(36756003)(356005)(81166007)(86362001)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2023 23:01:34.6621 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ba32ae97-7624-4a19-934f-08dbcaae0443 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6831 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The driver could possibly sleep while in atomic context resulting in the following call trace while CONFIG_DEBUG_ATOMIC_SLEEP=y is set: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:283 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 2817, name: bash preempt_count: 1, expected: 0 RCU nest depth: 0, expected: 0 Call Trace: dump_stack_lvl+0x36/0x50 __might_resched+0x123/0x170 mutex_lock+0x1e/0x50 pds_vfio_put_lm_file+0x1e/0xa0 [pds_vfio_pci] pds_vfio_put_save_file+0x19/0x30 [pds_vfio_pci] pds_vfio_state_mutex_unlock+0x2e/0x80 [pds_vfio_pci] pci_reset_function+0x4b/0x70 reset_store+0x5b/0xa0 kernfs_fop_write_iter+0x137/0x1d0 vfs_write+0x2de/0x410 ksys_write+0x5d/0xd0 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 This can happen if pds_vfio_put_restore_file() and/or pds_vfio_put_save_file() grab the mutex_lock(&lm_file->lock) while the spin_lock(&pds_vfio->reset_lock) is held, which can happen during while calling pds_vfio_state_mutex_unlock(). Fix this by using a local state variable to represent the current vfio device state and update all the next state values. Then release the spin_unlock(&pds_vfio->reset_lock) before checking if the current vfio device state is VFIO_DEVICE_STATE_ERROR. Then calling pds_vfio_put_restore_file() and pds_vfio_put_save_file() are no longer being done while holding the reset_lock. The only possible concerns are other threads that may call pds_vfio_put_restore_file(), pds_vfio_put_save_file(), and/or pds_vfio_dirty_disable(). However, those paths are already protected by the state mutex_lock(), which is held in this context. Reported-by: Dan Carpenter Closes: https://lore.kernel.org/kvm/1f9bc27b-3de9-4891-9687-ba2820c1b390@moroto.mountain/ Signed-off-by: Brett Creeley Reviewed-by: Shannon Nelson --- drivers/vfio/pci/pds/vfio_dev.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/drivers/vfio/pci/pds/vfio_dev.c b/drivers/vfio/pci/pds/vfio_dev.c index 306b1c25f016..cf7f639ba0ec 100644 --- a/drivers/vfio/pci/pds/vfio_dev.c +++ b/drivers/vfio/pci/pds/vfio_dev.c @@ -31,15 +31,17 @@ void pds_vfio_state_mutex_unlock(struct pds_vfio_pci_device *pds_vfio) again: spin_lock(&pds_vfio->reset_lock); if (pds_vfio->deferred_reset) { + enum vfio_device_mig_state current_state = pds_vfio->state; + pds_vfio->deferred_reset = false; - if (pds_vfio->state == VFIO_DEVICE_STATE_ERROR) { + pds_vfio->state = pds_vfio->deferred_reset_state; + pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING; + spin_unlock(&pds_vfio->reset_lock); + if (current_state == VFIO_DEVICE_STATE_ERROR) { pds_vfio_put_restore_file(pds_vfio); pds_vfio_put_save_file(pds_vfio); pds_vfio_dirty_disable(pds_vfio, false); } - pds_vfio->state = pds_vfio->deferred_reset_state; - pds_vfio->deferred_reset_state = VFIO_DEVICE_STATE_RUNNING; - spin_unlock(&pds_vfio->reset_lock); goto again; } mutex_unlock(&pds_vfio->state_mutex);