From patchwork Fri Dec 17 03:08:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alexander Bulekov X-Patchwork-Id: 12683605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60C7AC433EF for ; Fri, 17 Dec 2021 03:33:01 +0000 (UTC) Received: from localhost ([::1]:40482 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1my3zQ-0006zl-GO for qemu-devel@archiver.kernel.org; Thu, 16 Dec 2021 22:33:00 -0500 Received: from eggs.gnu.org ([209.51.188.92]:46552) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1my3wr-0005ac-ER for qemu-devel@nongnu.org; Thu, 16 Dec 2021 22:30:21 -0500 Received: from mail-dm6nam10on2104.outbound.protection.outlook.com ([40.107.93.104]:47968 helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1my3wo-00040n-RK for qemu-devel@nongnu.org; Thu, 16 Dec 2021 22:30:21 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZZBD7xxbAFQ/+XueDxkaf1vlD94da2vo0fmJPbbYFw4u8Um7XgVwHXtzSkdWvKFt7du1b/51UEsdLaitaAKf/5T+gWW57cqNPudqoN6Vyvt5TEj+zE39Coa2uaIcwxi3e3gv2+ttY0yqj8XkvvCwk4GmOtkbUBoNCIDJgGMKuRUKVY+sBlu9JS7bHf75Xklb4/mTHDloB1O+6T9hcyF7Yaybwoyp2q8QoNlGAzTRZiZYvbd0NiP+SfVT4ODoDnolX/dy1uhyaXD4JmGcVYgEy2trEdsprMzPAhrfpr9daCadvi6PCiTUhJmJNmE0eYWVHjeyiUdkH986pDv6wP743w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RKpOc3H+1FYLhHGKC3eCKcd+cP0Yw/7T/oOe9PJG0aU=; b=XCMRIJ3cuQQVwcpuau/bRpR95oP5Ls5CKvl3UuP/OSFPdFfTXxCGqNtT/o3LaO4AqwgBp44tcKrXWWgoML7o+WHQpzgPEZRjeH83sX57aTOhyOGIQCbVQhW79LxeJOwyJ1/ZCzsIR8JXbZfJHmV6aPh8cqeC8x+2bEV/lXV+r2U08fAy4dH6mYJWuyyLa05I6gU3SH25hzKUS/1P2JrOVTWoL4LOExPPSOBkQBZjZpPavR3og9GsZKyr5EvtTaXNL9aBe87R5htTGbTubIPy5sqL0YFJIfUK2XIjjSILFEerEUuYOlmDAMU6ZqSSmUIyDr/URClavEQ8yyUVdCeMTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=bu.edu; dmarc=pass action=none header.from=bu.edu; dkim=pass header.d=bu.edu; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bu.edu; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RKpOc3H+1FYLhHGKC3eCKcd+cP0Yw/7T/oOe9PJG0aU=; b=II6E0weYrpGxF2HKcAkFXVKEODDnPhbZ1z9cHM0+A/DVgTaYmXFWxsbKA4Ccn4ju4U/ZnR/QVbwNqvUACfaOKQ3ruYLD87jyq/aouA0rL27pbpt0VRckCAuHgxFe/Dn73RaGlPXkljPYvForSl0XzfKOAp2qngKRVAtqJWiOpBI0P0RVpN9zyywqpXzufn0UA9Z1GxRoSQK/dG0L52KIkK3E9OAzdl4EK9ByhDSWcoqFqnWzEbP0mFgBaBeI5EM3UQHmm4BqYtkNSNysku7WHpxfmnVE8guv/7qqKBXUQGym1FnKSZTYWacPA+kCw6E3On34YCfZWzY2yD1/FxWXIA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=bu.edu; Received: from SN6PR03MB3871.namprd03.prod.outlook.com (2603:10b6:805:6d::32) by SA0PR03MB5594.namprd03.prod.outlook.com (2603:10b6:806:b2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4801.14; Fri, 17 Dec 2021 03:09:29 +0000 Received: from SN6PR03MB3871.namprd03.prod.outlook.com ([fe80::6cf1:fa6a:fb65:131]) by SN6PR03MB3871.namprd03.prod.outlook.com ([fe80::6cf1:fa6a:fb65:131%3]) with mapi id 15.20.4778.019; Fri, 17 Dec 2021 03:09:29 +0000 From: Alexander Bulekov To: qemu-devel@nongnu.org Subject: [RFC PATCH] memory: Fix dma-reentrancy issues at the MMIO level Date: Thu, 16 Dec 2021 22:08:58 -0500 Message-Id: <20211217030858.834822-1-alxndr@bu.edu> X-Mailer: git-send-email 2.33.0 X-ClientProxiedBy: MN2PR12CA0020.namprd12.prod.outlook.com (2603:10b6:208:a8::33) To SN6PR03MB3871.namprd03.prod.outlook.com (2603:10b6:805:6d::32) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4432c79d-66d6-4246-fdf9-08d9c10aa393 X-MS-TrafficTypeDiagnostic: SA0PR03MB5594:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2276; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VCCk1XStVixvZLXWP0djIwOe1nprEvOa/27SlYWXtHAPiwlwS7btNn5pEjFuj0BqZQGo+eGLKBWzJu9xBcWwx4t7xpQ+Z/2Z2HOv+NgXhY2XXnbfUxZtEUJ906blJ0ldupuJAXETmyoj0y8NneVu3ZiSWxMKjJK4t10s5kVwzeZoORdeWtrH+2z6vEPUcDnskhoOX5IWea3e5iFteBgoAE+uKOaDzJxXeNAdzXJ37McG9qbw9mDcYbCRNb+DXEAGSnRTBWGOEeLVE+tZNzwFFyvL63mho75XHYYYB93iQNXIV64+lbm5UNU4UilI1q3kdsfJaYCPXZXvYz85/AaHfzLpHSI6tVI7XPLaZqgqwRnOKYI2ynm205YkR04075Q93Lh1YX5ku6SjWQ/G2Zg40zt0AkuZVv45Dj5Q4vG/aWrSY4VDdCPK127OjJ7UMGcQaQ8xhWUZ4KjIWkXgkOlOOwNDUYieQIhOhMZEIyF3MTKalJcE55tNq2hSSGGWaBTnYAbrlcCErgZ1dxzMcO6x9gVaRNDudxMIQET3hdhGybiZ8XiS376muTyz1NaXmClKQe6yaz1V/7xZKDc8g7fiE7ystV/GJZmO2gMRyDimAk/EZKEE9q8Ptq4Q7DEfSTSQo1aANMFxoNAPQkKpU1tnDoCu2PATe0f08nb5ymnbz/XpSlS5BoR3ucVA7+/+uLMwGUEiiQtPYdkfJAPpo8iFA4ZyJ/pe2y0ZPSMQYTFmhjeIZl3I7jsnnTb96MaTKM1vSeyuttlLwQMJSLYTKtPupA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR03MB3871.namprd03.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(966005)(54906003)(7416002)(8676002)(2616005)(4326008)(52116002)(316002)(8936002)(36756003)(5660300002)(6666004)(786003)(6486002)(6916009)(66556008)(66946007)(38350700002)(66476007)(6512007)(26005)(38100700002)(6506007)(83380400001)(2906002)(86362001)(75432002)(508600001)(186003)(1076003)(45080400002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?Dnc/2eGuC/3FSRxBacYs7EvxcXkb?= =?utf-8?q?ubF7ayPo4tZ5CubA9GoddwgfEq7Em2kdDkyqYPBYIhb3z7JTiv2uH0fxAZk7GoMZ4?= =?utf-8?q?youbf46Atc6JibxDnKzuuMJe296luIaeNPrtl4zve5WD7ycSTSWEu6bZ+NmHVhb2A?= =?utf-8?q?+2Yi79dZVYpg48fkMsmHNejgynTBlEqQScYgllQa2n9Njbz+KpWbti08cQU6eJAua?= =?utf-8?q?8eF2czu6FyEqDFygq8AP04ShpszcbSs+PMxVaZP+46Ip52LNmhsDTi93AraRzsmdm?= =?utf-8?q?lMkMm9TPADOVmRfCSOdKaXtBs4o4y+hjovzzjQBfLp5kZ0Q1bIMWMOjSaheNvmFPO?= =?utf-8?q?8yTDqkOX9euiiUtFvqTZ8wpot9ghRORab6wRvtgTksy2ysIW5Djrz8dO86locJMVI?= =?utf-8?q?WtS9s/rsIOghFyHFNVbYWZgCIMs7JYB3FdRUc067EkXAXCpfZ1t3ajUIfmawJ0SuQ?= =?utf-8?q?L1A6nOmL26Ru+j54HyxHNZpjMz+1wBrXlzyA0UogI0eD9G3Vi+4yvrCt3QKx617Xm?= =?utf-8?q?OrM5NbF1hM12jNLpQPLOZtr6L4p2A6Yord+2Ipp/C85fMWfTN3uAEtaJVaZvIabX1?= =?utf-8?q?4fTz4vHQGutM321QRLIPk/OwxiecdplD2+A7cGykB95tWSFdWCOvTTyqKFXehA78t?= =?utf-8?q?0G73eN/R9IHkg6Vg0/jPtsNsqa56XmMUS/5H7zckfUOLqyAYQW8oNyydtsycjzG6j?= =?utf-8?q?Igt5e21FWasyFmHkaOq03GOZonDIX6FRaJkJFlmyMpS9IwlxFWjH+qOU4R750j0Lz?= =?utf-8?q?/g4mfm5GPwblbe72ek1atRnUwV7xr0Rr91MB0QDmDOCml4nLKQ6NLC7wOKJPzWK62?= =?utf-8?q?MIixVXPcUP8Z7PteW3jPmhpTUcjI7uQhVclHuU0ZMN0jBloSy9GdWeO8DVah4l4jC?= =?utf-8?q?8MWhA18IPGouSV5aurJL6hyMjRcSdm2Gp08y1DrZWoat8o4L1vMEw94/x45dtMMZh?= =?utf-8?q?DDGVL0Wi+4VL8xTlKhaLFLr4W7UtcW2peeJp9rZtXem2T6vHNyL4JrL4yzWJKcOr2?= =?utf-8?q?ruvUdMmZ3GAdHZyvV5cNtY7NHaY/1YSkFcdhL1BcDatPNvqREUqsXA1cZ1B+LBi4d?= =?utf-8?q?TRxjlG7fBwMP5PIhbGTJkJjt9RVgZIP32+VYSd1Ohenx6Rz0rPiVI9PNg96fHB+SG?= =?utf-8?q?2qCeprP6rVTm0Un4kgaH1ARvkpy5GM9Z47xmQ8BO9Dj6YvaholfzpFyBDEwomhoG3?= =?utf-8?q?uQ6M5DVWxnUyv523XnHx0Q4guQD6xe4OUCy0GobUoxN90auqYNXYS/v9B+OhT7yR4?= =?utf-8?q?8xn3e3pq/w1BSJrPdL7AblyGar6OQQfxRCtM6D5V49W4BojQhJxIHKQh1dAr5kiXO?= =?utf-8?q?qyw/6vL+CUypAIoha9Xn2rr14We4fSxFeuq+1TmWiBRMkI2Euwld254gBjCbHxwek?= =?utf-8?q?svlNVvLN+ttQgaoOSIPycJtUTonmZH8V9qeOtuqBIkC8s25aspfpRbxgqDbcIs1i2?= =?utf-8?q?1PFXJOasJa0r/c6SPMdq5wv+KA3lUtzbfVw=3D=3D?= X-OriginatorOrg: bu.edu X-MS-Exchange-CrossTenant-Network-Message-Id: 4432c79d-66d6-4246-fdf9-08d9c10aa393 X-MS-Exchange-CrossTenant-AuthSource: SN6PR03MB3871.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Dec 2021 03:09:28.9363 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d57d32cc-c121-488f-b07b-dfe705680c71 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TIcdFQtkOLjqk5qbP9HMwWBSEAOsk7hqBPHAJNtmLi4bqxU5pV5ZlOK5p8FtSxdt X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR03MB5594 Received-SPF: pass client-ip=40.107.93.104; envelope-from=alxndr@bu.edu; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: 0 X-Spam_score: -0.1 X-Spam_bar: / X-Spam_report: (-0.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HK_RANDOM_ENVFROM=0.999, HK_RANDOM_FROM=1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Laurent Vivier , Peter Maydell , Mauro Matteo Cascella , =?utf-8?q?Daniel_P=2E_Berrang?= =?utf-8?q?=C3=A9?= , Darren Kenny , David Hildenbrand , Jason Wang , Bin Meng , Li Qiang , Qiuhao Li , Peter Xu , Eduardo Habkost , Alexander Bulekov , Bandan Das , Gerd Hoffmann , Stefan Hajnoczi , Paolo Bonzini , Thomas Huth , "Edgar E . Iglesias" , =?utf-8?q?Philippe_Mathieu?= =?utf-8?q?-Daud=C3=A9?= Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Here's my shot at fixing dma-reentracy issues. This patch adds a flag to the DeviceState, which is set/checked when we call an accessor associated with the device's IO MRs. The problem, in short, as I understand it: For the vast majority of cases, we want to prevent a device from accessing it's own PIO/MMIO regions over DMA. This patch/solution is based on some assumptions: 1. DMA accesses that hit mmio regions are only dangerous if they end up interacting with memory-regions belonging to the device initiating the DMA. Not dangerous: sdhci_pio->dma_write->e1000_mmio Dangerous: sdhci_pio->dma_write->sdhci_mmio 2. Most devices do not interact with their own PIO/MMIO memory-regions using DMA. 3. There is no way for there to be multiple simultaneous accesses to a device's PIO/MMIO memory-regions. 4. All devices are QOMified :-) With this patch, I wasn't able to reproduce the issues being tracked here, with QTest reproducers: https://gitlab.com/qemu-project/qemu/-/issues/556 This passes the i386 qos/qtests for me and I was able to boot some linux/windows VMs with basic devices configured, without any apparent problems. Cc: Philippe Mathieu-Daudé Cc: Mauro Matteo Cascella Cc: Qiuhao Li Cc: Peter Xu Cc: Jason Wang Cc: David Hildenbrand Cc: Gerd Hoffmann Cc: Peter Maydell Cc: Li Qiang Cc: Thomas Huth Cc: Laurent Vivier Cc: Bandan Das Cc: Edgar E. Iglesias Cc: Darren Kenny Cc: Bin Meng Cc: Paolo Bonzini Cc: Stefan Hajnoczi Signed-off-by: Alexander Bulekov --- include/hw/qdev-core.h | 1 + softmmu/memory.c | 15 +++++++++++++++ softmmu/trace-events | 1 + 3 files changed, 17 insertions(+) diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index 20d3066595..32f7c779ab 100644 --- a/include/hw/qdev-core.h +++ b/include/hw/qdev-core.h @@ -193,6 +193,7 @@ struct DeviceState { int instance_id_alias; int alias_required_for_version; ResettableState reset; + int engaged_in_direct_io; }; struct DeviceListener { diff --git a/softmmu/memory.c b/softmmu/memory.c index 7340e19ff5..255c3c602f 100644 --- a/softmmu/memory.c +++ b/softmmu/memory.c @@ -532,6 +532,7 @@ static MemTxResult access_with_adjusted_size(hwaddr addr, uint64_t access_mask; unsigned access_size; unsigned i; + DeviceState *dev = NULL; MemTxResult r = MEMTX_OK; if (!access_size_min) { @@ -541,6 +542,17 @@ static MemTxResult access_with_adjusted_size(hwaddr addr, access_size_max = 4; } + /* Do not allow more than one simultanous access to a device's IO Regions */ + if (mr->owner && + !mr->ram_device && !mr->ram && !mr->rom_device && !mr->readonly) { + dev = (DeviceState *) object_dynamic_cast(mr->owner, TYPE_DEVICE); + if (dev->engaged_in_direct_io) { + trace_memory_region_reentrant_io(get_cpu_index(), mr, addr, size); + return MEMTX_ERROR; + } + dev->engaged_in_direct_io = true; + } + /* FIXME: support unaligned access? */ access_size = MAX(MIN(size, access_size_max), access_size_min); access_mask = MAKE_64BIT_MASK(0, access_size * 8); @@ -555,6 +567,9 @@ static MemTxResult access_with_adjusted_size(hwaddr addr, access_mask, attrs); } } + if (dev) { + dev->engaged_in_direct_io = false; + } return r; } diff --git a/softmmu/trace-events b/softmmu/trace-events index 9c88887b3c..d7228316db 100644 --- a/softmmu/trace-events +++ b/softmmu/trace-events @@ -13,6 +13,7 @@ memory_region_ops_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, u memory_region_ops_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size, const char *name) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u name '%s'" memory_region_subpage_read(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_subpage_write(int cpu_index, void *mr, uint64_t offset, uint64_t value, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" value 0x%"PRIx64" size %u" +memory_region_reentrant_io(int cpu_index, void *mr, uint64_t offset, unsigned size) "cpu %d mr %p offset 0x%"PRIx64" size %u" memory_region_ram_device_read(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_ram_device_write(int cpu_index, void *mr, uint64_t addr, uint64_t value, unsigned size) "cpu %d mr %p addr 0x%"PRIx64" value 0x%"PRIx64" size %u" memory_region_sync_dirty(const char *mr, const char *listener, int global) "mr '%s' listener '%s' synced (global=%d)"