From patchwork Thu May 2 15:29:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: CLEMENT MATHIEU--DRIF X-Patchwork-Id: 13651804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1108FC4345F for ; Thu, 2 May 2024 15:30:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s2YNk-0003vs-6a; Thu, 02 May 2024 11:30:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s2YNb-0003pf-S1 for qemu-devel@nongnu.org; Thu, 02 May 2024 11:29:51 -0400 Received: from smarthost3.eviden.com ([80.78.11.84]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s2YNJ-0005zE-IP for qemu-devel@nongnu.org; Thu, 02 May 2024 11:29:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eviden.com; i=@eviden.com; q=dns/txt; s=mail; t=1714663773; x=1746199773; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=UM2oKYwJZ/PFCyVhG3rBpAgMLxmzRSRdT/0ePXifSdo=; b=RUSDFaGlBNv0InvFShfQBQa6Ag24FC1M1xDx8acoO3O5qa63bIgbCeZ/ g/WmS7h+dHQ1dBhfeQgZNhoZnU+XbPbKvP5PVamfSAXV1IVYT6d2DWJQE rw9ajC6yrT2gkf8Bz6a+Yk79p5qQQFIga2H1th/saqgBQlNP8OCQJJeBt dsCksPfbPaaCII2zA+H827aDdYsNQffpTK2GQ43u03edh1JNQdSz6WzvM 4THOm7iWicjqchVqHbMCfaPnWTY8WDVsPOfdpnKhpX+wDaDXadFQguvWj kaSp1B6sTJ1/4WfgnsTpm12jPbiNkyVJQRT+SOFQRhAYq7kAXf/prCPki A==; X-IronPort-AV: E=Sophos;i="6.07,247,1708383600"; d="scan'208";a="11730624" X-MGA-submission: MDGBOW60AbjKHZcGtBbmMTOWRYeEI/3NL1PpJ34PK7CB9abmHUXHRdu935Byd/hdcDWzNaZr1O11USRCEhLEG7Lbzv4727h8y46dQC4QqWSZKB/nveJy4Wi8/ck+spUEcOAS3X7rpP68LpzNyXmcmphltNm2ZFTI8PdLafUha3mrQQ== Received: from mail-am6eur05lp2105.outbound.protection.outlook.com (HELO EUR05-AM6-obe.outbound.protection.outlook.com) ([104.47.18.105]) by smarthost3.eviden.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 May 2024 17:29:23 +0200 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NLp1qHviByVLChC0B6PFHPdtJ47W7EtQbndNJ0qHHyVzmRZNNaXCEuGfwODcmFviplgis6uVOIJ7/CF5EefXLqT6pZrsTbbK+bbKh+Xdc8D/SmdLX6TdnjfRP0oZ6SAMwDmUTOxGN0tZOA76WpeJJMMSmKZ954+vJ+oCg6OvN++daRRtO10Alsuw2IHEHtzjJAl501JUWJwgfFpSs1qpNFBdHiNARAwBC0pFCiXaKBC76VOEikU/WIH53A8+6tWZfkad8QwdjCIV2v2eBUOAaO9r6vq+VrdH8tS0vIw+J5py/HneKn1yr+RyxJe5lHq721RG+fUhlmLtfAYDzWObQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UM2oKYwJZ/PFCyVhG3rBpAgMLxmzRSRdT/0ePXifSdo=; b=J3Gugyghl/1lQRj7vSI6G1YAKbfgbvEzm8F6pzUeEveZbW1r0beLq/DcXq0YsZKCPhXTIsQcMnohtZz8Vqjw1+rKDtw9P/tDrIZRalbFQCglSx+NXDJF8JSFdu+3l6k7TwV6Q+9Oj+FxRYMu3MoS5IVMHtHzOXd5xu+xTr4pggwli7B+yWS3avnmxfPGpba8G/mSyQnSpa6t8ZjTlhUirRUC6yZp5R9guSbt2sE1szjIMYDpwR8F3L0YqouJRNm+VBIbRZxb5tQWkEMwQvM4vjQolXSG6b5+UvMq0F49LdjRnndjy2X4YuvEkXkX3v/LC9D3u13lazCUSW30MiAyVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=eviden.com; dmarc=pass action=none header.from=eviden.com; dkim=pass header.d=eviden.com; arc=none Received: from AM8PR07MB7602.eurprd07.prod.outlook.com (2603:10a6:20b:24b::7) by AS8PR07MB7768.eurprd07.prod.outlook.com (2603:10a6:20b:351::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7544.29; Thu, 2 May 2024 15:29:21 +0000 Received: from AM8PR07MB7602.eurprd07.prod.outlook.com ([fe80::fbd7:ca71:b636:6f9d]) by AM8PR07MB7602.eurprd07.prod.outlook.com ([fe80::fbd7:ca71:b636:6f9d%4]) with mapi id 15.20.7519.035; Thu, 2 May 2024 15:29:21 +0000 From: CLEMENT MATHIEU--DRIF To: "qemu-devel@nongnu.org" CC: "jasowang@redhat.com" , "zhenzhong.duan@intel.com" , "kevin.tian@intel.com" , "yi.l.liu@intel.com" , "joao.m.martins@oracle.com" , "peterx@redhat.com" , CLEMENT MATHIEU--DRIF Subject: [PATCH ats_vtd v1 20/24] atc: generic ATC that can be used by PCIe devices that support SVM Thread-Topic: [PATCH ats_vtd v1 20/24] atc: generic ATC that can be used by PCIe devices that support SVM Thread-Index: AQHanKWByKvYDmYBd02klTL221t3Bg== Date: Thu, 2 May 2024 15:29:21 +0000 Message-ID: <20240502152810.187492-21-clement.mathieu--drif@eviden.com> References: <20240502152810.187492-1-clement.mathieu--drif@eviden.com> In-Reply-To: <20240502152810.187492-1-clement.mathieu--drif@eviden.com> Accept-Language: en-GB, fr-FR, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=eviden.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: AM8PR07MB7602:EE_|AS8PR07MB7768:EE_ x-ms-office365-filtering-correlation-id: ec7ed4a8-e661-4cd5-b22e-08dc6abca3cf x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; ARA:13230031|1800799015|366007|376005|38070700009; x-microsoft-antispam-message-info: =?utf-8?q?sL7FmSfONQ/MWe0ycGW01NfS5qteHoM?= =?utf-8?q?Chyf/OMsX/urINMNxez0F/64lM93URSKefjKgYMzpUuygP8KOK3/HQLQFzzNJcfm1?= =?utf-8?q?pzoUTvDSprRkJ67jqGumy+KZvS0TXlfo5lsn5VkPPIyEGj+F7GGIsjPVjkbNJ7kIi?= =?utf-8?q?k7Xsi2uF7JU0i0y51IsQbNcNZjltADJLWT+Z566wD9LdRrs8AW6Z2EVDd2xrtoIvT?= =?utf-8?q?45aUQH0Oz+e5YEWzrl6NyM3ShQsCh72OSIeC56iwEdBNvyI1lSUkWa0TDj0H1pcGX?= =?utf-8?q?572NQZ8kfcit5mqn0L8QbZmrEiRMF7umxqq3C4LZPSYgCuoznsPvaI7+qGy5L6g0T?= =?utf-8?q?tT5qVizuH1F74zZg/hMU6GHa3YltAnr72SZ1Z2GSwmwXpBHy7XXidbyN7xrSVrMfr?= =?utf-8?q?VXgozEdSYkPKEpeswdBmlJLApfhqZwE2l3DthoYA+vYbI9IYt1sD0M79aJYVvZvGR?= =?utf-8?q?ZOegaaeBDtuVn5hK5SuPpBhTpaPYUjmJZKaBcnB2QEHTYtFp/HIrtOtGANZFGzR9t?= =?utf-8?q?Mtw2MJfmUsGtDt7hPHPUJv2N1pgSWSiQx5o6ROSBNBux6I08IBzhIHHE+P1Qhrgr0?= =?utf-8?q?EcjabbwbKpjWXSl1MerSGKq9Cw0O+GRcgO/VPYCUfRoKGFHZeePi6Tyip1V2Ly6iC?= =?utf-8?q?+dQc3jn7zR2Xbzlj4WAAD5uz6H9mNpA2AgkVTpj3CgEulz9Lwdzr0zwQJ+2QLVpwz?= =?utf-8?q?pDKVC67zuh+uFoyXSWtR1Dyqt1n7ZZ1826BRJpRUwPCL8cPuQesCeIetINKiDAsKp?= =?utf-8?q?vgAX2nRLhgV9bjFoV4FVqM0fXlWSNiy98LpNrBajrxfW2SuJeX6/QAsSiEFOxccOc?= =?utf-8?q?9KZ/GFi9uqwwlgURKt7RJKOPJudbrqvdtQ8HzukQ7uebr2033Q6KWtiAv07UYlIu/?= =?utf-8?q?SpbEWTAIyVAzhovNyUvAaot52qx9bcGUu3BVye/qYcxziH9S7fOgH8XbqpUDukiWw?= =?utf-8?q?a0pZhOGzMrKiqUIiVuaaOUBHFDd2vZ3JPQKPeWX6/3Y3rCbiNn/rZugN6R3G64FrK?= =?utf-8?q?o5V+PPOQ4swlF+cCtnsuL59HirDkUhLDLzsHjtl+HOn19/msQUw1y3qgA6lujbIER?= =?utf-8?q?2jUQX/cs1+9NZSqzg6S7vwND4Fdfh9U53G9jzqikDWVbXeRu/WBycZkKK0sddRlvf?= =?utf-8?q?C+GbvLTYaoFbftn7DFYUIrgen7VoHDDt0xnCOQQDejkCJp8RpRIT+fEtmpHlozQJD?= =?utf-8?q?C1pCfVSzJYMSA8hVdUUinMsG5BgYjxhQ6skQ=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM8PR07MB7602.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(366007)(376005)(38070700009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?q?kD6w2vKs8gznVEJRZ+EJLLiDUt9L?= =?utf-8?q?BqXb+6S6RW33uazSqjWgW0JJoxu5Hz2L7Wfc7mwk/0ouuIHTbeeGwWUXPbr0nbKfL?= =?utf-8?q?EgyxYvYE7gJBgWnYFIsuAXm3gcU3ILc4fzy3WOIowrOQ5whC0uGVFEs47mWcswM6z?= =?utf-8?q?YXAfK3zercRiPVif2KaHL4lBZPeqpTr0R/QiQU2q7wm7bBoCgnaZjbvj5FmOu+yQq?= =?utf-8?q?IUhSQbmZFhe/1C7e2TRwJx75i7uAqeH02GunY0OyhDMuyFl39BuhV2GKVtug39KHa?= =?utf-8?q?2ieFbYPpugxxz46lJWk37ERKJuyt+/y3SGfBj2hbkq9gAItxHTZ0ESBjAMdPo6jnU?= =?utf-8?q?l3k/C3jtOeFkWwqzuGtY8ewBIPo2KcPwzEShl4cGDmdapLq2WgSiUKHpRDLr6hizt?= =?utf-8?q?Kq+1AHKt/d/ItTli/tjjJncKuwl/+RIHOlruUCxz3j1BTvv9FIH+1RrSJkzfiK1X0?= =?utf-8?q?ECKrGe+brlEIDVM3Syz38v6/vhbIdeE7xtrsxqwELQ78OVj5Nxw4/chsIfABplG2l?= =?utf-8?q?SsYX4EdZPOXUzrOkQnlrxx0XuVuIePa5F5f9htE0hFet0VTWZusJ9RZTRaRp3YDfq?= =?utf-8?q?e+OxwcfkmrZ+9Cj4+cmKgzczp+qviDoPfpBUSMjs9ijCXa6vJBTaXBgW6QddQfEM3?= =?utf-8?q?32gwWOi7548+bq7F2xE2zqSEaHWOS4MkqqTSNpWY4EG/+Lsgcjv9K7/B4LbG/WCO2?= =?utf-8?q?V0k+qW4FwQlCsW84Jjih4XAzt4rfKfuNGNMn+KABYbDfKWLqF5LZz8/VY2IN2XY8u?= =?utf-8?q?P1mSBlTDlk/UUetgLjV5vD8zzh3qTx9d/n+mNXIJefrT44Q1F59dUfOisSjQJwI8Q?= =?utf-8?q?wwRcj0+1nZolRqDp520vzQ7TL8+xYUwmcIcAG5Bxw0fi2MMC+a1MxwbTlGLO2lwLm?= =?utf-8?q?5cL5Ma4+pvHE2sqf9yw1KsMMfvCteZIj1lihbU/atQbsBqp/h4DX0xxvqNuEYV+Uh?= =?utf-8?q?XsKfskx81TvaIf/P7JqUCMtQkKyG/kY+wRbi41HJvimRL4bwYC+IM6RMx6ec/S5b8?= =?utf-8?q?H0+kv0wuJ/y1OwxbNzmWnzU6UnfX8DUEncm9VuedMoWtTJLRR+hRCbIwDJkiyI6Hb?= =?utf-8?q?g+fwRXEmuEuuoGM7fNtNsjA4zC99t+NRmahmjaMUMOld3ccuZ7oRH88/+/+bu1CSu?= =?utf-8?q?rvckYabR1ieb1Fw6r3W4RG/ygps3T10tEMmqxpwgR/L/e5FQ0cxrYfNIYUve5kSMD?= =?utf-8?q?xBbroYbFFQPypG2vSuCcC1yzB1Mp/dz6KqOcLabsjqJ9dxQYTbo6MrM1FcqBlmLCP?= =?utf-8?q?RzeUmnGdLTwnAqQcz+ZUOCqJ4sHvzoULljpnqt6jNQGJdtGzmVSjSvuwAahZDd3IO?= =?utf-8?q?0CIwY3jSzbCYlLZptNHGi9hSK1+EO5bSwSr9yE2Md2rIkeqzZmD2F5LaBq5d6xa76?= =?utf-8?q?ANXSJnSuXcTAdeCP9QSUvGCuPJUccwn99acp4fY4TBqZRleGaBKmeGrIt5+eUP6n7?= =?utf-8?q?tYIewSeF04J55XiqxlY36AFB5cibo39cpnNfqiClFxTWk4NqMNd5bPHvsyUDoD0Sg?= =?utf-8?q?tMrX0nGZMHSguw0O222oNetY+euYHxztI1YxjDwvop9csmCNQsRUBoM=3D?= Content-ID: MIME-Version: 1.0 X-OriginatorOrg: eviden.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM8PR07MB7602.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: ec7ed4a8-e661-4cd5-b22e-08dc6abca3cf X-MS-Exchange-CrossTenant-originalarrivaltime: 02 May 2024 15:29:21.3993 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 7d1c7785-2d8a-437d-b842-1ed5d8fbe00a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ySGOcTrVyw7Zv02b7SYp00JHLrdKU707IdGw6y8ADa+S0lggbdVEMt58G4XM9K8K57XVlxrEVZYk9EsYM871xEiG5qCqdPkkWvjuxQtlUFI1hkBfCpSIC8ri1gsXFKgW X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR07MB7768 Received-SPF: pass client-ip=80.78.11.84; envelope-from=clement.mathieu--drif@eviden.com; helo=smarthost3.eviden.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org As the SVM-capable devices will need to cache translations, we provide an first implementation. This cache uses a two-level design based on hash tables. The first level is indexed by a PASID and the second by a virtual addresse. Signed-off-by: Clément Mathieu--Drif --- tests/unit/meson.build | 1 + tests/unit/test-atc.c | 502 +++++++++++++++++++++++++++++++++++++++++ util/atc.c | 211 +++++++++++++++++ util/atc.h | 117 ++++++++++ util/meson.build | 1 + 5 files changed, 832 insertions(+) create mode 100644 tests/unit/test-atc.c create mode 100644 util/atc.c create mode 100644 util/atc.h diff --git a/tests/unit/meson.build b/tests/unit/meson.build index 228a21d03c..5c9a6fe9f4 100644 --- a/tests/unit/meson.build +++ b/tests/unit/meson.build @@ -52,6 +52,7 @@ tests = { 'test-interval-tree': [], 'test-xs-node': [qom], 'test-virtio-dmabuf': [meson.project_source_root() / 'hw/display/virtio-dmabuf.c'], + 'test-atc': [] } if have_system or have_tools diff --git a/tests/unit/test-atc.c b/tests/unit/test-atc.c new file mode 100644 index 0000000000..60fa60924a --- /dev/null +++ b/tests/unit/test-atc.c @@ -0,0 +1,502 @@ +/* + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + + * You should have received a copy of the GNU General Public License along + * with this program; if not, see . + */ + +#include "util/atc.h" + +static inline bool tlb_entry_equal(IOMMUTLBEntry *e1, IOMMUTLBEntry *e2) +{ + if (!e1 || !e2) { + return !e1 && !e2; + } + return e1->iova == e2->iova && + e1->addr_mask == e2->addr_mask && + e1->pasid == e2->pasid && + e1->perm == e2->perm && + e1->target_as == e2->target_as && + e1->translated_addr == e2->translated_addr; +} + +static void assert_lookup_equals(ATC *atc, IOMMUTLBEntry *target, + uint32_t pasid, hwaddr iova) +{ + IOMMUTLBEntry *result; + result = atc_lookup(atc, pasid, iova); + g_assert(tlb_entry_equal(result, target)); +} + +static void check_creation(uint64_t page_size, uint8_t address_width, + uint8_t levels, uint8_t level_offset, + bool should_work) { + ATC *atc = atc_new(page_size, address_width); + if (atc) { + if (atc->levels != levels || atc->level_offset != level_offset) { + g_assert(false); /* ATC created but invalid configuration : fail */ + } + atc_destroy(atc); + g_assert(should_work); + } else { + g_assert(!should_work); + } +} + +static void test_creation_parameters(void) +{ + check_creation(8, 39, 3, 9, false); + check_creation(4095, 39, 3, 9, false); + check_creation(4097, 39, 3, 9, false); + check_creation(8192, 48, 0, 0, false); + + check_creation(4096, 38, 0, 0, false); + check_creation(4096, 39, 3, 9, true); + check_creation(4096, 40, 0, 0, false); + check_creation(4096, 47, 0, 0, false); + check_creation(4096, 48, 4, 9, true); + check_creation(4096, 49, 0, 0, false); + check_creation(4096, 56, 0, 0, false); + check_creation(4096, 57, 5, 9, true); + check_creation(4096, 58, 0, 0, false); + + check_creation(16384, 35, 0, 0, false); + check_creation(16384, 36, 2, 11, true); + check_creation(16384, 37, 0, 0, false); + check_creation(16384, 46, 0, 0, false); + check_creation(16384, 47, 3, 11, true); + check_creation(16384, 48, 0, 0, false); + check_creation(16384, 57, 0, 0, false); + check_creation(16384, 58, 4, 11, true); + check_creation(16384, 59, 0, 0, false); +} + +static void test_single_entry(void) +{ + IOMMUTLBEntry entry = { + .iova = 0x123456789000ULL, + .addr_mask = 0xfffULL, + .pasid = 5, + .perm = IOMMU_RW, + .translated_addr = 0xdeadbeefULL, + }; + + ATC *atc = atc_new(4096, 48); + g_assert(atc); + + assert_lookup_equals(atc, NULL, entry.pasid, + entry.iova + (entry.addr_mask / 2)); + + atc_create_address_space_cache(atc, entry.pasid); + g_assert(atc_update(atc, &entry) == 0); + + assert_lookup_equals(atc, NULL, entry.pasid + 1, + entry.iova + (entry.addr_mask / 2)); + assert_lookup_equals(atc, &entry, entry.pasid, + entry.iova + (entry.addr_mask / 2)); + + atc_destroy(atc); +} + +static void test_page_boundaries(void) +{ + static const uint32_t pasid = 5; + static const hwaddr page_size = 4096; + + /* 2 consecutive entries */ + IOMMUTLBEntry e1 = { + .iova = 0x123456789000ULL, + .addr_mask = page_size - 1, + .pasid = pasid, + .perm = IOMMU_RW, + .translated_addr = 0xdeadbeefULL, + }; + IOMMUTLBEntry e2 = { + .iova = e1.iova + page_size, + .addr_mask = page_size - 1, + .pasid = pasid, + .perm = IOMMU_RW, + .translated_addr = 0x900df00dULL, + }; + + ATC *atc = atc_new(page_size, 48); + + atc_create_address_space_cache(atc, e1.pasid); + /* creating the address space twice should not be a problem */ + atc_create_address_space_cache(atc, e1.pasid); + + atc_update(atc, &e1); + atc_update(atc, &e2); + + assert_lookup_equals(atc, NULL, e1.pasid, e1.iova - 1); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova + e1.addr_mask); + g_assert((e1.iova + e1.addr_mask + 1) == e2.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova + e2.addr_mask); + assert_lookup_equals(atc, NULL, e2.pasid, e2.iova + e2.addr_mask + 1); + + assert_lookup_equals(atc, NULL, e1.pasid + 10, e1.iova); + assert_lookup_equals(atc, NULL, e2.pasid + 10, e2.iova); + atc_destroy(atc); +} + +static void test_huge_page(void) +{ + static const uint32_t pasid = 5; + static const hwaddr page_size = 4096; + IOMMUTLBEntry e1 = { + .iova = 0x123456600000ULL, + .addr_mask = 0x1fffffULL, + .pasid = pasid, + .perm = IOMMU_RW, + .translated_addr = 0xdeadbeefULL, + }; + hwaddr addr; + + ATC *atc = atc_new(page_size, 48); + + atc_create_address_space_cache(atc, e1.pasid); + atc_update(atc, &e1); + + for (addr = e1.iova; addr <= e1.iova + e1.addr_mask; addr += page_size) { + assert_lookup_equals(atc, &e1, e1.pasid, addr); + } + /* addr is now out of the huge page */ + assert_lookup_equals(atc, NULL, e1.pasid, addr); + atc_destroy(atc); +} + +static void test_pasid(void) +{ + hwaddr addr = 0xaaaaaaaaa000ULL; + IOMMUTLBEntry e1 = { + .iova = addr, + .addr_mask = 0xfffULL, + .pasid = 8, + .perm = IOMMU_RW, + .translated_addr = 0xdeadbeefULL, + }; + IOMMUTLBEntry e2 = { + .iova = addr, + .addr_mask = 0xfffULL, + .pasid = 2, + .perm = IOMMU_RW, + .translated_addr = 0xb001ULL, + }; + uint16_t i; + + ATC *atc = atc_new(4096, 48); + + atc_create_address_space_cache(atc, e1.pasid); + atc_create_address_space_cache(atc, e2.pasid); + atc_update(atc, &e1); + atc_update(atc, &e2); + + for (i = 0; i <= MAX(e1.pasid, e2.pasid) + 1; ++i) { + if (i == e1.pasid || i == e2.pasid) { + continue; + } + assert_lookup_equals(atc, NULL, i, addr); + } + assert_lookup_equals(atc, &e1, e1.pasid, addr); + assert_lookup_equals(atc, &e1, e1.pasid, addr); + atc_destroy(atc); +} + +static void test_large_address(void) +{ + IOMMUTLBEntry e1 = { + .iova = 0xaaaaaaaaa000ULL, + .addr_mask = 0xfffULL, + .pasid = 8, + .perm = IOMMU_RW, + .translated_addr = 0x5eeeeeedULL, + }; + IOMMUTLBEntry e2 = { + .iova = 0x1f00baaaaabf000ULL, + .addr_mask = 0xfffULL, + .pasid = e1.pasid, + .perm = IOMMU_RW, + .translated_addr = 0xdeadbeefULL, + }; + + ATC *atc = atc_new(4096, 57); + + atc_create_address_space_cache(atc, e1.pasid); + atc_update(atc, &e1); + atc_update(atc, &e2); + + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + atc_destroy(atc); +} + +static void test_bigger_page(void) +{ + IOMMUTLBEntry e1 = { + .iova = 0xaabbccdde000ULL, + .addr_mask = 0x1fffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0x5eeeeeedULL, + }; + hwaddr i; + + ATC *atc = atc_new(8192, 43); + + atc_create_address_space_cache(atc, e1.pasid); + atc_update(atc, &e1); + + i = e1.iova & (~e1.addr_mask); + assert_lookup_equals(atc, NULL, e1.pasid, i - 1); + while (i <= e1.iova + e1.addr_mask) { + assert_lookup_equals(atc, &e1, e1.pasid, i); + ++i; + } + assert_lookup_equals(atc, NULL, e1.pasid, i); + atc_destroy(atc); +} + +static void test_unknown_pasid(void) +{ + IOMMUTLBEntry e1 = { + .iova = 0xaabbccfff000ULL, + .addr_mask = 0xfffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0x5eeeeeedULL, + }; + + ATC *atc = atc_new(4096, 48); + g_assert(atc_update(atc, &e1) != 0); + assert_lookup_equals(atc, NULL, e1.pasid, e1.iova); +} + +static void test_invalidation(void) +{ + static uint64_t page_size = 4096; + IOMMUTLBEntry e1 = { + .iova = 0xaabbccddf000ULL, + .addr_mask = 0xfffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0x5eeeeeedULL, + }; + IOMMUTLBEntry e2 = { + .iova = 0xffe00000ULL, + .addr_mask = 0x1fffffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0xb000001ULL, + }; + IOMMUTLBEntry e3; + + ATC *atc = atc_new(page_size , 48); + atc_create_address_space_cache(atc, e1.pasid); + + atc_update(atc, &e1); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + atc_invalidate(atc, &e1); + assert_lookup_equals(atc, NULL, e1.pasid, e1.iova); + + atc_update(atc, &e1); + atc_update(atc, &e2); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + atc_invalidate(atc, &e2); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, NULL, e2.pasid, e2.iova); + + /* invalidate a huge page by invalidating a small region */ + for (hwaddr addr = e2.iova; addr <= (e2.iova + e2.addr_mask); + addr += page_size) { + atc_update(atc, &e2); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + e3 = (IOMMUTLBEntry){ + .iova = addr, + .addr_mask = page_size - 1, + .pasid = e2.pasid, + .perm = IOMMU_RW, + .translated_addr = 0, + }; + atc_invalidate(atc, &e3); + assert_lookup_equals(atc, NULL, e2.pasid, e2.iova); + } +} + +static void test_delete_address_space_cache(void) +{ + static uint64_t page_size = 4096; + IOMMUTLBEntry e1 = { + .iova = 0xaabbccddf000ULL, + .addr_mask = 0xfffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0x5eeeeeedULL, + }; + IOMMUTLBEntry e2 = { + .iova = e1.iova, + .addr_mask = 0xfffULL, + .pasid = 2, + .perm = IOMMU_RW, + .translated_addr = 0x5eeeeeedULL, + }; + + ATC *atc = atc_new(page_size , 48); + atc_create_address_space_cache(atc, e1.pasid); + + atc_update(atc, &e1); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + atc_invalidate(atc, &e2); /* unkown pasid : is a nop*/ + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + + atc_create_address_space_cache(atc, e2.pasid); + atc_update(atc, &e2); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + atc_invalidate(atc, &e1); + /* e1 has been removed but e2 is still there */ + assert_lookup_equals(atc, NULL, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + + atc_update(atc, &e1); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + + atc_delete_address_space_cache(atc, e2.pasid); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, NULL, e2.pasid, e2.iova); +} + +static void test_invalidate_entire_address_space(void) +{ + static uint64_t page_size = 4096; + IOMMUTLBEntry e1 = { + .iova = 0x1000ULL, + .addr_mask = 0xfffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0x5eedULL, + }; + IOMMUTLBEntry e2 = { + .iova = 0xfffffffff000ULL, + .addr_mask = 0xfffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0xbeefULL, + }; + IOMMUTLBEntry e3 = { + .iova = 0, + .addr_mask = 0xffffffffffffffffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0, + }; + + ATC *atc = atc_new(page_size , 48); + atc_create_address_space_cache(atc, e1.pasid); + + atc_update(atc, &e1); + atc_update(atc, &e2); + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + atc_invalidate(atc, &e3); + /* e1 has been removed but e2 is still there */ + assert_lookup_equals(atc, NULL, e1.pasid, e1.iova); + assert_lookup_equals(atc, NULL, e2.pasid, e2.iova); + + atc_destroy(atc); +} + +static void test_reset(void) +{ + static uint64_t page_size = 4096; + IOMMUTLBEntry e1 = { + .iova = 0x1000ULL, + .addr_mask = 0xfffULL, + .pasid = 1, + .perm = IOMMU_RW, + .translated_addr = 0x5eedULL, + }; + IOMMUTLBEntry e2 = { + .iova = 0xfffffffff000ULL, + .addr_mask = 0xfffULL, + .pasid = 2, + .perm = IOMMU_RW, + .translated_addr = 0xbeefULL, + }; + + ATC *atc = atc_new(page_size , 48); + atc_create_address_space_cache(atc, e1.pasid); + atc_create_address_space_cache(atc, e2.pasid); + atc_update(atc, &e1); + atc_update(atc, &e2); + + assert_lookup_equals(atc, &e1, e1.pasid, e1.iova); + assert_lookup_equals(atc, &e2, e2.pasid, e2.iova); + + atc_reset(atc); + + assert_lookup_equals(atc, NULL, e1.pasid, e1.iova); + assert_lookup_equals(atc, NULL, e2.pasid, e2.iova); +} + +static void test_get_max_number_of_pages(void) +{ + static uint64_t page_size = 4096; + hwaddr base = 0xc0fee000; /* aligned */ + ATC *atc = atc_new(page_size , 48); + g_assert(atc_get_max_number_of_pages(atc, base, page_size / 2) == 1); + g_assert(atc_get_max_number_of_pages(atc, base, page_size) == 1); + g_assert(atc_get_max_number_of_pages(atc, base, page_size + 1) == 2); + + g_assert(atc_get_max_number_of_pages(atc, base + 10, 1) == 1); + g_assert(atc_get_max_number_of_pages(atc, base + 10, page_size - 10) == 1); + g_assert(atc_get_max_number_of_pages(atc, base + 10, + page_size - 10 + 1) == 2); + g_assert(atc_get_max_number_of_pages(atc, base + 10, + page_size - 10 + 2) == 2); + + g_assert(atc_get_max_number_of_pages(atc, base + page_size - 1, 1) == 1); + g_assert(atc_get_max_number_of_pages(atc, base + page_size - 1, 2) == 2); + g_assert(atc_get_max_number_of_pages(atc, base + page_size - 1, 3) == 2); + + g_assert(atc_get_max_number_of_pages(atc, base + 10, page_size * 20) == 21); + g_assert(atc_get_max_number_of_pages(atc, base + 10, + (page_size * 20) + (page_size - 10)) + == 21); + g_assert(atc_get_max_number_of_pages(atc, base + 10, + (page_size * 20) + + (page_size - 10 + 1)) == 22); +} + +int main(int argc, char **argv) +{ + g_test_init(&argc, &argv, NULL); + g_test_add_func("/atc/test_creation_parameters", test_creation_parameters); + g_test_add_func("/atc/test_single_entry", test_single_entry); + g_test_add_func("/atc/test_page_boundaries", test_page_boundaries); + g_test_add_func("/atc/test_huge_page", test_huge_page); + g_test_add_func("/atc/test_pasid", test_pasid); + g_test_add_func("/atc/test_large_address", test_large_address); + g_test_add_func("/atc/test_bigger_page", test_bigger_page); + g_test_add_func("/atc/test_unknown_pasid", test_unknown_pasid); + g_test_add_func("/atc/test_invalidation", test_invalidation); + g_test_add_func("/atc/test_delete_address_space_cache", + test_delete_address_space_cache); + g_test_add_func("/atc/test_invalidate_entire_address_space", + test_invalidate_entire_address_space); + g_test_add_func("/atc/test_reset", test_reset); + g_test_add_func("/atc/test_get_max_number_of_pages", + test_get_max_number_of_pages); + return g_test_run(); +} diff --git a/util/atc.c b/util/atc.c new file mode 100644 index 0000000000..3332d7e2b8 --- /dev/null +++ b/util/atc.c @@ -0,0 +1,211 @@ +/* + * QEMU emulation of an ATC + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + + * You should have received a copy of the GNU General Public License along + * with this program; if not, see . + */ + +#include "util/atc.h" + + +#define PAGE_TABLE_ENTRY_SIZE 8 + +/* a pasid is hashed using the identity function */ +static guint atc_pasid_key_hash(gconstpointer v) +{ + return (guint)(uintptr_t)v; /* pasid */ +} + +/* pasid equality */ +static gboolean atc_pasid_key_equal(gconstpointer v1, gconstpointer v2) +{ + return v1 == v2; +} + +/* Hash function for IOTLB entries */ +static guint atc_addr_key_hash(gconstpointer v) +{ + hwaddr addr = (hwaddr)v; + return (guint)((addr >> 32) ^ (addr & 0xffffffffU)); +} + +/* Equality test for IOTLB entries */ +static gboolean atc_addr_key_equal(gconstpointer v1, gconstpointer v2) +{ + return (hwaddr)v1 == (hwaddr)v2; +} + +static void atc_address_space_free(void *as) +{ + g_hash_table_unref(as); +} + +/* return log2(val), or UINT8_MAX if val is not a power of 2 */ +static uint8_t ilog2(uint64_t val) +{ + uint8_t result = 0; + while (val != 1) { + if (val & 1) { + return UINT8_MAX; + } + + val >>= 1; + result += 1; + } + return result; +} + +ATC *atc_new(uint64_t page_size, uint8_t address_width) +{ + ATC *atc; + uint8_t log_page_size = ilog2(page_size); + /* number of bits each used to store all the intermediate indexes */ + uint64_t addr_lookup_indexes_size; + + if (log_page_size == UINT8_MAX) { + return NULL; + } + /* + * We only support page table entries of 8 (PAGE_TABLE_ENTRY_SIZE) bytes + * log2(page_size / 8) = log2(page_size) - 3 + * is the level offset + */ + if (log_page_size <= 3) { + return NULL; + } + + atc = g_new0(ATC, 1); + atc->address_spaces = g_hash_table_new_full(atc_pasid_key_hash, + atc_pasid_key_equal, + NULL, atc_address_space_free); + atc->level_offset = log_page_size - 3; + /* at this point, we know that page_size is a power of 2 */ + atc->min_addr_mask = page_size - 1; + addr_lookup_indexes_size = address_width - log_page_size; + if ((addr_lookup_indexes_size % atc->level_offset) != 0) { + goto error; + } + atc->levels = addr_lookup_indexes_size / atc->level_offset; + atc->page_size = page_size; + return atc; + +error: + g_free(atc); + return NULL; +} + +static inline GHashTable *atc_get_address_space_cache(ATC *atc, uint32_t pasid) +{ + return g_hash_table_lookup(atc->address_spaces, + (gconstpointer)(uintptr_t)pasid); +} + +void atc_create_address_space_cache(ATC *atc, uint32_t pasid) +{ + GHashTable *as_cache; + + as_cache = atc_get_address_space_cache(atc, pasid); + if (!as_cache) { + as_cache = g_hash_table_new_full(atc_addr_key_hash, + atc_addr_key_equal, + NULL, g_free); + g_hash_table_replace(atc->address_spaces, + (gpointer)(uintptr_t)pasid, as_cache); + } +} + +void atc_delete_address_space_cache(ATC *atc, uint32_t pasid) +{ + g_hash_table_remove(atc->address_spaces, (gpointer)(uintptr_t)pasid); +} + +int atc_update(ATC *atc, IOMMUTLBEntry *entry) +{ + IOMMUTLBEntry *value; + GHashTable *as_cache = atc_get_address_space_cache(atc, entry->pasid); + if (!as_cache) { + return -ENODEV; + } + value = g_memdup2(entry, sizeof(*value)); + g_hash_table_replace(as_cache, (gpointer)(entry->iova), value); + return 0; +} + +IOMMUTLBEntry *atc_lookup(ATC *atc, uint32_t pasid, hwaddr addr) +{ + IOMMUTLBEntry *entry; + hwaddr mask = atc->min_addr_mask; + hwaddr key = addr & (~mask); + GHashTable *as_cache = atc_get_address_space_cache(atc, pasid); + + if (!as_cache) { + return NULL; + } + + /* + * Iterate over the possible page sizes and try to find a hit + */ + for (uint8_t level = 0; level < atc->levels; ++level) { + entry = g_hash_table_lookup(as_cache, (gconstpointer)key); + if (entry) { + return entry; + } + mask = (mask << atc->level_offset) | ((1 << atc->level_offset) - 1); + key = addr & (~mask); + } + + return NULL; +} + +static gboolean atc_invalidate_entry_predicate(gpointer key, gpointer value, + gpointer user_data) +{ + IOMMUTLBEntry *entry = (IOMMUTLBEntry *)value; + IOMMUTLBEntry *target = (IOMMUTLBEntry *)user_data; + hwaddr target_mask = ~target->addr_mask; + hwaddr entry_mask = ~entry->addr_mask; + return ((target->iova & target_mask) == (entry->iova & target_mask)) || + ((target->iova & entry_mask) == (entry->iova & entry_mask)); +} + +void atc_invalidate(ATC *atc, IOMMUTLBEntry *entry) +{ + GHashTable *as_cache = atc_get_address_space_cache(atc, entry->pasid); + if (!as_cache) { + return; + } + g_hash_table_foreach_remove(as_cache, + atc_invalidate_entry_predicate, + entry); +} + +void atc_destroy(ATC *atc) +{ + g_hash_table_unref(atc->address_spaces); +} + +size_t atc_get_max_number_of_pages(ATC *atc, hwaddr addr, size_t length) +{ + hwaddr page_mask = ~(atc->min_addr_mask); + size_t result = (length / atc->page_size); + if ((((addr & page_mask) + length - 1) & page_mask) != + ((addr + length - 1) & page_mask)) { + result += 1; + } + return result + (length % atc->page_size != 0 ? 1 : 0); +} + +void atc_reset(ATC *atc) +{ + g_hash_table_remove_all(atc->address_spaces); +} diff --git a/util/atc.h b/util/atc.h new file mode 100644 index 0000000000..8be95f5cca --- /dev/null +++ b/util/atc.h @@ -0,0 +1,117 @@ +/* + * QEMU emulation of an ATC + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + + * You should have received a copy of the GNU General Public License along + * with this program; if not, see . + */ + +#ifndef UTIL_ATC_H +#define UTIL_ATC_H + +#include "qemu/osdep.h" +#include "exec/memory.h" + +typedef struct ATC { + GHashTable *address_spaces; /* Key : pasid, value : GHashTable */ + hwaddr min_addr_mask; + uint64_t page_size; + uint8_t levels; + uint8_t level_offset; +} ATC; + +/* + * atc_new: Create an ATC. + * + * Return an ATC or NULL if the creation failed + * + * @page_size: #PCIDevice doing the memory access + * @address_width: width of the virtual addresses used by the IOMMU (in bits) + */ +ATC *atc_new(uint64_t page_size, uint8_t address_width); + +/* + * atc_update: Insert or update an entry in the cache + * + * Return 0 if the operation succeeds, a negative error code otherwise + * + * The insertion will fail if the address space associated with this pasid + * has not been created with atc_create_address_space_cache + * + * @atc: the ATC to update + * @entry: the tlb entry to insert into the cache + */ +int atc_update(ATC *atc, IOMMUTLBEntry *entry); + +/* + * atc_create_address_space_cache: delare a new address space + * identified by a PASID + * + * @atc: the ATC to update + * @pasid: the pasid of the address space to be created + */ +void atc_create_address_space_cache(ATC *atc, uint32_t pasid); + +/* + * atc_delete_address_space_cache: delete an address space + * identified by a PASID + * + * @atc: the ATC to update + * @pasid: the pasid of the address space to be deleted + */ +void atc_delete_address_space_cache(ATC *atc, uint32_t pasid); + +/* + * atc_lookup: query the cache in a given address space + * + * @atc: the ATC to query + * @pasid: the pasid of the address space to query + * @addr: the virtual address to translate + */ +IOMMUTLBEntry *atc_lookup(ATC *atc, uint32_t pasid, hwaddr addr); + +/* + * atc_invalidate: invalidate an entry in the cache + * + * @atc: the ATC to update + * @entry: the entry to invalidate + */ +void atc_invalidate(ATC *atc, IOMMUTLBEntry *entry); + +/* + * atc_destroy: delete an ATC + * + * @atc: the cache to be deleted + */ +void atc_destroy(ATC *atc); + +/* + * atc_get_max_number_of_pages: get the number of pages a memory operation + * will access if all the pages concerned have the minimum size. + * + * This function can be used to determine the size of the result array to be + * allocated when issuing an ATS request. + * + * @atc: the cache + * @addr: start address + * @length: number of bytes accessed from addr + */ +size_t atc_get_max_number_of_pages(ATC *atc, hwaddr addr, size_t length); + +/* + * atc_reset: invalidates all the entries stored in the ATC + * + * @atc: the cache + */ +void atc_reset(ATC *atc); + +#endif diff --git a/util/meson.build b/util/meson.build index 0ef9886be0..a2e0e9e5d7 100644 --- a/util/meson.build +++ b/util/meson.build @@ -94,6 +94,7 @@ if have_block util_ss.add(files('hbitmap.c')) util_ss.add(files('hexdump.c')) util_ss.add(files('iova-tree.c')) + util_ss.add(files('atc.c')) util_ss.add(files('iov.c', 'uri.c')) util_ss.add(files('nvdimm-utils.c')) util_ss.add(files('block-helpers.c'))