From patchwork Mon Aug 26 07:16:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Manoj Vishwanathan X-Patchwork-Id: 13777331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 61343C5321D for ; Mon, 26 Aug 2024 07:23:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:Mime-Version:Date:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=QKa/kYIJy5Dsz6zNa7Wz95w7f1KZUUXm7OUrx2gbGns=; b=y7oK2oLY/uh4HssUPHCsy8ILfw waQ4SuuGBriZiXcVLTFOkcan3YS+Ytqh+M5RzCkvj/1fDaFKH0oXu/SpCFb+4vwHKO1J2h5wpaEfp 3VV3U4gIHQ8HDPSqfoq1hXPjYJE0/B4OcNbOZJ7ep/sx+ViLiyP1Iasrov+KqHmS64JBfIfIsnr2f eQNEK/CX4v8vmd1G51S94bj48wz8sGRPSXJXNBx6M2hxT+9WyKEm8Zch7qb+7/d3BIKUK35E3gFCI llGt/ukWnExk6H46RKi/HrsIGFRQDgnFgdscORAZ+zLjl/0O40uMr+WuLh0ZJQFZCXEGcCPk+2VX0 sfrTDgFg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1siU4a-00000006FDf-4Afv; Mon, 26 Aug 2024 07:23:33 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1siTy8-00000006EFF-3Wd2 for linux-arm-kernel@lists.infradead.org; Mon, 26 Aug 2024 07:16:54 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-6b3fec974e5so76447497b3.1 for ; Mon, 26 Aug 2024 00:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1724656610; x=1725261410; darn=lists.infradead.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=QKa/kYIJy5Dsz6zNa7Wz95w7f1KZUUXm7OUrx2gbGns=; b=wiNb9GVBISgmkoE35DDQ2JIvqXKANJd0TBezNHzlrlPJVF+/m5a1pAY8j8COmIfOvf kbCOOSoMlxqxYEk0GkkCxXDKrpOz4r3461bv7PBNlZhAtlCPyzGoG966RzTYzJAvVPz3 JdfOkrP/2v5um2+UuAMchLN5hxqmMe4zF5VRclV7TkiMwO3jOQaRw2ju34HthsIhLf+h SGiozMv0xPP5stzcKrLVTpPHAYF2vX6fkc3ojOp0P5MwQik+SIervK2X5xgpAmD8WOC/ h3lr1vPgAasEBAZ264RBYf5kOly4zx9iZufEwmwWsyPPMtQqxLY3rIHmcC8igegDANTr WOBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724656610; x=1725261410; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=QKa/kYIJy5Dsz6zNa7Wz95w7f1KZUUXm7OUrx2gbGns=; b=AGpby5qaJpifecktyZfXY5TbQ5T/QUfzkUP7LNfwMWQw3HemJZPd7sSxFOHwjHh9go pPlDA41Jz7wZeaKwZJU6NmRlPCIWWSFGn3WPQ5+Kq9vdbHvQMMiB7NonT6x5Qlxg9pEY luH8JIIBL9ot6CCTM9EBiifJAtH9nke6ccK0E6iqiGnIwFNehCkFIu9dEisezR1nyJ5t F79fuiZlk4//KInocOseZz72N9WU2B050m/R3EAPipIhBFA/UuW1hevpanlhkU7DLHWd 1fIilUKb2fgDC3ufHinF5RdwlhMqebrx0U1fTrHr2DP2eqeG7j3HMESwFkK1tjBG/13g fobg== X-Forwarded-Encrypted: i=1; AJvYcCWtOrE/S/kffqcqLOJBnbZAsGei6IPla3AsQOpw90Y5ebGOgUDWB7LNIwXguYJv2jxkLhQ6n73Y4BRiKoLSdpaS@lists.infradead.org X-Gm-Message-State: AOJu0YzBjnuAxTJzzLyAVJFLazZgCM5xP93+HxXR1cdM6S3MB8DGaBSf n37xH55WIjM9ZkEdOnkiD4fX6UHQp7ITVUQnCfUNJgJveTImPIt2vkF+fG9Zjiw7HCvLtz9iFnU 8KdWZWUUhn2GiFkLj3g== X-Google-Smtp-Source: AGHT+IGnGHU5UoNWSFReeRjXc204LYtfrKxuoFl5Xjmcsh2ogPXEKrWFLUTvo69hIgjLsAr3T4NF8XbqX4tetajV X-Received: from manojvishy.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:413f]) (user=manojvishy job=sendgmr) by 2002:a05:690c:4485:b0:6c1:298e:5a7 with SMTP id 00721157ae682-6c627ef7d23mr1963427b3.5.1724656610306; Mon, 26 Aug 2024 00:16:50 -0700 (PDT) Date: Mon, 26 Aug 2024 07:16:37 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.46.0.295.g3b9ea8a38a-goog Message-ID: <20240826071641.2691374-1-manojvishy@google.com> Subject: [PATCH v1 0/4] vfio/iommu: Flag to allow userspace to set DMA buffers system cacheable From: Manoj Vishwanathan To: Will Deacon , Robin Murphy , Joerg Roedel , Alex Williamson , linux-arm-kernel@lists.infradead.org Cc: kvm@vger.kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, David Dillow , Manoj Vishwanathan X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240826_001652_899459_983A53DC X-CRM114-Status: GOOD ( 19.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi maintainers, This RFC patch introduces the ability for userspace to control whether device (DMA) buffers are marked as cacheable, enabling them to utilize the system-level cache. The specific changes made in this patch are: * Introduce a new flag in `include/linux/iommu.h`: * `IOMMU_SYS_CACHE` - Indicates if the associated page should be cached in the system's cache hierarchy. * Add `VFIO_DMA_MAP_FLAG_SYS_CACHE` to `include/uapi/linux/vfio.h`: * Allows userspace to set the cacheable attribute to PTE when mapping DMA regions using the VFIO interface. * Update `vfio_iommu_type1.c`: * Handle the `VFIO_DMA_MAP_FLAG_SYS_CACHE` flag during DMA mapping operations. * Set the `IOMMU_SYS_CACHE` flag in the IOMMU page table entry if the `VFIO_DMA_MAP_FLAG_SYS_CACHE` is set. * arm/smmu/io-pgtable-arm: Set the MAIR for SYS_CACHE The reasoning behind these changes is to provide userspace with finer-grained control over memory access patterns for devices, potentially improving performance in scenarios where caching is beneficial. We saw in some of the use cases where the buffers were for transient data ( in and out without processing). I have tested this patch on certain arm64 machines and observed the following: * There is 14-21% improvement in jitter measurements, where the buffer on System Level Cache vs DDR buffers * There was not much of an improvement in latency in the diration of the tests that I have tried. I am open to feedback and suggestions for further improvements. Please let me know if you have any questions or concerns. Also, I am working on adding a check in the VFIO layer to ensure that if there is no architecture supported implementation for sys cache, if should not apply them. Thanks, Manoj Vishwanathan Manoj Vishwanathan (4): iommu: Add IOMMU_SYS_CACHE flag for system cache control iommu/io-pgtable-arm: Force outer cache for page-level MAIR via user flag vfio: Add VFIO_DMA_MAP_FLAG_SYS_CACHE to control device access to system cache vfio/type1: Add support for VFIO_DMA_MAP_FLAG_SYS_CACHE drivers/iommu/io-pgtable-arm.c | 3 +++ drivers/vfio/vfio_iommu_type1.c | 5 +++-- include/linux/iommu.h | 6 ++++++ include/uapi/linux/vfio.h | 1 + 4 files changed, 13 insertions(+), 2 deletions(-)