From patchwork Mon Oct 21 20:48:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuanchu Xie X-Patchwork-Id: 13844664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D04A6D1713B for ; Mon, 21 Oct 2024 20:48:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 673FF6B009B; Mon, 21 Oct 2024 16:48:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 622BB6B009C; Mon, 21 Oct 2024 16:48:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EB186B009D; Mon, 21 Oct 2024 16:48:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2F23B6B009B for ; Mon, 21 Oct 2024 16:48:56 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6161E141C1C for ; Mon, 21 Oct 2024 20:48:39 +0000 (UTC) X-FDA: 82698798084.06.393414E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 343ECC0011 for ; Mon, 21 Oct 2024 20:48:47 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dYiPDVHY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3NL4WZwcKCEsB7n0pu7t11tyr.p1zyv07A-zzx8npx.14t@flex--yuanchu.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3NL4WZwcKCEsB7n0pu7t11tyr.p1zyv07A-zzx8npx.14t@flex--yuanchu.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729543697; a=rsa-sha256; cv=none; b=CBrp7MPtcp7Rx+fPx7GLU9kSeWJpWvXrsaPqNn/COb8ZRMXfkg95553kNqNDZUZLE2Acpi yfQs2PegOiTgLKVH0IbnmtQuLQ5tos7p9S+AStJgI12fhtFrWZfOkx+JO2Xgk592Z6dkck 7d2oxH7AY5TVqO6779d7pbjivvzDKSI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dYiPDVHY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3NL4WZwcKCEsB7n0pu7t11tyr.p1zyv07A-zzx8npx.14t@flex--yuanchu.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3NL4WZwcKCEsB7n0pu7t11tyr.p1zyv07A-zzx8npx.14t@flex--yuanchu.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729543697; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gFipddvuQRq5wKjN98NA7ZcCjLSN+r6af04knVlvJ7E=; b=oOlRhPe/BWgJHiRR6v0oTHwRPinMuR2CYVOmHCgh0bEgeJpz/f+ClT2VBHaBIAr+zAH8wP ERtK0qbpTpRAZXI02O1vYx3W8dN/nJ6r+7CG1yOHU+v5hbXSgryJSzES9WOYmSMaQA0mT+ LT7sTy148afuSV2ihIacQ5qsm/ZdMZk= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e290222fde4so6579341276.1 for ; Mon, 21 Oct 2024 13:48:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729543733; x=1730148533; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=gFipddvuQRq5wKjN98NA7ZcCjLSN+r6af04knVlvJ7E=; b=dYiPDVHYVy0tIll1mZ5U6/zyX3/YQZNlqKyM2fxsIGgLCLT0PxV2ZbLudLepjF6JaU 9jsCsjreZs3fmtz3KFDpo8hSfMmoqCbLUT4cADxCxpY270bh9zTj9Z1YKxyTPewHMBMJ uYgiC/AZYwnxtbJKL00UQ9Jzp5RiMHdtSnzjW9k/c8w8WU1DaucwbjLVTtH00cLplXrU mpqLLSmnItbpo/sYVM/RgfrldeR8dpI3SmPmfTZ5ioPIEh69+Kk5oq2vpIw1xAPgzGW1 eRS5HJExXWTa1KaBwLL/KDw8QouZoMPSvA4OEAiNZNugffzjWvaIizdJ7AHRepJFkv57 ZWaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729543733; x=1730148533; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gFipddvuQRq5wKjN98NA7ZcCjLSN+r6af04knVlvJ7E=; b=nQbyeOce1fHgMPt6DNRn1M/RtdfpTGj/VOqa3HnIr72mY6Y7btxiPu3Qyi9HktmXVP eb5Ihn8WgOwGcasd1IV7KJV8Ucpn6pLw5GrhJBhAX4PVGADA5GQKAdMwod8KQ0KW7GD4 JiBq7A9TIO2vR+Yvcxxe4bsbsoqAELlHqvTYXltNIKl6Fg1Xrw/rs7rjJvQfxZByFP0r 3+pKM+89TRWnilhXnKZ6edU4m0N0tG9uAshForrxLHGTKU755qI1xeBMI7EwRTh2cSIz qR4Xa5w3LuBsmpuk7EFcKfpsUCN4kL1SNKuz0fbmcqM5jXN4zQsPgKKsjDtmhVPlBD4m 7+2Q== X-Forwarded-Encrypted: i=1; AJvYcCUpmsPeXM0L0CyaFr5E1wXCpy3wb5ibBPPBkX6Tk0lyKc3uoP/aLBjuWNIXzVwgqb+FpgFdqT78LA==@kvack.org X-Gm-Message-State: AOJu0Yyyc/OW1IjYAmym1S8sznwXZ9ymXVx+zRlo13vKRMGtkOmYyUsH vn80Mv7ORt92LyYI8ESMQZTn0YCe1bydvqZrsUTBhq8a52sC79EZoXugDLkuG/X4792YnXUCgzx ZKaQ+zg== X-Google-Smtp-Source: AGHT+IECG34H/8CIOWyMGaxyVI44dtWpDZcKl0vI5Xy2jjIdWsoP0W167bOrDo7G5HUXH4q4YdwF90xCR8m5 X-Received: from yuanchu.svl.corp.google.com ([2620:15c:2c5:11:202a:34de:6d58:91d2]) (user=yuanchu job=sendgmr) by 2002:a25:c5c3:0:b0:e20:210d:4e4e with SMTP id 3f1490d57ef6-e2bb158ec32mr26663276.7.1729543732421; Mon, 21 Oct 2024 13:48:52 -0700 (PDT) Date: Mon, 21 Oct 2024 13:48:48 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.46.0 Message-ID: <20241021204849.1580384-1-yuanchu@google.com> Subject: [PATCH v4 1/2] virt: pvmemcontrol: control guest physical memory properties From: Yuanchu Xie To: Greg Kroah-Hartman Cc: Wei Liu , Rob Bradford , "Theodore Ts'o" , Pasha Tatashin , linux-kernel@vger.kernel.org, linux-mm@kvack.org, virtualization@lists.linux.dev, dev@lists.cloudhypervisor.org, Yuanchu Xie X-Rspam-User: X-Stat-Signature: o4pp8wxfrq9s3q4bbi9nqpxywpmqxsrr X-Rspamd-Queue-Id: 343ECC0011 X-Rspamd-Server: rspam02 X-HE-Tag: 1729543727-373200 X-HE-Meta: U2FsdGVkX1/pdt3NG/6PVxK7tkR2J39lBm47EDdNUaQfR3/EirpYObO2UkRWCYF2YxMmMnRwCghMBsmKr5VUJN+7Pplec5JzVGADbPo/V08PIGEHyg2JYuwUiaJT+ZY+mycTiLbc3JeLLMfUGCRONc1gy4yJse9TsJBkQ1FWKJRhAmXiielLKJD6eEJaKo+IQcR7Q5Bj4PZCRIVz+rgCepWrWh/W+lk3OvDOh/4s9OSzqlJzwnpMGhVtMs9lZgSKnjio+TlaZduvqWRgVdXESeOWZjNsw7km0YlcxlGuFCS9Bq5uJaGLUyBehQrdzRvsL6cATJ39G3hD3g5NJn49d0lo1R06oM0NYGvfmqntKU8jTxuQPaBXZSLSGi3noY3v2QYhr+Mtq8JRFmInGJ2+mEo0g4wtS/gUbsre1q4m6DgWQ4NPehOziTeWa8Zb53RVYmjGPavmo5x+clF1Rkkmpjuc9q5rAVF/2SNN0F86S2U61Tt7plM7zru1nMqxG6Ib3YBPWgMnkSUFeQ2+Yv7yrgBNe4MsxNjwjnZFFZI1ECMfPcGx4cXh7JuehmKBPZnFVooCgcMQ9r7NjF7Cuxw4UlxELKAy3We6UHAbzl1pwKYkXSbKbs2kPajjFZCTXYZzLN7CQLT8mh0AdOcG1PBsKYdQHELHGgZrRruiPyyhxXKTmldjClSzeU6yYmvlREZz4UQgn2hnLm+pSQVKceEYMqK3TuR1P3gSnhd+hYCQMzZTCq04nlXH9Q/C2PxkpPBBaZFQjw5adoO/q9nXt21wDd/1c1v04cPOq//0xgyYLZfCw+ux6/JPjcGOqkZgxP4gWhO3XqEgxUSl2c2Bw/X/sSHJaLzB7TpP+9DX6OagPxE3hY8fAU+n5BfJHMhgJENVLRwCLcKjbKRE6aLYw6yQiG7R8DbTDIsdHipFBkMvOBJPSlK0EqNcIrRsR6GH0nVFJwPEMhokH6G3xjScnCp 4PsTyCJu eKOgRD1te02gSrtit5Hb7oo8ln7tyw8aDaOs/jIO+2dBYRGxW+uV3LtSuaIKYb/LNEtAkl6BqUAt4MxHfjym2LwGAABSKAWvxzuu5xRDdmlxZoPegI8nVwghJxamVmYsxdPPswahGCUrHT4nzZkz5gDF8OwJ+EKZX1f4+8ZJEBJB4fvtguQldueT8S7kWssB3z0dbEmvZmXJYJxWmSTuYqpmpcPI7xGK6TKHneNtih/m/7ExjbqIg6H/nF8BQiePVKDNooC0PwF0zm+9nNQtIDsUrdZxV4UrHEUVquP2crjiBLpGOGtH1JmEHLildDSBk/4TjulWIPAXPHNiJiqMEZxcU/pl4T05kRkiUm/oNEBMrP8Qoh79/yK+e2GnRMZb53K9it8lyxmUKM0nVK0ChUjXe3VuwEoA2mqoSjfH1FMJg3RqTfVV01ZAbXNWV6xsc9f2hI5zRMGjFT246XXn42MoPXfFleGWnVUcJJYTByyzQ8tNEGCCeJX18ht470+Wu/CVo4ktf6d3BTN0E84fJyxpo82d0ycMSS2rVHWCUaBSHBi21jj+JvpKrRtQTukvCOJfbYduMb7X3pWsXUHe48tVHIWkr/+liBYBbhcT9UqFZPJ/JyVNIFoPalpjw1p7obpmbee++sHD+e16WaqBTBAJUIXjSWRaVp5tG8MQYLIt93R0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pvmemcontrol provides a way for the guest to control its physical memory properties and enables optimizations and security features. For example, the guest can provide information to the host where parts of a hugepage may be unbacked, or sensitive data may not be swapped out, etc. Pvmemcontrol allows guests to manipulate its gPTE entries in the SLAT, and also some other properties of the memory mapping on the host. This is achieved by using the KVM_CAP_SYNC_MMU capability. When this capability is available, the changes in the backing of the memory region on the host are automatically reflected into the guest. For example, an mmap() or madvise() that affects the region will be made visible immediately. There are two components of the implementation: the guest Linux driver and Virtual Machine Monitor (VMM) device. A guest-allocated shared buffer is negotiated per-cpu through a few PCI MMIO registers; the VMM device assigns a unique command for each per-cpu buffer. The guest writes its pvmemcontrol request in the per-cpu buffer, then writes the corresponding command into the command register, calling into the VMM device to perform the pvmemcontrol request. The synchronous per-cpu shared buffer approach avoids the kick and busy waiting that the guest would have to do with virtio virtqueue transport. User API From the userland, the pvmemcontrol guest driver is controlled via the ioctl(2) call. It requires CAP_SYS_ADMIN. ioctl(fd, PVMEMCONTROL_IOCTL, struct pvmemcontrol_buf *buf); Guest userland applications can tag VMAs and guest hugepages, or advise the host on how to handle sensitive guest pages. Supported function codes and their use cases: PVMEMCONTROL_FREE/REMOVE/DONTNEED/PAGEOUT. For the guest. One can reduce the struct page and page table lookup overhead by using hugepages backed by smaller pages on the host. These pvmemcontrol commands can allow for partial freeing of private guest hugepages to save memory. They also allow kernel memory, such as kernel stacks and task_structs to be paravirtualized if we expose kernel APIs. PVMEMCONTROL_MERGEABLE can inform the host KSM to deduplicate VM pages. PVMEMCONTROL_UNMERGEABLE is useful for security, when the VM does not want to share its backing pages. The same with PVMEMCONTROL_DONTDUMP, so sensitive pages are not included in a dump. MLOCK/UNLOCK can advise the host that sensitive information is not swapped out on the host. PVMEMCONTROL_MPROTECT_NONE/R/W/RW. For guest stacks backed by hugepages, stack guard pages can be handled in the host and memory can be saved in the hugepage. PVMEMCONTROL_SET_VMA_ANON_NAME is useful for observability and debugging how guest memory is being mapped on the host. Sample program making use of PVMEMCONTROL_DONTNEED: https://github.com/Dummyc0m/pvmemcontrol-user The VMM implementation is part of Cloud Hypervisor, the feature pvmemcontrol can be enabled and the VMM can then provide the device to a supporting guest. https://github.com/cloud-hypervisor/cloud-hypervisor Signed-off-by: Yuanchu Xie --- PATCH v3 -> v4 - changed dev_info to dev_dbg so the driver is quiet when it works properly. - Edited the changelog section to be included in the diffstat. PATCH v2 -> v3 - added PVMEMCONTROL_MERGEABLE for memory dedupe. - updated link to the upstream Cloud Hypervisor repo, and specify the feature required to enable the device. PATCH v1 -> v2 - fixed byte order sparse warning. ioread/write already does little-endian. - add include for linux/percpu.h RFC v1 -> PATCH v1 - renamed memctl to pvmemcontrol - defined device endianness as little endian v1: https://lore.kernel.org/linux-mm/20240518072422.771698-1-yuanchu@google.com/ v2: https://lore.kernel.org/linux-mm/20240612021207.3314369-1-yuanchu@google.com/ v3: https://lore.kernel.org/linux-mm/20241016193947.48534-1-yuanchu@google.com/ .../userspace-api/ioctl/ioctl-number.rst | 2 + drivers/virt/Kconfig | 2 + drivers/virt/Makefile | 1 + drivers/virt/pvmemcontrol/Kconfig | 10 + drivers/virt/pvmemcontrol/Makefile | 2 + drivers/virt/pvmemcontrol/pvmemcontrol.c | 459 ++++++++++++++++++ include/uapi/linux/pvmemcontrol.h | 76 +++ 7 files changed, 552 insertions(+) create mode 100644 drivers/virt/pvmemcontrol/Kconfig create mode 100644 drivers/virt/pvmemcontrol/Makefile create mode 100644 drivers/virt/pvmemcontrol/pvmemcontrol.c create mode 100644 include/uapi/linux/pvmemcontrol.h diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index e4be1378ba26..bde53044b53c 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -378,6 +378,8 @@ Code Seq# Include File Comments 0xCD 01 linux/reiserfs_fs.h 0xCE 01-02 uapi/linux/cxl_mem.h Compute Express Link Memory Devices 0xCF 02 fs/smb/client/cifs_ioctl.h +0xDA 00 uapi/linux/pvmemcontrol.h Pvmemcontrol Device + 0xDB 00-0F drivers/char/mwave/mwavepub.h 0xDD 00-3F ZFCP device driver see drivers/s390/scsi/ diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig index d8c848cf09a6..454e347a90cf 100644 --- a/drivers/virt/Kconfig +++ b/drivers/virt/Kconfig @@ -49,4 +49,6 @@ source "drivers/virt/acrn/Kconfig" source "drivers/virt/coco/Kconfig" +source "drivers/virt/pvmemcontrol/Kconfig" + endif diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile index f29901bd7820..3a1fd6e076ad 100644 --- a/drivers/virt/Makefile +++ b/drivers/virt/Makefile @@ -10,3 +10,4 @@ obj-y += vboxguest/ obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/ obj-$(CONFIG_ACRN_HSM) += acrn/ obj-y += coco/ +obj-$(CONFIG_PVMEMCONTROL) += pvmemcontrol/ diff --git a/drivers/virt/pvmemcontrol/Kconfig b/drivers/virt/pvmemcontrol/Kconfig new file mode 100644 index 000000000000..9fe16da23bd8 --- /dev/null +++ b/drivers/virt/pvmemcontrol/Kconfig @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0 +config PVMEMCONTROL + tristate "pvmemcontrol Guest Service Module" + depends on KVM_GUEST + help + pvmemcontrol is a guest kernel module that allows to communicate + with hypervisor / VMM and control the guest memory backing. + + To compile as a module, choose M, the module will be called + pvmemcontrol. If unsure, say N. diff --git a/drivers/virt/pvmemcontrol/Makefile b/drivers/virt/pvmemcontrol/Makefile new file mode 100644 index 000000000000..2fc087ef3ef5 --- /dev/null +++ b/drivers/virt/pvmemcontrol/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_PVMEMCONTROL) := pvmemcontrol.o diff --git a/drivers/virt/pvmemcontrol/pvmemcontrol.c b/drivers/virt/pvmemcontrol/pvmemcontrol.c new file mode 100644 index 000000000000..b4fff71cfdc8 --- /dev/null +++ b/drivers/virt/pvmemcontrol/pvmemcontrol.c @@ -0,0 +1,459 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Control guest physical memory properties by sending + * madvise-esque requests to the host VMM. + * + * Author: Yuanchu Xie + * Author: Pasha Tatashin + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define PCI_VENDOR_ID_GOOGLE 0x1ae0 +#define PCI_DEVICE_ID_GOOGLE_PVMEMCONTROL 0x0087 + +#define PVMEMCONTROL_COMMAND_OFFSET 0x08 +#define PVMEMCONTROL_REQUEST_OFFSET 0x00 +#define PVMEMCONTROL_RESPONSE_OFFSET 0x00 + +/* + * Magic values that perform the action specified when written to + * the command register. + */ +enum pvmemcontrol_transport_command { + PVMEMCONTROL_TRANSPORT_RESET = 0x060FE6D2, + PVMEMCONTROL_TRANSPORT_REGISTER = 0x0E359539, + PVMEMCONTROL_TRANSPORT_READY = 0x0CA8D227, + PVMEMCONTROL_TRANSPORT_DISCONNECT = 0x030F5DA0, + PVMEMCONTROL_TRANSPORT_ACK = 0x03CF5196, + PVMEMCONTROL_TRANSPORT_ERROR = 0x01FBA249, +}; + +/* Contains the function code and arguments for specific function */ +struct pvmemcontrol_vmm_call_le { + __le64 func_code; /* pvmemcontrol set function code */ + __le64 addr; /* hyper. page size aligned guest phys. addr */ + __le64 length; /* hyper. page size aligned length */ + __le64 arg; /* function code specific argument */ +}; + +/* Is filled on return to guest from VMM from most function calls */ +struct pvmemcontrol_vmm_ret_le { + __le32 ret_errno; /* on error, value of errno */ + __le32 ret_code; /* pvmemcontrol internal error code, on success 0 */ + __le64 ret_value; /* return value from the function call */ + __le64 arg0; /* currently unused */ + __le64 arg1; /* currently unused */ +}; + +struct pvmemcontrol_buf_le { + union { + struct pvmemcontrol_vmm_call_le call; + struct pvmemcontrol_vmm_ret_le ret; + }; +}; + +struct pvmemcontrol_percpu_channel { + struct pvmemcontrol_buf_le buf; + u64 buf_phys_addr; + u32 command; +}; + +struct pvmemcontrol { + void __iomem *base_addr; + struct device *device; + /* cache the info call */ + struct pvmemcontrol_vmm_ret pvmemcontrol_vmm_info; + struct pvmemcontrol_percpu_channel __percpu *pcpu_channels; +}; + +static DEFINE_RWLOCK(pvmemcontrol_lock); +static struct pvmemcontrol *pvmemcontrol __read_mostly; + +static void pvmemcontrol_write_command(void __iomem *base_addr, u32 command) +{ + iowrite32(command, base_addr + PVMEMCONTROL_COMMAND_OFFSET); +} + +static u32 pvmemcontrol_read_command(void __iomem *base_addr) +{ + return ioread32(base_addr + PVMEMCONTROL_COMMAND_OFFSET); +} + +static void pvmemcontrol_write_reg(void __iomem *base_addr, u64 buf_phys_addr) +{ + iowrite64_lo_hi(buf_phys_addr, base_addr + PVMEMCONTROL_REQUEST_OFFSET); +} + +static u32 pvmemcontrol_read_resp(void __iomem *base_addr) +{ + return ioread32(base_addr + PVMEMCONTROL_RESPONSE_OFFSET); +} + +static void pvmemcontrol_buf_call_to_le(struct pvmemcontrol_buf_le *le, + const struct pvmemcontrol_buf *buf) +{ + le->call.func_code = cpu_to_le64(buf->call.func_code); + le->call.addr = cpu_to_le64(buf->call.addr); + le->call.length = cpu_to_le64(buf->call.length); + le->call.arg = cpu_to_le64(buf->call.arg); +} + +static void pvmemcontrol_buf_ret_from_le(struct pvmemcontrol_buf *buf, + const struct pvmemcontrol_buf_le *le) +{ + buf->ret.ret_errno = le32_to_cpu(le->ret.ret_errno); + buf->ret.ret_code = le32_to_cpu(le->ret.ret_code); + buf->ret.ret_value = le64_to_cpu(le->ret.ret_value); + buf->ret.arg0 = le64_to_cpu(le->ret.arg0); + buf->ret.arg1 = le64_to_cpu(le->ret.arg1); +} + +static void pvmemcontrol_send_request(struct pvmemcontrol *pvmemcontrol, + struct pvmemcontrol_buf *buf) +{ + struct pvmemcontrol_percpu_channel *channel; + + preempt_disable(); + channel = this_cpu_ptr(pvmemcontrol->pcpu_channels); + + pvmemcontrol_buf_call_to_le(&channel->buf, buf); + pvmemcontrol_write_command(pvmemcontrol->base_addr, channel->command); + pvmemcontrol_buf_ret_from_le(buf, &channel->buf); + + preempt_enable(); +} + +static int __pvmemcontrol_vmm_call(struct pvmemcontrol_buf *buf) +{ + int err = 0; + + if (!pvmemcontrol) + return -EINVAL; + + read_lock(&pvmemcontrol_lock); + if (!pvmemcontrol) { + err = -EINVAL; + goto unlock; + } + if (buf->call.func_code == PVMEMCONTROL_INFO) { + memcpy(&buf->ret, &pvmemcontrol->pvmemcontrol_vmm_info, + sizeof(buf->ret)); + goto unlock; + } + + pvmemcontrol_send_request(pvmemcontrol, buf); + +unlock: + read_unlock(&pvmemcontrol_lock); + return err; +} + +static int pvmemcontrol_init_info(struct pvmemcontrol *dev, + struct pvmemcontrol_buf *buf) +{ + buf->call.func_code = PVMEMCONTROL_INFO; + + pvmemcontrol_send_request(dev, buf); + if (buf->ret.ret_code) + return buf->ret.ret_code; + + /* Initialize global pvmemcontrol_vmm_info */ + memcpy(&dev->pvmemcontrol_vmm_info, &buf->ret, + sizeof(dev->pvmemcontrol_vmm_info)); + dev_dbg(dev->device, + "pvmemcontrol_vmm_info.ret_errno = %u\n" + "pvmemcontrol_vmm_info.ret_code = %u\n" + "pvmemcontrol_vmm_info.major_version = %llu\n" + "pvmemcontrol_vmm_info.minor_version = %llu\n" + "pvmemcontrol_vmm_info.page_size = %llu\n", + dev->pvmemcontrol_vmm_info.ret_errno, + dev->pvmemcontrol_vmm_info.ret_code, + dev->pvmemcontrol_vmm_info.arg0, + dev->pvmemcontrol_vmm_info.arg1, + dev->pvmemcontrol_vmm_info.ret_value); + + return 0; +} + +static int pvmemcontrol_open(struct inode *inode, struct file *filp) +{ + struct pvmemcontrol_buf *buf = NULL; + + if (!capable(CAP_SYS_ADMIN)) + return -EACCES; + + /* Do not allow exclusive open */ + if (filp->f_flags & O_EXCL) + return -EINVAL; + + buf = kzalloc(sizeof(struct pvmemcontrol_buf), GFP_KERNEL); + if (!buf) + return -ENOMEM; + + /* Overwrite the misc device set by misc_register */ + filp->private_data = buf; + return 0; +} + +static int pvmemcontrol_release(struct inode *inode, struct file *filp) +{ + kfree(filp->private_data); + filp->private_data = NULL; + return 0; +} + +static long pvmemcontrol_ioctl(struct file *filp, unsigned int cmd, + unsigned long ioctl_param) +{ + struct pvmemcontrol_buf *buf = filp->private_data; + int err; + + if (cmd != PVMEMCONTROL_IOCTL_VMM) + return -EINVAL; + + if (copy_from_user(&buf->call, (void __user *)ioctl_param, + sizeof(struct pvmemcontrol_buf))) + return -EFAULT; + + err = __pvmemcontrol_vmm_call(buf); + if (err) + return err; + + if (copy_to_user((void __user *)ioctl_param, &buf->ret, + sizeof(struct pvmemcontrol_buf))) + return -EFAULT; + + return 0; +} + +static const struct file_operations pvmemcontrol_fops = { + .owner = THIS_MODULE, + .open = pvmemcontrol_open, + .release = pvmemcontrol_release, + .unlocked_ioctl = pvmemcontrol_ioctl, + .compat_ioctl = compat_ptr_ioctl, +}; + +static struct miscdevice pvmemcontrol_dev = { + .minor = MISC_DYNAMIC_MINOR, + .name = KBUILD_MODNAME, + .fops = &pvmemcontrol_fops, +}; + +static int pvmemcontrol_connect(struct pvmemcontrol *pvmemcontrol) +{ + int cpu; + u32 cmd; + + pvmemcontrol_write_command(pvmemcontrol->base_addr, + PVMEMCONTROL_TRANSPORT_RESET); + cmd = pvmemcontrol_read_command(pvmemcontrol->base_addr); + if (cmd != PVMEMCONTROL_TRANSPORT_ACK) { + dev_err(pvmemcontrol->device, + "failed to reset device, cmd 0x%x\n", cmd); + return -EINVAL; + } + + for_each_possible_cpu(cpu) { + struct pvmemcontrol_percpu_channel *channel = + per_cpu_ptr(pvmemcontrol->pcpu_channels, cpu); + + pvmemcontrol_write_reg(pvmemcontrol->base_addr, + channel->buf_phys_addr); + pvmemcontrol_write_command(pvmemcontrol->base_addr, + PVMEMCONTROL_TRANSPORT_REGISTER); + + cmd = pvmemcontrol_read_command(pvmemcontrol->base_addr); + if (cmd != PVMEMCONTROL_TRANSPORT_ACK) { + dev_err(pvmemcontrol->device, + "failed to register pcpu buf, cmd 0x%x\n", cmd); + return -EINVAL; + } + channel->command = + pvmemcontrol_read_resp(pvmemcontrol->base_addr); + } + + pvmemcontrol_write_command(pvmemcontrol->base_addr, + PVMEMCONTROL_TRANSPORT_READY); + cmd = pvmemcontrol_read_command(pvmemcontrol->base_addr); + if (cmd != PVMEMCONTROL_TRANSPORT_ACK) { + dev_err(pvmemcontrol->device, + "failed to ready device, cmd 0x%x\n", cmd); + return -EINVAL; + } + return 0; +} + +static int pvmemcontrol_disconnect(struct pvmemcontrol *pvmemcontrol) +{ + u32 cmd; + + pvmemcontrol_write_command(pvmemcontrol->base_addr, + PVMEMCONTROL_TRANSPORT_DISCONNECT); + + cmd = pvmemcontrol_read_command(pvmemcontrol->base_addr); + if (cmd != PVMEMCONTROL_TRANSPORT_ERROR) { + dev_err(pvmemcontrol->device, + "failed to disconnect device, cmd 0x%x\n", cmd); + return -EINVAL; + } + return 0; +} + +static int pvmemcontrol_alloc_percpu_channels(struct pvmemcontrol *pvmemcontrol) +{ + int cpu; + + pvmemcontrol->pcpu_channels = alloc_percpu_gfp( + struct pvmemcontrol_percpu_channel, GFP_ATOMIC | __GFP_ZERO); + if (!pvmemcontrol->pcpu_channels) + return -ENOMEM; + + for_each_possible_cpu(cpu) { + struct pvmemcontrol_percpu_channel *channel = + per_cpu_ptr(pvmemcontrol->pcpu_channels, cpu); + phys_addr_t buf_phys = per_cpu_ptr_to_phys(&channel->buf); + + channel->buf_phys_addr = buf_phys; + } + return 0; +} + +static int pvmemcontrol_init(struct device *device, void __iomem *base_addr) +{ + struct pvmemcontrol_buf *buf = NULL; + struct pvmemcontrol *dev = NULL; + int err = 0; + + err = misc_register(&pvmemcontrol_dev); + if (err) + return err; + + /* We take a spinlock for a long time, but this is only during init. */ + write_lock(&pvmemcontrol_lock); + if (READ_ONCE(pvmemcontrol)) { + dev_warn(device, "multiple pvmemcontrol devices present\n"); + err = -EEXIST; + goto fail_free; + } + + dev = kzalloc(sizeof(struct pvmemcontrol), GFP_ATOMIC); + buf = kzalloc(sizeof(struct pvmemcontrol_buf), GFP_ATOMIC); + if (!dev || !buf) { + err = -ENOMEM; + goto fail_free; + } + + dev->base_addr = base_addr; + dev->device = device; + + err = pvmemcontrol_alloc_percpu_channels(dev); + if (err) + goto fail_free; + + err = pvmemcontrol_connect(dev); + if (err) + goto fail_free; + + err = pvmemcontrol_init_info(dev, buf); + if (err) + goto fail_free; + + WRITE_ONCE(pvmemcontrol, dev); + write_unlock(&pvmemcontrol_lock); + return 0; + +fail_free: + write_unlock(&pvmemcontrol_lock); + kfree(dev); + kfree(buf); + misc_deregister(&pvmemcontrol_dev); + return err; +} + +static int pvmemcontrol_pci_probe(struct pci_dev *dev, + const struct pci_device_id *id) +{ + void __iomem *base_addr; + int err; + + err = pcim_enable_device(dev); + if (err < 0) + return err; + + base_addr = pcim_iomap(dev, 0, 0); + if (!base_addr) + return -ENOMEM; + + err = pvmemcontrol_init(&dev->dev, base_addr); + if (err) + pci_disable_device(dev); + + return err; +} + +static void pvmemcontrol_pci_remove(struct pci_dev *pci_dev) +{ + int err; + struct pvmemcontrol *dev; + + write_lock(&pvmemcontrol_lock); + dev = READ_ONCE(pvmemcontrol); + if (!dev) { + err = -EINVAL; + dev_err(&pci_dev->dev, "cleanup called when uninitialized\n"); + write_unlock(&pvmemcontrol_lock); + return; + } + + /* disconnect */ + err = pvmemcontrol_disconnect(dev); + if (err) + dev_err(&pci_dev->dev, "device did not ack disconnect\n"); + /* free percpu channels */ + free_percpu(dev->pcpu_channels); + + kfree(dev); + WRITE_ONCE(pvmemcontrol, NULL); + write_unlock(&pvmemcontrol_lock); + misc_deregister(&pvmemcontrol_dev); +} + +static const struct pci_device_id pvmemcontrol_pci_id_tbl[] = { + { PCI_DEVICE(PCI_VENDOR_ID_GOOGLE, PCI_DEVICE_ID_GOOGLE_PVMEMCONTROL) }, + { 0 } +}; +MODULE_DEVICE_TABLE(pci, pvmemcontrol_pci_id_tbl); + +static struct pci_driver pvmemcontrol_pci_driver = { + .name = "pvmemcontrol", + .id_table = pvmemcontrol_pci_id_tbl, + .probe = pvmemcontrol_pci_probe, + .remove = pvmemcontrol_pci_remove, +}; +module_pci_driver(pvmemcontrol_pci_driver); + +MODULE_AUTHOR("Yuanchu Xie "); +MODULE_DESCRIPTION("pvmemcontrol Guest Service Module"); +MODULE_LICENSE("GPL"); diff --git a/include/uapi/linux/pvmemcontrol.h b/include/uapi/linux/pvmemcontrol.h new file mode 100644 index 000000000000..31b366dee796 --- /dev/null +++ b/include/uapi/linux/pvmemcontrol.h @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Userspace interface for /dev/pvmemcontrol + * pvmemcontrol Guest Memory Service Module + * + * Copyright (c) 2024, Google LLC. + * Yuanchu Xie + * Pasha Tatashin + */ + +#ifndef _UAPI_PVMEMCONTROL_H +#define _UAPI_PVMEMCONTROL_H + +#include +#include +#include + +/* Contains the function code and arguments for specific function */ +struct pvmemcontrol_vmm_call { + __u64 func_code; /* pvmemcontrol set function code */ + __u64 addr; /* hyper. page size aligned guest phys. addr */ + __u64 length; /* hyper. page size aligned length */ + __u64 arg; /* function code specific argument */ +}; + +/* Is filled on return to guest from VMM from most function calls */ +struct pvmemcontrol_vmm_ret { + __u32 ret_errno; /* on error, value of errno */ + __u32 ret_code; /* pvmemcontrol internal error code, on success 0 */ + __u64 ret_value; /* return value from the function call */ + __u64 arg0; /* major version for func_code INFO */ + __u64 arg1; /* minor version for func_code INFO */ +}; + +struct pvmemcontrol_buf { + union { + struct pvmemcontrol_vmm_call call; + struct pvmemcontrol_vmm_ret ret; + }; +}; + +/* The ioctl type, documented in ioctl-number.rst */ +#define PVMEMCONTROL_IOCTL_TYPE 0xDA + +#define PVMEMCONTROL_IOCTL_VMM _IOWR(PVMEMCONTROL_IOCTL_TYPE, 0x00, struct pvmemcontrol_buf) + +/* + * Returns the host page size in ret_value. + * major version in arg0. + * minor version in arg1. + */ +#define PVMEMCONTROL_INFO 0 + +/* Pvmemcontrol calls, pvmemcontrol_vmm_return is returned */ +#define PVMEMCONTROL_DONTNEED 1 /* madvise(addr, len, MADV_DONTNEED); */ +#define PVMEMCONTROL_REMOVE 2 /* madvise(addr, len, MADV_MADV_REMOVE); */ +#define PVMEMCONTROL_FREE 3 /* madvise(addr, len, MADV_FREE); */ +#define PVMEMCONTROL_PAGEOUT 4 /* madvise(addr, len, MADV_PAGEOUT); */ +#define PVMEMCONTROL_DONTDUMP 5 /* madvise(addr, len, MADV_DONTDUMP); */ + +/* prctl(PR_SET_VMA, PR_SET_VMA_ANON_NAME, addr, len, arg) */ +#define PVMEMCONTROL_SET_VMA_ANON_NAME 6 + +#define PVMEMCONTROL_MLOCK 7 /* mlock2(addr, len, 0) */ +#define PVMEMCONTROL_MUNLOCK 8 /* munlock(addr, len) */ + +#define PVMEMCONTROL_MPROTECT_NONE 9 /* mprotect(addr, len, PROT_NONE) */ +#define PVMEMCONTROL_MPROTECT_R 10 /* mprotect(addr, len, PROT_READ) */ +#define PVMEMCONTROL_MPROTECT_W 11 /* mprotect(addr, len, PROT_WRITE) */ +/* mprotect(addr, len, PROT_READ | PROT_WRITE) */ +#define PVMEMCONTROL_MPROTECT_RW 12 + +#define PVMEMCONTROL_MERGEABLE 13 /* madvise(addr, len, MADV_MERGEABLE); */ +#define PVMEMCONTROL_UNMERGEABLE 14 /* madvise(addr, len, MADV_UNMERGEABLE); */ + +#endif /* _UAPI_PVMEMCONTROL_H */