| Message ID | 20231027182217.3615211-14-seanjc@google.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | KVM: guest_memfd() and per-page attributes | expand |
On Fri, Oct 27, 2023 at 11:21:55AM -0700, Sean Christopherson wrote: >From: Chao Peng <chao.p.peng@linux.intel.com> > >In confidential computing usages, whether a page is private or shared is >necessary information for KVM to perform operations like page fault >handling, page zapping etc. There are other potential use cases for >per-page memory attributes, e.g. to make memory read-only (or no-exec, >or exec-only, etc.) without having to modify memslots. > >Introduce two ioctls (advertised by KVM_CAP_MEMORY_ATTRIBUTES) to allow >userspace to operate on the per-page memory attributes. > - KVM_SET_MEMORY_ATTRIBUTES to set the per-page memory attributes to > a guest memory range. > - KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to return the KVM supported > memory attributes. This ioctl() is already removed. So, the changelog is out-of-date and needs an update. > >+ >+:Capability: KVM_CAP_MEMORY_ATTRIBUTES >+:Architectures: x86 >+:Type: vm ioctl >+:Parameters: struct kvm_memory_attributes(in) ^ add one space here? >+static bool kvm_pre_set_memory_attributes(struct kvm *kvm, >+ struct kvm_gfn_range *range) >+{ >+ /* >+ * Unconditionally add the range to the invalidation set, regardless of >+ * whether or not the arch callback actually needs to zap SPTEs. E.g. >+ * if KVM supports RWX attributes in the future and the attributes are >+ * going from R=>RW, zapping isn't strictly necessary. Unconditionally >+ * adding the range allows KVM to require that MMU invalidations add at >+ * least one range between begin() and end(), e.g. allows KVM to detect >+ * bugs where the add() is missed. Rexlaing the rule *might* be safe, ^^^^^^^^ Relaxing >@@ -4640,6 +4850,17 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) > case KVM_CAP_BINARY_STATS_FD: > case KVM_CAP_SYSTEM_EVENT_DATA: > return 1; >+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES >+ case KVM_CAP_MEMORY_ATTRIBUTES: >+ u64 attrs = kvm_supported_mem_attributes(kvm); >+ >+ r = -EFAULT; >+ if (copy_to_user(argp, &attrs, sizeof(attrs))) >+ goto out; >+ r = 0; >+ break; This cannot work, e.g., no @argp in this function and is fixed by a later commit: fcbef1e5e5d2 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory")
On Mon, Oct 30, 2023, Chao Gao wrote: > On Fri, Oct 27, 2023 at 11:21:55AM -0700, Sean Christopherson wrote: > >From: Chao Peng <chao.p.peng@linux.intel.com> > > > >In confidential computing usages, whether a page is private or shared is > >necessary information for KVM to perform operations like page fault > >handling, page zapping etc. There are other potential use cases for > >per-page memory attributes, e.g. to make memory read-only (or no-exec, > >or exec-only, etc.) without having to modify memslots. > > > >Introduce two ioctls (advertised by KVM_CAP_MEMORY_ATTRIBUTES) to allow > >userspace to operate on the per-page memory attributes. > > - KVM_SET_MEMORY_ATTRIBUTES to set the per-page memory attributes to > > a guest memory range. > > > - KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to return the KVM supported > > memory attributes. > > This ioctl() is already removed. So, the changelog is out-of-date and needs > an update. Doh, I lost track of this and the fixup for KVM_CAP_MEMORY_ATTRIBUTES below. > >+:Capability: KVM_CAP_MEMORY_ATTRIBUTES > >+:Architectures: x86 > >+:Type: vm ioctl > >+:Parameters: struct kvm_memory_attributes(in) > > ^ add one space here? Ah, yeah, that does appear to be the standard. > > > >+static bool kvm_pre_set_memory_attributes(struct kvm *kvm, > >+ struct kvm_gfn_range *range) > >+{ > >+ /* > >+ * Unconditionally add the range to the invalidation set, regardless of > >+ * whether or not the arch callback actually needs to zap SPTEs. E.g. > >+ * if KVM supports RWX attributes in the future and the attributes are > >+ * going from R=>RW, zapping isn't strictly necessary. Unconditionally > >+ * adding the range allows KVM to require that MMU invalidations add at > >+ * least one range between begin() and end(), e.g. allows KVM to detect > >+ * bugs where the add() is missed. Rexlaing the rule *might* be safe, > > ^^^^^^^^ Relaxing > > >@@ -4640,6 +4850,17 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) > > case KVM_CAP_BINARY_STATS_FD: > > case KVM_CAP_SYSTEM_EVENT_DATA: > > return 1; > >+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES > >+ case KVM_CAP_MEMORY_ATTRIBUTES: > >+ u64 attrs = kvm_supported_mem_attributes(kvm); > >+ > >+ r = -EFAULT; > >+ if (copy_to_user(argp, &attrs, sizeof(attrs))) > >+ goto out; > >+ r = 0; > >+ break; > > This cannot work, e.g., no @argp in this function and is fixed by a later commit: > > fcbef1e5e5d2 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory") I'll post a fixup patch for all of these, thanks much!
On Mon, Oct 30, 2023, Sean Christopherson wrote: > On Mon, Oct 30, 2023, Chao Gao wrote: > > On Fri, Oct 27, 2023 at 11:21:55AM -0700, Sean Christopherson wrote: > > >From: Chao Peng <chao.p.peng@linux.intel.com> > > > > > >In confidential computing usages, whether a page is private or shared is > > >necessary information for KVM to perform operations like page fault > > >handling, page zapping etc. There are other potential use cases for > > >per-page memory attributes, e.g. to make memory read-only (or no-exec, > > >or exec-only, etc.) without having to modify memslots. > > > > > >Introduce two ioctls (advertised by KVM_CAP_MEMORY_ATTRIBUTES) to allow > > >userspace to operate on the per-page memory attributes. > > > - KVM_SET_MEMORY_ATTRIBUTES to set the per-page memory attributes to > > > a guest memory range. > > > > > - KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to return the KVM supported > > > memory attributes. > > > > This ioctl() is already removed. So, the changelog is out-of-date and needs > > an update. > > Doh, I lost track of this and the fixup for KVM_CAP_MEMORY_ATTRIBUTES below. > > > >+:Capability: KVM_CAP_MEMORY_ATTRIBUTES > > >+:Architectures: x86 > > >+:Type: vm ioctl > > >+:Parameters: struct kvm_memory_attributes(in) > > > > ^ add one space here? > > Ah, yeah, that does appear to be the standard. > > > > > > >+static bool kvm_pre_set_memory_attributes(struct kvm *kvm, > > >+ struct kvm_gfn_range *range) > > >+{ > > >+ /* > > >+ * Unconditionally add the range to the invalidation set, regardless of > > >+ * whether or not the arch callback actually needs to zap SPTEs. E.g. > > >+ * if KVM supports RWX attributes in the future and the attributes are > > >+ * going from R=>RW, zapping isn't strictly necessary. Unconditionally > > >+ * adding the range allows KVM to require that MMU invalidations add at > > >+ * least one range between begin() and end(), e.g. allows KVM to detect > > >+ * bugs where the add() is missed. Rexlaing the rule *might* be safe, > > > > ^^^^^^^^ Relaxing > > > > >@@ -4640,6 +4850,17 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) > > > case KVM_CAP_BINARY_STATS_FD: > > > case KVM_CAP_SYSTEM_EVENT_DATA: > > > return 1; > > >+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES > > >+ case KVM_CAP_MEMORY_ATTRIBUTES: > > >+ u64 attrs = kvm_supported_mem_attributes(kvm); > > >+ > > >+ r = -EFAULT; > > >+ if (copy_to_user(argp, &attrs, sizeof(attrs))) > > >+ goto out; > > >+ r = 0; > > >+ break; > > > > This cannot work, e.g., no @argp in this function and is fixed by a later commit: > > > > fcbef1e5e5d2 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory") > > I'll post a fixup patch for all of these, thanks much! Heh, that was an -ENOCOFFEE. Fixup patches for a changelog goof and an ephemeral bug are going to be hard to post. Paolo, do you want to take care of all of these fixups and typos, or would you prefer that I start a v14 branch and then hand it off to you at some point?
On 2023-10-27 11:21 AM, Sean Christopherson wrote: > From: Chao Peng <chao.p.peng@linux.intel.com> > > diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h > index 89c1a991a3b8..df573229651b 100644 > --- a/include/linux/kvm_host.h > +++ b/include/linux/kvm_host.h > @@ -808,6 +809,9 @@ struct kvm { > > #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER > struct notifier_block pm_notifier; > +#endif > +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES > + struct xarray mem_attr_array; Please document how access to mem_attr_array is synchronized. If I'm reading the code correctly I think it's... /* Protected by slots_locks (for writes) and RCU (for reads) */
On Fri, 2023-10-27 at 11:21 -0700, Sean Christopherson wrote: > From: Chao Peng <chao.p.peng@linux.intel.com> > > In confidential computing usages, whether a page is private or shared is > necessary information for KVM to perform operations like page fault > handling, page zapping etc. There are other potential use cases for > per-page memory attributes, e.g. to make memory read-only (or no-exec, > or exec-only, etc.) without having to modify memslots. > > Introduce two ioctls (advertised by KVM_CAP_MEMORY_ATTRIBUTES) to allow > userspace to operate on the per-page memory attributes. > - KVM_SET_MEMORY_ATTRIBUTES to set the per-page memory attributes to > a guest memory range. > - KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to return the KVM supported > memory attributes. > > Use an xarray to store the per-page attributes internally, with a naive, > not fully optimized implementation, i.e. prioritize correctness over > performance for the initial implementation. > > Use bit 3 for the PRIVATE attribute so that KVM can use bits 0-2 for RWX > attributes/protections in the future, e.g. to give userspace fine-grained > control over read, write, and execute protections for guest memory. > > Provide arch hooks for handling attribute changes before and after common > code sets the new attributes, e.g. x86 will use the "pre" hook to zap all > relevant mappings, and the "post" hook to track whether or not hugepages > can be used to map the range. > > To simplify the implementation wrap the entire sequence with > kvm_mmu_invalidate_{begin,end}() even though the operation isn't strictly > guaranteed to be an invalidation. For the initial use case, x86 *will* > always invalidate memory, and preventing arch code from creating new > mappings while the attributes are in flux makes it much easier to reason > about the correctness of consuming attributes. > > It's possible that future usages may not require an invalidation, e.g. > if KVM ends up supporting RWX protections and userspace grants _more_ > protections, but again opt for simplicity and punt optimizations to > if/when they are needed. > > Suggested-by: Sean Christopherson <seanjc@google.com> > Link: https://lore.kernel.org/all/Y2WB48kD0J4VGynX@google.com > Cc: Fuad Tabba <tabba@google.com> > Cc: Xu Yilun <yilun.xu@intel.com> > Cc: Mickaël Salaün <mic@digikod.net> > Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com> > Co-developed-by: Sean Christopherson <seanjc@google.com> > Signed-off-by: Sean Christopherson <seanjc@google.com> > [...] > +Note, there is no "get" API. Userspace is responsible for explicitly tracking > +the state of a gfn/page as needed. > + > [...] > > +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES > +static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn) > +{ > + return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)); > +} Only call xa_to_value() when xa_load() returns !NULL? > + > +bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, > + unsigned long attrs); Seems it's not immediately clear why this function is needed in this patch, especially when you said there is no "get" API above. Add some material to changelog? > +bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm, > + struct kvm_gfn_range *range); > +bool kvm_arch_post_set_memory_attributes(struct kvm *kvm, > + struct kvm_gfn_range *range); Looks if this Kconfig is on, the above two arch hooks won't have implementation. Is it better to have two __weak empty versions here in this patch? Anyway, from the changelog it seems it's not mandatory for some ARCH to provide the above two if one wants to turn this on, i.e., the two hooks can be empty and the ARCH can just use the __weak version.
On 11/2/23 04:01, Huang, Kai wrote: > On Fri, 2023-10-27 at 11:21 -0700, Sean Christopherson wrote: >> From: Chao Peng <chao.p.peng@linux.intel.com> >> >> In confidential computing usages, whether a page is private or shared is >> necessary information for KVM to perform operations like page fault >> handling, page zapping etc. There are other potential use cases for >> per-page memory attributes, e.g. to make memory read-only (or no-exec, >> or exec-only, etc.) without having to modify memslots. >> >> Introduce two ioctls (advertised by KVM_CAP_MEMORY_ATTRIBUTES) to allow >> userspace to operate on the per-page memory attributes. >> - KVM_SET_MEMORY_ATTRIBUTES to set the per-page memory attributes to >> a guest memory range. >> - KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES to return the KVM supported >> memory attributes. >> >> Use an xarray to store the per-page attributes internally, with a naive, >> not fully optimized implementation, i.e. prioritize correctness over >> performance for the initial implementation. >> >> Use bit 3 for the PRIVATE attribute so that KVM can use bits 0-2 for RWX >> attributes/protections in the future, e.g. to give userspace fine-grained >> control over read, write, and execute protections for guest memory. >> >> Provide arch hooks for handling attribute changes before and after common >> code sets the new attributes, e.g. x86 will use the "pre" hook to zap all >> relevant mappings, and the "post" hook to track whether or not hugepages >> can be used to map the range. >> >> To simplify the implementation wrap the entire sequence with >> kvm_mmu_invalidate_{begin,end}() even though the operation isn't strictly >> guaranteed to be an invalidation. For the initial use case, x86 *will* >> always invalidate memory, and preventing arch code from creating new >> mappings while the attributes are in flux makes it much easier to reason >> about the correctness of consuming attributes. >> >> It's possible that future usages may not require an invalidation, e.g. >> if KVM ends up supporting RWX protections and userspace grants _more_ >> protections, but again opt for simplicity and punt optimizations to >> if/when they are needed. >> >> Suggested-by: Sean Christopherson <seanjc@google.com> >> Link: https://lore.kernel.org/all/Y2WB48kD0J4VGynX@google.com >> Cc: Fuad Tabba <tabba@google.com> >> Cc: Xu Yilun <yilun.xu@intel.com> >> Cc: Mickaël Salaün <mic@digikod.net> >> Signed-off-by: Chao Peng <chao.p.peng@linux.intel.com> >> Co-developed-by: Sean Christopherson <seanjc@google.com> >> Signed-off-by: Sean Christopherson <seanjc@google.com> >> > > [...] > >> +Note, there is no "get" API. Userspace is responsible for explicitly tracking >> +the state of a gfn/page as needed. >> + >> > > [...] > >> >> +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES >> +static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn) >> +{ >> + return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)); >> +} > > Only call xa_to_value() when xa_load() returns !NULL? This xarray does not store a pointer, therefore xa_load() actually returns an integer that is tagged with 1 in the low bit: static inline unsigned long xa_to_value(const void *entry) { return (unsigned long)entry >> 1; } Returning zero for an empty entry is okay, so the result of xa_load() can be used directly. >> + >> +bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, >> + unsigned long attrs); > > Seems it's not immediately clear why this function is needed in this patch, > especially when you said there is no "get" API above. Add some material to > changelog? It's used by later patches; even without a "get" API, it's a pretty fundamental functionality. >> +bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm, >> + struct kvm_gfn_range *range); >> +bool kvm_arch_post_set_memory_attributes(struct kvm *kvm, >> + struct kvm_gfn_range *range); > > Looks if this Kconfig is on, the above two arch hooks won't have implementation. > > Is it better to have two __weak empty versions here in this patch? > > Anyway, from the changelog it seems it's not mandatory for some ARCH to provide > the above two if one wants to turn this on, i.e., the two hooks can be empty and > the ARCH can just use the __weak version. I think this can be added by the first arch that needs memory attributes and also doesn't need one of these hooks. Or perhaps the x86 kvm_arch_pre_set_memory_attributes() could be made generic and thus that would be the __weak version. It's too early to tell, so it's better to leave the implementation to the architectures for now. Paolo
On Thu, 2023-11-02 at 11:32 +0100, Paolo Bonzini wrote: > > > +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES > > > +static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn) > > > +{ > > > + return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)); > > > +} > > > > Only call xa_to_value() when xa_load() returns !NULL? > > This xarray does not store a pointer, therefore xa_load() actually > returns an integer that is tagged with 1 in the low bit: > > static inline unsigned long xa_to_value(const void *entry) > { > return (unsigned long)entry >> 1; > } > > Returning zero for an empty entry is okay, so the result of xa_load() > can be used directly. Thanks for explaining. I was thinking perhaps it's better to do: void *entry = xa_load(...); return xa_is_value(entry) ? xa_to_value(entry) : 0; But "NULL (0) >> 1" is still 0, so yes we can use directly.
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 860216536810..e2252c748fd6 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6091,6 +6091,42 @@ applied. See KVM_SET_USER_MEMORY_REGION. +4.140 KVM_SET_MEMORY_ATTRIBUTES +------------------------------- + +:Capability: KVM_CAP_MEMORY_ATTRIBUTES +:Architectures: x86 +:Type: vm ioctl +:Parameters: struct kvm_memory_attributes(in) +:Returns: 0 on success, <0 on error + +KVM_SET_MEMORY_ATTRIBUTES allows userspace to set memory attributes for a range +of guest physical memory. + +:: + + struct kvm_memory_attributes { + __u64 address; + __u64 size; + __u64 attributes; + __u64 flags; + }; + + #define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) + +The address and size must be page aligned. The supported attributes can be +retrieved via ioctl(KVM_CHECK_EXTENSION) on KVM_CAP_MEMORY_ATTRIBUTES. If +executed on a VM, KVM_CAP_MEMORY_ATTRIBUTES precisely returns the attributes +supported by that VM. If executed at system scope, KVM_CAP_MEMORY_ATTRIBUTES +returns all attributes supported by KVM. The only attribute defined at this +time is KVM_MEMORY_ATTRIBUTE_PRIVATE, which marks the associated gfn as being +guest private memory. + +Note, there is no "get" API. Userspace is responsible for explicitly tracking +the state of a gfn/page as needed. + +The "flags" field is reserved for future extensions and must be '0'. + 5. The kvm_run structure ======================== diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 89c1a991a3b8..df573229651b 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -256,6 +256,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg { pte_t pte; + unsigned long attributes; }; struct kvm_gfn_range { @@ -808,6 +809,9 @@ struct kvm { #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER struct notifier_block pm_notifier; +#endif +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES + struct xarray mem_attr_array; #endif char stats_id[KVM_STATS_NAME_SIZE]; }; @@ -2340,4 +2344,18 @@ static inline void kvm_prepare_memory_fault_exit(struct kvm_vcpu *vcpu, vcpu->run->memory_fault.flags = 0; } +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn) +{ + return xa_to_value(xa_load(&kvm->mem_attr_array, gfn)); +} + +bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, + unsigned long attrs); +bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range); +bool kvm_arch_post_set_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range); +#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ + #endif diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 7ae9987b48dd..547837feaa28 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1211,6 +1211,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES 229 #define KVM_CAP_USER_MEMORY2 230 #define KVM_CAP_MEMORY_FAULT_INFO 231 +#define KVM_CAP_MEMORY_ATTRIBUTES 232 #ifdef KVM_CAP_IRQ_ROUTING @@ -2277,4 +2278,16 @@ struct kvm_s390_zpci_op { /* flags for kvm_s390_zpci_op->u.reg_aen.flags */ #define KVM_S390_ZPCIOP_REGAEN_HOST (1 << 0) +/* Available with KVM_CAP_MEMORY_ATTRIBUTES */ +#define KVM_SET_MEMORY_ATTRIBUTES _IOW(KVMIO, 0xd2, struct kvm_memory_attributes) + +struct kvm_memory_attributes { + __u64 address; + __u64 size; + __u64 attributes; + __u64 flags; +}; + +#define KVM_MEMORY_ATTRIBUTE_PRIVATE (1ULL << 3) + #endif /* __LINUX_KVM_H */ diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index ecae2914c97e..5bd7fcaf9089 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -96,3 +96,7 @@ config KVM_GENERIC_HARDWARE_ENABLING config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool + +config KVM_GENERIC_MEMORY_ATTRIBUTES + select KVM_GENERIC_MMU_NOTIFIER + bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 302ccb87b4c1..78a0b09ef2a5 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1218,6 +1218,9 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname) spin_lock_init(&kvm->mn_invalidate_lock); rcuwait_init(&kvm->mn_memslots_update_rcuwait); xa_init(&kvm->vcpu_array); +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES + xa_init(&kvm->mem_attr_array); +#endif INIT_LIST_HEAD(&kvm->gpc_list); spin_lock_init(&kvm->gpc_lock); @@ -1398,6 +1401,9 @@ static void kvm_destroy_vm(struct kvm *kvm) } cleanup_srcu_struct(&kvm->irq_srcu); cleanup_srcu_struct(&kvm->srcu); +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES + xa_destroy(&kvm->mem_attr_array); +#endif kvm_arch_free_vm(kvm); preempt_notifier_dec(); hardware_disable_all(); @@ -2396,6 +2402,210 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm, } #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */ +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES +/* + * Returns true if _all_ gfns in the range [@start, @end) have attributes + * matching @attrs. + */ +bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, + unsigned long attrs) +{ + XA_STATE(xas, &kvm->mem_attr_array, start); + unsigned long index; + bool has_attrs; + void *entry; + + rcu_read_lock(); + + if (!attrs) { + has_attrs = !xas_find(&xas, end - 1); + goto out; + } + + has_attrs = true; + for (index = start; index < end; index++) { + do { + entry = xas_next(&xas); + } while (xas_retry(&xas, entry)); + + if (xas.xa_index != index || xa_to_value(entry) != attrs) { + has_attrs = false; + break; + } + } + +out: + rcu_read_unlock(); + return has_attrs; +} + +static u64 kvm_supported_mem_attributes(struct kvm *kvm) +{ + if (!kvm) + return KVM_MEMORY_ATTRIBUTE_PRIVATE; + + return 0; +} + +static __always_inline void kvm_handle_gfn_range(struct kvm *kvm, + struct kvm_mmu_notifier_range *range) +{ + struct kvm_gfn_range gfn_range; + struct kvm_memory_slot *slot; + struct kvm_memslots *slots; + struct kvm_memslot_iter iter; + bool found_memslot = false; + bool ret = false; + int i; + + gfn_range.arg = range->arg; + gfn_range.may_block = range->may_block; + + /* + * If/when KVM supports more attributes beyond private .vs shared, this + * _could_ set only_{private,shared} appropriately if the entire target + * range already has the desired private vs. shared state (it's unclear + * if that is a net win). For now, KVM reaches this point if and only + * if the private flag is being toggled, i.e. all mappings are in play. + */ + gfn_range.only_private = false; + gfn_range.only_shared = false; + + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + slots = __kvm_memslots(kvm, i); + + kvm_for_each_memslot_in_gfn_range(&iter, slots, range->start, range->end) { + slot = iter.slot; + gfn_range.slot = slot; + + gfn_range.start = max(range->start, slot->base_gfn); + gfn_range.end = min(range->end, slot->base_gfn + slot->npages); + if (gfn_range.start >= gfn_range.end) + continue; + + if (!found_memslot) { + found_memslot = true; + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + } + + ret |= range->handler(kvm, &gfn_range); + } + } + + if (range->flush_on_ret && ret) + kvm_flush_remote_tlbs(kvm); + + if (found_memslot) + KVM_MMU_UNLOCK(kvm); +} + +static bool kvm_pre_set_memory_attributes(struct kvm *kvm, + struct kvm_gfn_range *range) +{ + /* + * Unconditionally add the range to the invalidation set, regardless of + * whether or not the arch callback actually needs to zap SPTEs. E.g. + * if KVM supports RWX attributes in the future and the attributes are + * going from R=>RW, zapping isn't strictly necessary. Unconditionally + * adding the range allows KVM to require that MMU invalidations add at + * least one range between begin() and end(), e.g. allows KVM to detect + * bugs where the add() is missed. Rexlaing the rule *might* be safe, + * but it's not obvious that allowing new mappings while the attributes + * are in flux is desirable or worth the complexity. + */ + kvm_mmu_invalidate_range_add(kvm, range->start, range->end); + + return kvm_arch_pre_set_memory_attributes(kvm, range); +} + +/* Set @attributes for the gfn range [@start, @end). */ +static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end, + unsigned long attributes) +{ + struct kvm_mmu_notifier_range pre_set_range = { + .start = start, + .end = end, + .handler = kvm_pre_set_memory_attributes, + .on_lock = kvm_mmu_invalidate_begin, + .flush_on_ret = true, + .may_block = true, + }; + struct kvm_mmu_notifier_range post_set_range = { + .start = start, + .end = end, + .arg.attributes = attributes, + .handler = kvm_arch_post_set_memory_attributes, + .on_lock = kvm_mmu_invalidate_end, + .may_block = true, + }; + unsigned long i; + void *entry; + int r = 0; + + entry = attributes ? xa_mk_value(attributes) : NULL; + + mutex_lock(&kvm->slots_lock); + + /* Nothing to do if the entire range as the desired attributes. */ + if (kvm_range_has_memory_attributes(kvm, start, end, attributes)) + goto out_unlock; + + /* + * Reserve memory ahead of time to avoid having to deal with failures + * partway through setting the new attributes. + */ + for (i = start; i < end; i++) { + r = xa_reserve(&kvm->mem_attr_array, i, GFP_KERNEL_ACCOUNT); + if (r) + goto out_unlock; + } + + kvm_handle_gfn_range(kvm, &pre_set_range); + + for (i = start; i < end; i++) { + r = xa_err(xa_store(&kvm->mem_attr_array, i, entry, + GFP_KERNEL_ACCOUNT)); + KVM_BUG_ON(r, kvm); + } + + kvm_handle_gfn_range(kvm, &post_set_range); + +out_unlock: + mutex_unlock(&kvm->slots_lock); + + return r; +} +static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, + struct kvm_memory_attributes *attrs) +{ + gfn_t start, end; + + /* flags is currently not used. */ + if (attrs->flags) + return -EINVAL; + if (attrs->attributes & ~kvm_supported_mem_attributes(kvm)) + return -EINVAL; + if (attrs->size == 0 || attrs->address + attrs->size < attrs->address) + return -EINVAL; + if (!PAGE_ALIGNED(attrs->address) || !PAGE_ALIGNED(attrs->size)) + return -EINVAL; + + start = attrs->address >> PAGE_SHIFT; + end = (attrs->address + attrs->size) >> PAGE_SHIFT; + + /* + * xarray tracks data using "unsigned long", and as a result so does + * KVM. For simplicity, supports generic attributes only on 64-bit + * architectures. + */ + BUILD_BUG_ON(sizeof(attrs->attributes) != sizeof(unsigned long)); + + return kvm_vm_set_mem_attributes(kvm, start, end, attrs->attributes); +} +#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ + struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn) { return __gfn_to_memslot(kvm_memslots(kvm), gfn); @@ -4640,6 +4850,17 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) case KVM_CAP_BINARY_STATS_FD: case KVM_CAP_SYSTEM_EVENT_DATA: return 1; +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES + case KVM_CAP_MEMORY_ATTRIBUTES: + u64 attrs = kvm_supported_mem_attributes(kvm); + + r = -EFAULT; + if (copy_to_user(argp, &attrs, sizeof(attrs))) + goto out; + r = 0; + break; + } +#endif default: break; } @@ -5022,6 +5243,18 @@ static long kvm_vm_ioctl(struct file *filp, break; } #endif /* CONFIG_HAVE_KVM_IRQ_ROUTING */ +#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES + case KVM_SET_MEMORY_ATTRIBUTES: { + struct kvm_memory_attributes attrs; + + r = -EFAULT; + if (copy_from_user(&attrs, argp, sizeof(attrs))) + goto out; + + r = kvm_vm_ioctl_set_mem_attributes(kvm, &attrs); + break; + } +#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ case KVM_CREATE_DEVICE: { struct kvm_create_device cd;