| Message ID | 20240424214104.3248214-4-keescook@chromium.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | slab: Introduce dedicated bucket allocator | expand |
On 4/24/24 11:41 PM, Kees Cook wrote: > Dedicated caches are available for fixed size allocations via > kmem_cache_alloc(), but for dynamically sized allocations there is only > the global kmalloc API's set of buckets available. This means it isn't > possible to separate specific sets of dynamically sized allocations into > a separate collection of caches. > > This leads to a use-after-free exploitation weakness in the Linux > kernel since many heap memory spraying/grooming attacks depend on using > userspace-controllable dynamically sized allocations to collide with > fixed size allocations that end up in same cache. > > While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense > against these kinds of "type confusion" attacks, including for fixed > same-size heap objects, we can create a complementary deterministic > defense for dynamically sized allocations that are directly user > controlled. Addressing these cases is limited in scope, so isolation these > kinds of interfaces will not become an unbounded game of whack-a-mole. For > example, pass through memdup_user(), making isolation there very > effective. > > In order to isolate user-controllable sized allocations from system > allocations, introduce kmem_buckets_create(), which behaves like > kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like > kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for > where caller tracking is needed. Introduce kmem_buckets_valloc() for > cases where vmalloc callback is needed. > > Allows for confining allocations to a dedicated set of sized caches > (which have the same layout as the kmalloc caches). > > This can also be used in the future to extend codetag allocation > annotations to implement per-caller allocation cache isolation[1] even > for dynamic allocations. > > Memory allocation pinning[2] is still needed to plug the Use-After-Free > cross-allocator weakness, but that is an existing and separate issue > which is complementary to this improvement. Development continues for > that feature via the SLAB_VIRTUAL[3] series (which could also provide > guard pages -- another complementary improvement). > > Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] > Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] > Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] > Signed-off-by: Kees Cook <keescook@chromium.org> So this seems like it's all unconditional and not depending on a config option? I'd rather see one, as has been done for all similar functionality before, as not everyone will want this trade-off. Also AFAIU every new user (patches 5, 6) will add new bunch of lines to /proc/slabinfo? And when you integrate alloc tagging, do you expect every callsite will do that as well? Any idea how many there would be and what kind of memory overhead it will have in the end?
On Fri, May 24, 2024 at 03:43:33PM +0200, Vlastimil Babka wrote: > On 4/24/24 11:41 PM, Kees Cook wrote: > > Dedicated caches are available for fixed size allocations via > > kmem_cache_alloc(), but for dynamically sized allocations there is only > > the global kmalloc API's set of buckets available. This means it isn't > > possible to separate specific sets of dynamically sized allocations into > > a separate collection of caches. > > > > This leads to a use-after-free exploitation weakness in the Linux > > kernel since many heap memory spraying/grooming attacks depend on using > > userspace-controllable dynamically sized allocations to collide with > > fixed size allocations that end up in same cache. > > > > While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense > > against these kinds of "type confusion" attacks, including for fixed > > same-size heap objects, we can create a complementary deterministic > > defense for dynamically sized allocations that are directly user > > controlled. Addressing these cases is limited in scope, so isolation these > > kinds of interfaces will not become an unbounded game of whack-a-mole. For > > example, pass through memdup_user(), making isolation there very > > effective. > > > > In order to isolate user-controllable sized allocations from system > > allocations, introduce kmem_buckets_create(), which behaves like > > kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like > > kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for > > where caller tracking is needed. Introduce kmem_buckets_valloc() for > > cases where vmalloc callback is needed. > > > > Allows for confining allocations to a dedicated set of sized caches > > (which have the same layout as the kmalloc caches). > > > > This can also be used in the future to extend codetag allocation > > annotations to implement per-caller allocation cache isolation[1] even > > for dynamic allocations. > > > > Memory allocation pinning[2] is still needed to plug the Use-After-Free > > cross-allocator weakness, but that is an existing and separate issue > > which is complementary to this improvement. Development continues for > > that feature via the SLAB_VIRTUAL[3] series (which could also provide > > guard pages -- another complementary improvement). > > > > Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] > > Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] > > Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] > > Signed-off-by: Kees Cook <keescook@chromium.org> > > So this seems like it's all unconditional and not depending on a config > option? I'd rather see one, as has been done for all similar functionality > before, as not everyone will want this trade-off. Okay, sure, I can do that. Since it was changing some of the core APIs to pass in the target buckets (instead of using the global buckets), it seemed less complex to me to just make the simple changes unconditional. > Also AFAIU every new user (patches 5, 6) will add new bunch of lines to > /proc/slabinfo? And when you integrate alloc tagging, do you expect every Yes, that's true, but that's the goal: they want to live in a separate bucket collection. At present, it's only two, and arguable almost everything that the manual isolation provides should be using the mem_from_user() interface anyway. > callsite will do that as well? Any idea how many there would be and what > kind of memory overhead it will have in the end? I haven't measured the per-codetag-site overhead, but I don't expect it to have giant overhead. If the buckets are created on demand, it should be small. But one step at a time. This provides the infrastructure needed to immediately solve the low-hanging exploitable fruit and to pave the way for the per-codetag-site automation. -Kees
diff --git a/include/linux/slab.h b/include/linux/slab.h index 23b13be0ac95..1f14a65741a6 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -552,6 +552,11 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -666,6 +671,12 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmem_buckets_alloc(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_noprof(_b, _size, _flags, NUMA_NO_NODE)) + +#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ + alloc_hooks(kmalloc_node_track_caller_noprof(_b, _size, _flags, NUMA_NO_NODE, _RET_IP_)) + static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -792,6 +803,8 @@ extern void *kvmalloc_node_noprof(kmem_buckets *b, size_t size, gfp_t flags, int #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) +#define kmem_buckets_valloc(_b, _size, _flags) __kvmalloc_node(_b, _size, _flags, NUMA_NO_NODE) + static inline __alloc_size(1, 2) void *kvmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; diff --git a/mm/slab_common.c b/mm/slab_common.c index 7cb4e8fd1275..e36360e63ebd 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,74 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if ((*b)[idx]) + kmem_cache_destroy((*b)[idx]); + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -938,6 +1006,10 @@ void __init create_kmalloc_caches(void) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, 0, NULL); } /**
Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolation these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc callback is needed. Allows for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future to extend codetag allocation annotations to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness, but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] Signed-off-by: Kees Cook <keescook@chromium.org> --- Cc: Vlastimil Babka <vbabka@suse.cz> Cc: Christoph Lameter <cl@linux.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: David Rientjes <rientjes@google.com> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 13 ++++++++ mm/slab_common.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+)