From patchwork Mon Feb 1 19:43:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8B20C433E6 for ; Mon, 1 Feb 2021 19:43:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 451DD64EA9 for ; Mon, 1 Feb 2021 19:43:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 451DD64EA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C5F2C6B0073; Mon, 1 Feb 2021 14:43:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C111C6B0074; Mon, 1 Feb 2021 14:43:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB1BA6B0078; Mon, 1 Feb 2021 14:43:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0008.hostedemail.com [216.40.44.8]) by kanga.kvack.org (Postfix) with ESMTP id 8F9086B0073 for ; Mon, 1 Feb 2021 14:43:44 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5063918000BEA for ; Mon, 1 Feb 2021 19:43:44 +0000 (UTC) X-FDA: 77770723968.01.sign78_0a15c64275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 2D39710046465 for ; Mon, 1 Feb 2021 19:43:44 +0000 (UTC) X-HE-Tag: sign78_0a15c64275c4 X-Filterd-Recvd-Size: 8123 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:43 +0000 (UTC) Received: by mail-wr1-f73.google.com with SMTP id n14so11067928wru.6 for ; Mon, 01 Feb 2021 11:43:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=xUtTTHPe7DRCgRtwSi+4scq+pNy2/ibyykEGsvZRuk0=; b=e176a67fK1UTibZXT++SH5zHy4zoNchdCjdpLCcct5knc2liGlNDF3Yamqdq+stmzY UD1Pf4fnW56aHo0VWMIdu0X7NSh9tglx3uCnpOggFaqp7pk/zA0jLS91Vw8jJ1oBwAbv ZdfBExGNllbfqfRyNbp+stdcka9VoQV4HuCvyYKhFs4zFM3kQb0iNgSNATnD3q8SQnlS bo94eGEJb9nNfzLhllE6oILm36yWPrzsRKbNhMrwH4R+PLCtzzlDsNrAtHgIeRJqSKJ7 KKt7zUyIWygctX6LyJdzJWWpJMrEjfTNbfEc77Ex501VCB2gl3GBeZo2AsAuv/+hTAsk ImoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xUtTTHPe7DRCgRtwSi+4scq+pNy2/ibyykEGsvZRuk0=; b=RcsRFbfiUMNoJy1nwx8NYS2hyCSPh945EEmN28EGoVwasuUcC6Vsr+Xw4E3YRXRiyb axax/DkjsZ9vHp5FZXHKCnPe23CPoHDlunqpAXEHn/qOCfMYqn6cIeAhiaEHCq+bxZFX LSkhl8/IkqZ9Na3L4o2hN6q4gR+UhBgb2P4r85JPcZSxSoJCRPelWqxXVhoQzuOhqAUJ xyJj7vKOML1FbAyYIEVyaBq5lucwgDM2Tg0QDo1waOr/R59tjSMj3lalu3yM9p3pg5mr A/9mDMid36CsiM30wOCnTi+axPcl3bsF6wvrBfhcVY4qx+a9OvlocMgqoW3Lxdkl+607 PTOQ== X-Gm-Message-State: AOAM532nh/Mf9qsMQtQDjP5A9giXq7eLpe7lm8xMYaBNTPqGf3pEwljF FOJBKBAnf4UJdrvgq+/ztCGQu5zZWVbEjq2j X-Google-Smtp-Source: ABdhPJw8tIfQ1xoMxW7zeobkrkFUwn4wKnbjCuC6YkGhqiO0pJ6PYdc4Nfm8BLCkJlGJR9HYDF5LqYLdtDMXC9jx X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:8109:: with SMTP id c9mr380514wmd.137.1612208622386; Mon, 01 Feb 2021 11:43:42 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:25 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 01/12] kasan, mm: don't save alloc stacks twice From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently KASAN saves allocation stacks in both kasan_slab_alloc() and kasan_kmalloc() annotations. This patch changes KASAN to save allocation stacks for slab objects from kmalloc caches in kasan_kmalloc() only, and stacks for other slab objects in kasan_slab_alloc() only. This change requires ____kasan_kmalloc() knowing whether the object belongs to a kmalloc cache. This is implemented by adding a flag field to the kasan_info structure. That flag is only set for kmalloc caches via a new kasan_cache_create_kmalloc() annotation. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- include/linux/kasan.h | 9 +++++++++ mm/kasan/common.c | 18 ++++++++++++++---- mm/slab_common.c | 1 + 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 6d8f3227c264..2d5de4092185 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -83,6 +83,7 @@ static inline void kasan_disable_current(void) {} struct kasan_cache { int alloc_meta_offset; int free_meta_offset; + bool is_kmalloc; }; #ifdef CONFIG_KASAN_HW_TAGS @@ -143,6 +144,13 @@ static __always_inline void kasan_cache_create(struct kmem_cache *cache, __kasan_cache_create(cache, size, flags); } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache); +static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + if (kasan_enabled()) + __kasan_cache_create_kmalloc(cache); +} + size_t __kasan_metadata_size(struct kmem_cache *cache); static __always_inline size_t kasan_metadata_size(struct kmem_cache *cache) { @@ -278,6 +286,7 @@ static inline void kasan_free_pages(struct page *page, unsigned int order) {} static inline void kasan_cache_create(struct kmem_cache *cache, unsigned int *size, slab_flags_t *flags) {} +static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {} static inline size_t kasan_metadata_size(struct kmem_cache *cache) { return 0; } static inline void kasan_poison_slab(struct page *page) {} static inline void kasan_unpoison_object_data(struct kmem_cache *cache, diff --git a/mm/kasan/common.c b/mm/kasan/common.c index fe852f3cfa42..374049564ea3 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -210,6 +210,11 @@ void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size, *size = optimal_size; } +void __kasan_cache_create_kmalloc(struct kmem_cache *cache) +{ + cache->kasan_info.is_kmalloc = true; +} + size_t __kasan_metadata_size(struct kmem_cache *cache) { if (!kasan_stack_collection_enabled()) @@ -394,17 +399,22 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) } } -static void set_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) +static void set_alloc_info(struct kmem_cache *cache, void *object, + gfp_t flags, bool kmalloc) { struct kasan_alloc_meta *alloc_meta; + /* Don't save alloc info for kmalloc caches in kasan_slab_alloc(). */ + if (cache->kasan_info.is_kmalloc && !kmalloc) + return; + alloc_meta = kasan_get_alloc_meta(cache, object); if (alloc_meta) kasan_set_track(&alloc_meta->alloc_track, flags); } static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool keep_tag) + size_t size, gfp_t flags, bool kmalloc) { unsigned long redzone_start; unsigned long redzone_end; @@ -423,7 +433,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_GRANULE_SIZE); redzone_end = round_up((unsigned long)object + cache->object_size, KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, keep_tag); + tag = assign_tag(cache, object, false, kmalloc); /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ kasan_unpoison(set_tag(object, tag), size); @@ -431,7 +441,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, KASAN_KMALLOC_REDZONE); if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags); + set_alloc_info(cache, (void *)object, flags, kmalloc); return set_tag(object, tag); } diff --git a/mm/slab_common.c b/mm/slab_common.c index 9aa3d2fe4c55..39d1a8ff9bb8 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -647,6 +647,7 @@ struct kmem_cache *__init create_kmalloc_cache(const char *name, panic("Out of memory when creating slab %s\n", name); create_boot_cache(s, name, size, flags, useroffset, usersize); + kasan_cache_create_kmalloc(s); list_add(&s->list, &slab_caches); s->refcount = 1; return s; From patchwork Mon Feb 1 19:43:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0309BC433DB for ; Mon, 1 Feb 2021 19:43:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 914C364DBA for ; Mon, 1 Feb 2021 19:43:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 914C364DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 338046B0078; Mon, 1 Feb 2021 14:43:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E62F6B007D; Mon, 1 Feb 2021 14:43:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13EA36B007E; Mon, 1 Feb 2021 14:43:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id E65A36B0078 for ; Mon, 1 Feb 2021 14:43:46 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A894C1EE6 for ; Mon, 1 Feb 2021 19:43:46 +0000 (UTC) X-FDA: 77770724052.09.grain24_3212e79275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 88F77180189E4 for ; Mon, 1 Feb 2021 19:43:46 +0000 (UTC) X-HE-Tag: grain24_3212e79275c4 X-Filterd-Recvd-Size: 14642 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:45 +0000 (UTC) Received: by mail-wr1-f74.google.com with SMTP id d7so11038227wri.23 for ; Mon, 01 Feb 2021 11:43:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=gyuOP94GIgUUwGD/R4wbCK1VboClPgYqAgz+f87pwxc=; b=p6grNAgNSSmz4spdKOzPcLMCvMxZx+ypM/Z0hojwKABm+roASresPc80+/eRGVuxel OpVYuKNtzemJ8RN9fIAsZa17CLHx4RtgQLbmecKjnjXdhypXuZLzDs9KVxxCgTAGYGLM hOCJBlwOv6oaUImzofs1uABzksjehzXQ9b67CqvHviePgGp+4zhjip3fK8cGMejierU1 bfUGAqtxMZ47Yfi+RT7MpDk1c6fejdydShZeUTjRJ/jxSOx7MUTCyhcFjJ9g3W9uZTlo kh/WrJVGsCAappLR6h6FBN9nIlo4HCgfniFUAg/ml+Ijnbqww5XHxQ4LGAhZsZie8cqI xG7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=gyuOP94GIgUUwGD/R4wbCK1VboClPgYqAgz+f87pwxc=; b=HWwXC4aHZlqShAfW+Q1MOFGU6WSjcybbidw9URXVw/jGfiX0sqYXurpbgz6YHYwmvh 2xam/vAkqcYJkJiZ4bCMUKG0F2/C12QMuTAzpfnwruccY1eTRu2VhCpiUbdlyK9ajI7O BaSE3qDfL0oy0deYTq41WzBPyEgShh6VMsr8rjyUTsNvOAxOgAIhXZAWzcdr7ipEygk0 dfTN5mJQ0weTdkwqU4ogovDP6dlntAXoKREKA0b4Dha+mX140FeL521t3ivJkm6NmCt2 zStlIXUrqbrLmGdHGZj8CuXprwkmbH+5IiNgwbaA9HuYB/fnYIjdDPxK5wL3ALgFXHBl kgpw== X-Gm-Message-State: AOAM532PgaPT5cgjO83nH5s1FgoB/+1xGg8BXZB1yH5qK4UuQaTmrPdl lTSVuvZnn52mgGe/7BcGn+N/jx1FmYxnFudL X-Google-Smtp-Source: ABdhPJzJ6kx5AQUwcSj5oiYP9SG+Bcwu0bDpq3j6i1QYTGx/ueFGRJy0q0ehWdKd19HNpyyvuA5K7r45uS6o7KUl X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:2d8a:: with SMTP id t132mr377860wmt.119.1612208624836; Mon, 01 Feb 2021 11:43:44 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:26 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 02/12] kasan, mm: optimize kmalloc poisoning From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For allocations from kmalloc caches, kasan_kmalloc() always follows kasan_slab_alloc(). Currenly, both of them unpoison the whole object, which is unnecessary. This patch provides separate implementations for both annotations: kasan_slab_alloc() unpoisons the whole object, and kasan_kmalloc() only poisons the redzone. For generic KASAN, the redzone start might not be aligned to KASAN_GRANULE_SIZE. Therefore, the poisoning is split in two parts: kasan_poison_last_granule() poisons the unaligned part, and then kasan_poison() poisons the rest. This patch also clarifies alignment guarantees of each of the poisoning functions and drops the unnecessary round_up() call for redzone_end. With this change, the early SLUB cache annotation needs to be changed to kasan_slab_alloc(), as kasan_kmalloc() doesn't unpoison objects now. The number of poisoned bytes for objects in this cache stays the same, as kmem_cache_node->object_size is equal to sizeof(struct kmem_cache_node). Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 93 +++++++++++++++++++++++++++++++---------------- mm/kasan/kasan.h | 43 +++++++++++++++++++++- mm/kasan/shadow.c | 28 +++++++------- mm/slub.c | 3 +- 4 files changed, 119 insertions(+), 48 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 374049564ea3..128cb330ca73 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -278,21 +278,11 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, - bool init, bool keep_tag) +static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; - /* - * 1. When an object is kmalloc()'ed, two hooks are called: - * kasan_slab_alloc() and kasan_kmalloc(). We assign the - * tag only in the first one. - * 2. We reuse the same tag for krealloc'ed objects. - */ - if (keep_tag) - return get_tag(object); - /* * If the cache neither has a constructor nor has SLAB_TYPESAFE_BY_RCU * set, assign a tag when the object is being allocated (init == false). @@ -325,7 +315,7 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, } /* Tag is ignored in set_tag() without CONFIG_KASAN_SW/HW_TAGS */ - object = set_tag(object, assign_tag(cache, object, true, false)); + object = set_tag(object, assign_tag(cache, object, true)); return (void *)object; } @@ -413,12 +403,46 @@ static void set_alloc_info(struct kmem_cache *cache, void *object, kasan_set_track(&alloc_meta->alloc_track, flags); } +void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, + void *object, gfp_t flags) +{ + u8 tag; + void *tagged_object; + + if (gfpflags_allow_blocking(flags)) + kasan_quarantine_reduce(); + + if (unlikely(object == NULL)) + return NULL; + + if (is_kfence_address(object)) + return (void *)object; + + /* + * Generate and assign random tag for tag-based modes. + * Tag is ignored in set_tag() for the generic mode. + */ + tag = assign_tag(cache, object, false); + tagged_object = set_tag(object, tag); + + /* + * Unpoison the whole object. + * For kmalloc() allocations, kasan_kmalloc() will do precise poisoning. + */ + kasan_unpoison(tagged_object, cache->object_size); + + /* Save alloc info (if possible) for non-kmalloc() allocations. */ + if (kasan_stack_collection_enabled()) + set_alloc_info(cache, (void *)object, flags, false); + + return tagged_object; +} + static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags, bool kmalloc) + size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; - u8 tag; if (gfpflags_allow_blocking(flags)) kasan_quarantine_reduce(); @@ -429,33 +453,41 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, if (is_kfence_address(kasan_reset_tag(object))) return (void *)object; + /* + * The object has already been unpoisoned by kasan_slab_alloc() for + * kmalloc() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule((void *)object, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = round_up((unsigned long)object + cache->object_size, - KASAN_GRANULE_SIZE); - tag = assign_tag(cache, object, false, kmalloc); - - /* Tag is ignored in set_tag without CONFIG_KASAN_SW/HW_TAGS */ - kasan_unpoison(set_tag(object, tag), size); + redzone_end = (unsigned long)object + cache->object_size; kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); + /* + * Save alloc info (if possible) for kmalloc() allocations. + * This also rewrites the alloc info when called from kasan_krealloc(). + */ if (kasan_stack_collection_enabled()) - set_alloc_info(cache, (void *)object, flags, kmalloc); + set_alloc_info(cache, (void *)object, flags, true); - return set_tag(object, tag); -} - -void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, - void *object, gfp_t flags) -{ - return ____kasan_kmalloc(cache, object, cache->object_size, flags, false); + /* Keep the tag that was set by kasan_slab_alloc(). */ + return (void *)object; } void * __must_check __kasan_kmalloc(struct kmem_cache *cache, const void *object, size_t size, gfp_t flags) { - return ____kasan_kmalloc(cache, object, size, flags, true); + return ____kasan_kmalloc(cache, object, size, flags); } EXPORT_SYMBOL(__kasan_kmalloc); @@ -496,8 +528,7 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else - return ____kasan_kmalloc(page->slab_cache, object, size, - flags, true); + return ____kasan_kmalloc(page->slab_cache, object, size, flags); } void __kasan_kfree_large(void *ptr, unsigned long ip) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index dd14e8870023..6a2882997f23 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -358,12 +358,51 @@ static inline bool kasan_byte_accessible(const void *addr) #else /* CONFIG_KASAN_HW_TAGS */ -void kasan_poison(const void *address, size_t size, u8 value); -void kasan_unpoison(const void *address, size_t size); +/** + * kasan_poison - mark the memory range as unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * @value - value that's written to metadata for the range + * + * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. + */ +void kasan_poison(const void *addr, size_t size, u8 value); + +/** + * kasan_unpoison - mark the memory range as accessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before + * marking the range. + * For the generic mode, the last granule of the memory range gets partially + * unpoisoned based on the @size. + */ +void kasan_unpoison(const void *addr, size_t size); + bool kasan_byte_accessible(const void *addr); #endif /* CONFIG_KASAN_HW_TAGS */ +#ifdef CONFIG_KASAN_GENERIC + +/** + * kasan_poison_last_granule - mark the last granule of the memory range as + * unaccessible + * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE + * @size - range size + * + * This function is only available for the generic mode, as it's the only mode + * that has partially poisoned memory granules. + */ +void kasan_poison_last_granule(const void *address, size_t size); + +#else /* CONFIG_KASAN_GENERIC */ + +static inline void kasan_poison_last_granule(const void *address, size_t size) { } + +#endif /* CONFIG_KASAN_GENERIC */ + /* * Exported functions for interfaces called from assembly or from generated * code. Declarations here to avoid warning about missing declarations. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1372a2fc0ca9..1ed7817e4ee6 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,10 +69,6 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -/* - * Poisons the shadow memory for 'size' bytes starting from 'addr'. - * Memory addresses should be aligned to KASAN_GRANULE_SIZE. - */ void kasan_poison(const void *address, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -83,12 +79,12 @@ void kasan_poison(const void *address, size_t size, u8 value) * addresses to this function. */ address = kasan_reset_tag(address); - size = round_up(size, KASAN_GRANULE_SIZE); /* Skip KFENCE memory if called explicitly outside of sl*b. */ if (is_kfence_address(address)) return; + size = round_up(size, KASAN_GRANULE_SIZE); shadow_start = kasan_mem_to_shadow(address); shadow_end = kasan_mem_to_shadow(address + size); @@ -96,6 +92,16 @@ void kasan_poison(const void *address, size_t size, u8 value) } EXPORT_SYMBOL(kasan_poison); +#ifdef CONFIG_KASAN_GENERIC +void kasan_poison_last_granule(const void *address, size_t size) +{ + if (size & KASAN_GRANULE_MASK) { + u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + *shadow = size & KASAN_GRANULE_MASK; + } +} +#endif + void kasan_unpoison(const void *address, size_t size) { u8 tag = get_tag(address); @@ -115,16 +121,12 @@ void kasan_unpoison(const void *address, size_t size) if (is_kfence_address(address)) return; + /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ kasan_poison(address, size, tag); - if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); - - if (IS_ENABLED(CONFIG_KASAN_SW_TAGS)) - *shadow = tag; - else /* CONFIG_KASAN_GENERIC */ - *shadow = size & KASAN_GRANULE_MASK; - } + /* Partially poison the last granule for the generic mode. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(address, size); } #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/mm/slub.c b/mm/slub.c index 176b1cb0d006..e564008c2329 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3565,8 +3565,7 @@ static void early_kmem_cache_node_alloc(int node) init_object(kmem_cache_node, n, SLUB_RED_ACTIVE); init_tracking(kmem_cache_node, n); #endif - n = kasan_kmalloc(kmem_cache_node, n, sizeof(struct kmem_cache_node), - GFP_KERNEL); + n = kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL); page->freelist = get_freepointer(kmem_cache_node, n); page->inuse = 1; page->frozen = 0; From patchwork Mon Feb 1 19:43:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B023C433E9 for ; Mon, 1 Feb 2021 19:43:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C7C6764EC7 for ; Mon, 1 Feb 2021 19:43:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C7C6764EC7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 24FA36B007E; Mon, 1 Feb 2021 14:43:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18B686B0080; Mon, 1 Feb 2021 14:43:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF2276B0081; Mon, 1 Feb 2021 14:43:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0148.hostedemail.com [216.40.44.148]) by kanga.kvack.org (Postfix) with ESMTP id D17346B007E for ; Mon, 1 Feb 2021 14:43:48 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 91F61824999B for ; Mon, 1 Feb 2021 19:43:48 +0000 (UTC) X-FDA: 77770724136.20.horn06_0012014275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 66FBF180BF332 for ; Mon, 1 Feb 2021 19:43:48 +0000 (UTC) X-HE-Tag: horn06_0012014275c4 X-Filterd-Recvd-Size: 5154 Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:47 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id a75so14096886qkg.16 for ; Mon, 01 Feb 2021 11:43:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=5os4NdBbDONsbLVxABgdrPD9x64VnOptbOzgU3pEDKs=; b=icvI25ft44tZNJLareqZm2eV6LU3rjKFoSzXua9cf+7734+1fWcCmnYf7hEl5Qbz+D X1HGWJkAzxbfnIv7aZkYjPhjP0onDk8oLebvt5EFUyH9inK3uQ5w7A/juLiNsTrOpW6D 4QB/f/PJ605/JfJndxR6OL1twXraBxmad0by++3RFa1uvStOQIe+cLBvnXzyUaTfu+2v IDNsEAuTbXwUW2bZwOGOX6UO1kpsjI9ybzJEZjTQQpoYleLdUk5xqzFbAzvVhdKy/Qr8 WEoqEGKqEd0hnV7F9350O0q+eRFF2yKHGuEA4AGGamvMh48mEenN5FlPFbNm7at+3K3H B7wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5os4NdBbDONsbLVxABgdrPD9x64VnOptbOzgU3pEDKs=; b=gPshZOUGhi3jTKrgAegXyMayrB3NJm+MU+PvkH/cSrmt2deu56rGog4NQMR6uEZQ0U vutZH/VXP7fulvZdj/S68orwjFmsuDe1meKHIIMnjQh5SjGX5Ia4cxJHCWIZtY/shpKP dol3VfXNz/g931J7CPyvXCs1jRJylpt00Ict8etkaAvFpUa7FLrx1mGhGVEgh4NUKxTC MfxeuUEy6MxZU7Cj1Y1P1WplqwGTrYGVaLrHMOTg9KYTZQi5Aa0DPXWvZokMe4K2dxpX TxFrYCBfxmXyumqs0i6+Z7wmags++nlaPndAusHjwwYGL6AwYOzzVFCJi7SPIIKN93k+ wbOA== X-Gm-Message-State: AOAM530ESxJe+UpW/lNEDGApkVZHjB0szOJF/jiTK3pasUmizSnffqIY Y3zIcf4nBmYhJ8qyBHgQ15+y01fN/22iJB4+ X-Google-Smtp-Source: ABdhPJzR+YEpFc2Ch11Rl6BTXMIoXfaJyZzPai737uCYCgWg2fJznC+a5+pVgD+X3paVHk7gPsjQcrXSotCeIBAj X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a0c:c488:: with SMTP id u8mr16899450qvi.9.1612208627196; Mon, 01 Feb 2021 11:43:47 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:27 +0100 In-Reply-To: Message-Id: <8fdbf86842f4eaf2458ecd23d0844058dbc2c7a2.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 03/12] kasan: optimize large kmalloc poisoning From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similarly to kasan_kmalloc(), kasan_kmalloc_large() doesn't need to unpoison the object as it as already unpoisoned by alloc_pages() (or by ksize() for krealloc()). This patch changes kasan_kmalloc_large() to only poison the redzone. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 128cb330ca73..a7eb553c8e91 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -494,7 +494,6 @@ EXPORT_SYMBOL(__kasan_kmalloc); void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags) { - struct page *page; unsigned long redzone_start; unsigned long redzone_end; @@ -504,12 +503,23 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, if (unlikely(ptr == NULL)) return NULL; - page = virt_to_page(ptr); + /* + * The object has already been unpoisoned by kasan_alloc_pages() for + * alloc_pages() or by ksize() for krealloc(). + */ + + /* + * The redzone has byte-level precision for the generic mode. + * Partially poison the last object granule to cover the unaligned + * part of the redzone. + */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + kasan_poison_last_granule(ptr, size); + + /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(ptr + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)ptr + page_size(page); - - kasan_unpoison(ptr, size); + redzone_end = (unsigned long)ptr + page_size(virt_to_page(ptr)); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_PAGE_REDZONE); From patchwork Mon Feb 1 19:43:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADFF2C433E0 for ; Mon, 1 Feb 2021 19:43:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4153A64EA9 for ; Mon, 1 Feb 2021 19:43:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4153A64EA9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D1ECC6B0081; Mon, 1 Feb 2021 14:43:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA7E86B0083; Mon, 1 Feb 2021 14:43:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA2BD6B0081; Mon, 1 Feb 2021 14:43:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0248.hostedemail.com [216.40.44.248]) by kanga.kvack.org (Postfix) with ESMTP id 979CD6B0081 for ; Mon, 1 Feb 2021 14:43:52 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 59C461802850F for ; Mon, 1 Feb 2021 19:43:52 +0000 (UTC) X-FDA: 77770724304.01.cap48_3d0a716275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 350B110046465 for ; Mon, 1 Feb 2021 19:43:52 +0000 (UTC) X-HE-Tag: cap48_3d0a716275c4 X-Filterd-Recvd-Size: 4537 Received: from mail-lj1-f201.google.com (mail-lj1-f201.google.com [209.85.208.201]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:51 +0000 (UTC) Received: by mail-lj1-f201.google.com with SMTP id z5so9994318ljo.6 for ; Mon, 01 Feb 2021 11:43:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=vdSQ7N299ttJfkAji9JrIXHPqKuzBqRIB40yh5wvGsM=; b=Phj+HIZS3dxn0owHV5yLJaaC0AGJkEU7TfgxvzoS86S8PoZm4Rt1g+God+bs67EJc+ oAD4CGWKbkQlIIbCUNovOV1zLvhKFi5m+ZzhOblNupaIUd5ggX6TEZOlIrDz5XWmWLta JTZs7Uw7XmdRI9FLQF1x63YHp2a9KJzeObyFVOK0Yf424coQsCFXhiGhs5WSS+Wfdr5+ DV7O2YJKktRTJmORtVpVqwjlVS56TeRw4d0zGoTpGadYHfwg5/12VFuGKUvo3egw2HjJ JyuLK5fUm/yWENE6Hh73fMc8gwZMa7zXTtIL5S8EwT3PsxdE5NtR2aqDG2mEhfgqxwDS Cr8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vdSQ7N299ttJfkAji9JrIXHPqKuzBqRIB40yh5wvGsM=; b=uQAYQ69hH0ePEda6oJsgaRFgxEXJXAcpKDsJjXTCP2qOi+n7CXevRtCQoLXjzC1O/L 58gjVNSaIl2mmvH+qkiNxYXwJoyJBWAUUVZNIjsehdXE0VKYFSq4gW4IoXM1y/UHXNTA U8hKhm31io4X7ZCVLuzWJ9/DO5tkoUytH9FLl8S8Xc5lMWu+XHpyc5o0jbDzquAh8pQz ADDfXx8UnCnw01KeEfGqt/yN+LQQChKz90kqNf8xXorNU4I1NDIN0FUBT+VDbvOUDlO4 PKF4T2YnaNih6vOgMzWvF/Z0rctjxCdjyWxlBdg/OWXUkHFA9VZRMXPjbYCNyMLCYmoM prbg== X-Gm-Message-State: AOAM532C1LP+IZoMIXRR0nSdksbAmoiH7kNQtLVU0hKHtRJxrY8wcROy Idm9nZh/6+mzX4WpRudRteehGaUptL9dsCDY X-Google-Smtp-Source: ABdhPJyTPeHvPlSg3evueEvog+d2Wbu3vFUyi7JAGczcnckPOWrvdcuYouS1E9V9e/DkjSKSFt3vDyjaDwqvcObf X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ac2:44b8:: with SMTP id c24mr9569637lfm.155.1612208629709; Mon, 01 Feb 2021 11:43:49 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:28 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 04/12] kasan: clean up setting free info in kasan_slab_free From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put kasan_stack_collection_enabled() check and kasan_set_free_info() calls next to each other. The way this was previously implemented was a minor optimization that relied of the the fact that kasan_stack_collection_enabled() is always true for generic KASAN. The confusion that this brings outweights saving a few instructions. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index a7eb553c8e91..086bb77292b6 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -350,13 +350,11 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); - if (!kasan_stack_collection_enabled()) - return false; - if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; - kasan_set_free_info(cache, object, tag); + if (kasan_stack_collection_enabled()) + kasan_set_free_info(cache, object, tag); return kasan_quarantine_put(cache, object); } From patchwork Mon Feb 1 19:43:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D769AC433DB for ; Mon, 1 Feb 2021 19:43:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 749A564EAB for ; Mon, 1 Feb 2021 19:43:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 749A564EAB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2C2416B0083; Mon, 1 Feb 2021 14:43:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 222376B0085; Mon, 1 Feb 2021 14:43:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C55E6B0087; Mon, 1 Feb 2021 14:43:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id DDBEF6B0083 for ; Mon, 1 Feb 2021 14:43:53 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9D6AC3630 for ; Mon, 1 Feb 2021 19:43:53 +0000 (UTC) X-FDA: 77770724346.17.waste18_1116516275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 7C4D41802DE0F for ; Mon, 1 Feb 2021 19:43:53 +0000 (UTC) X-HE-Tag: waste18_1116516275c4 X-Filterd-Recvd-Size: 7457 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:52 +0000 (UTC) Received: by mail-wm1-f74.google.com with SMTP id j204so186604wmj.4 for ; Mon, 01 Feb 2021 11:43:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=BaU4zsBnF6d3TSTU+6bwjMLYV3de6hEhX75uuk8b+O4=; b=nJiXwOEo+jDWjutl3bPtMddLIEbYjySSqFB9nsRMPC/OyMH1cWW9MJCeCANQqWCKmB 4mwP+1VFCx3hJfom58GEOvLp6QCaI9DthbxNDJt5E19vA2IenGYT//kFZ3tt5SKMwPII JUvkgGhGqUDAruV7bZhAZMJVREM4TVtH2aoavGSYKjlNdtFGTAYff54a63RosxywMuHm 1rEjzRp24e2Mh+Qobh7XPtIFaq29nychvE8goMeWtgJmRCYM9XxEicpGU9vrO7yzeJf3 0BwkX+2NT+5+L7FlbRszkE0nJeL7H4X8e4lzext+ZMwX8Dzov8Ow4AQrlTK8RDIsbmNp XaFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BaU4zsBnF6d3TSTU+6bwjMLYV3de6hEhX75uuk8b+O4=; b=At5XpM/+RFZ7ztXcd2+yJZT8CuTe7JVOp1B8DvuhMlWztTKxuGWb63Eojs/9W3Xw8T +uT5+omLK0KnsRXW9qWpITc7zdAKhEKf11MSgsIIBjemqZg2GPSS5hYYVWovyEueisP7 0TVBJE71Lq6mjVZY8AIKaEYPgv4GluhqeHUETZ6uBXf4O4aoX/4mVFvFp+zm2TXewwGL 3Dx98UFMJqxw3iuzrjgCyGTrYFmvxsanHmkceOlKtBNQwLxFkOXOj2XPiS7AgXf5zgOq xNEUKgWeJh+ZHb+dvff/T3t+gSh5UQKy9Op6AklteRyE3W6IbAsbBCZJNoxpkCAVHi2i WInw== X-Gm-Message-State: AOAM530lzEJw+DGbT3MaWGWu8IiUFeIphn8rGHCodXWO/vCaRggsPFtn 8fCEuYzEc5gJg1dpAk5pE5YzC/aSnvhahHW6 X-Google-Smtp-Source: ABdhPJw57hRKvoeOE3jkg8dsfKFGhvdZTC70UClQcRWBXmSodKiJDsoN3TWrUtEegspLXmVAevtVI9fb+W66gF3H X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:6a02:: with SMTP id f2mr447589wmc.36.1612208631997; Mon, 01 Feb 2021 11:43:51 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:29 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 05/12] kasan: unify large kfree checks From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Unify checks in kasan_kfree_large() and in kasan_slab_free_mempool() for large allocations as it's done for small kfree() allocations. With this change, kasan_slab_free_mempool() starts checking that the first byte of the memory that's being freed is accessible. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- include/linux/kasan.h | 16 ++++++++-------- mm/kasan/common.c | 36 ++++++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 2d5de4092185..d53ea3c047bc 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -200,6 +200,13 @@ static __always_inline bool kasan_slab_free(struct kmem_cache *s, void *object) return false; } +void __kasan_kfree_large(void *ptr, unsigned long ip); +static __always_inline void kasan_kfree_large(void *ptr) +{ + if (kasan_enabled()) + __kasan_kfree_large(ptr, _RET_IP_); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip); static __always_inline void kasan_slab_free_mempool(void *ptr) { @@ -247,13 +254,6 @@ static __always_inline void * __must_check kasan_krealloc(const void *object, return (void *)object; } -void __kasan_kfree_large(void *ptr, unsigned long ip); -static __always_inline void kasan_kfree_large(void *ptr) -{ - if (kasan_enabled()) - __kasan_kfree_large(ptr, _RET_IP_); -} - /* * Unlike kasan_check_read/write(), kasan_check_byte() is performed even for * the hardware tag-based mode that doesn't rely on compiler instrumentation. @@ -302,6 +302,7 @@ static inline bool kasan_slab_free(struct kmem_cache *s, void *object) { return false; } +static inline void kasan_kfree_large(void *ptr) {} static inline void kasan_slab_free_mempool(void *ptr) {} static inline void *kasan_slab_alloc(struct kmem_cache *s, void *object, gfp_t flags) @@ -322,7 +323,6 @@ static inline void *kasan_krealloc(const void *object, size_t new_size, { return (void *)object; } -static inline void kasan_kfree_large(void *ptr) {} static inline bool kasan_check_byte(const void *address) { return true; diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 086bb77292b6..9c64a00bbf9c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -364,6 +364,31 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } +static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +{ + if (ptr != page_address(virt_to_head_page(ptr))) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + if (!kasan_byte_accessible(ptr)) { + kasan_report_invalid_free(ptr, ip); + return true; + } + + /* + * The object will be poisoned by kasan_free_pages() or + * kasan_slab_free_mempool(). + */ + + return false; +} + +void __kasan_kfree_large(void *ptr, unsigned long ip) +{ + ____kasan_kfree_large(ptr, ip); +} + void __kasan_slab_free_mempool(void *ptr, unsigned long ip) { struct page *page; @@ -377,10 +402,8 @@ void __kasan_slab_free_mempool(void *ptr, unsigned long ip) * KMALLOC_MAX_SIZE, and kmalloc falls back onto page_alloc. */ if (unlikely(!PageSlab(page))) { - if (ptr != page_address(page)) { - kasan_report_invalid_free(ptr, ip); + if (____kasan_kfree_large(ptr, ip)) return; - } kasan_poison(ptr, page_size(page), KASAN_FREE_PAGE); } else { ____kasan_slab_free(page->slab_cache, ptr, ip, false); @@ -539,13 +562,6 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag return ____kasan_kmalloc(page->slab_cache, object, size, flags); } -void __kasan_kfree_large(void *ptr, unsigned long ip) -{ - if (ptr != page_address(virt_to_head_page(ptr))) - kasan_report_invalid_free(ptr, ip); - /* The object will be poisoned by kasan_free_pages(). */ -} - bool __kasan_check_byte(const void *address, unsigned long ip) { if (!kasan_byte_accessible(address)) { From patchwork Mon Feb 1 19:43:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6961FC433DB for ; Mon, 1 Feb 2021 19:43:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A1A964EB8 for ; Mon, 1 Feb 2021 19:43:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A1A964EB8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5F47F6B0087; Mon, 1 Feb 2021 14:43:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 555456B0088; Mon, 1 Feb 2021 14:43:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AB256B0089; Mon, 1 Feb 2021 14:43:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0082.hostedemail.com [216.40.44.82]) by kanga.kvack.org (Postfix) with ESMTP id 1898B6B0087 for ; Mon, 1 Feb 2021 14:43:56 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C2BC51EF2 for ; Mon, 1 Feb 2021 19:43:55 +0000 (UTC) X-FDA: 77770724430.18.boats23_1012bad275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 7827D100EB1D7 for ; Mon, 1 Feb 2021 19:43:55 +0000 (UTC) X-HE-Tag: boats23_1012bad275c4 X-Filterd-Recvd-Size: 8240 Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:54 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id v130so14116420qkb.14 for ; Mon, 01 Feb 2021 11:43:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=XCxo0tPucACpabOxuZtkTLtcboXhH0Xr2jZowtJcY6I=; b=nnt2QvzSfV0fa4ZZ03BZbnCp26ygACCoyTH46dpZyV9pLif0zOs6SXCEH2kcnjoFoV U1X5EXE1b/4nl5HZlwRerfFqKzt7jlwQz2d6uc95LKZ+SqbWeT8/Jp5hmPe3wZ5Q5tR+ Me7mRSARwZsr3WcDQyZ/mm/huwmItQuMt6csyGU8hTuyjthwd5A/Nn8C9a4mByWlS1VZ 1MkGWO4bG9xlswrcA618UKRyKeogViFJD3prqZJKBHRo3djA6zeLo3w47rwnsunN0gz3 flvwfX50AHe7/LCWla+sbrMmy1N/cEUeB+vO+5PGoJ4Hd85oD2y2azxexCmHYTpLxzJp /Jiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=XCxo0tPucACpabOxuZtkTLtcboXhH0Xr2jZowtJcY6I=; b=pvIJuL1SAo6drpTQleRHKdyjHtinOt6y4BYfvUItg5gjhJ2eFooXc8POFrdLEhMYk7 oRiDyDobjZeSJYQgsm6xy9hEbmGXkUKJ9g5LTIfyIANigq9CbQCtj5/9iSGucmcJ20J0 D014yVS2eMa6Go4pljdoEvZ/PAbmaxWwDoBj0CmHbcGwPVnxBs7ja6kINNG79fCJ1/Tk 2UjCaOZma9LSWLwVB0DJv75hwlHyxomVpKbQ62dEJd7S45/sPY+OH1dN01+OVpT3U+YP qz4ihEP44xVVUtHun9N0KPLR81CBZSEQOsVxKITtk61D28GaZdypCKffTnPL2dqG5RCo LbzQ== X-Gm-Message-State: AOAM531k9Lswlr7xiHTJp9ASg9Q/fFd/4zZcN4TqCsmSojm/JmuxlfkD +WroavKFkxouw3dEUFh65Z6wuYYqSsxkljPY X-Google-Smtp-Source: ABdhPJwdXbY1qM/NMUi7jaSpz4LhDzmb7rta6x+9O6xGnsVRwYvyMYrhWT+3uFTKuYlZPyvi6dklTIkdGEdweICZ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ad4:40c6:: with SMTP id x6mr16803415qvp.10.1612208634261; Mon, 01 Feb 2021 11:43:54 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:30 +0100 In-Reply-To: Message-Id: <995edb531f4f976277d7da9ca8a78a96a2ea356e.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 06/12] kasan: rework krealloc tests From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch reworks KASAN-KUnit tests for krealloc() to: 1. Check both slab and page_alloc based krealloc() implementations. 2. Allow at least one full granule to fit between old and new sizes for each KASAN mode, and check accesses to that granule accordingly. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- lib/test_kasan.c | 91 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 10 deletions(-) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index 5699e43ca01b..2bb52853f341 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -258,11 +258,14 @@ static void kmalloc_large_oob_right(struct kunit *test) kfree(ptr); } -static void kmalloc_oob_krealloc_more(struct kunit *test) +static void krealloc_more_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 19; + size_t middle; + + KUNIT_ASSERT_LT(test, size1, size2); + middle = size1 + (size2 - size1) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -270,15 +273,31 @@ static void kmalloc_oob_krealloc_more(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* All offsets up to size2 must be accessible. */ + ptr2[size1 - 1] = 'x'; + ptr2[size1] = 'x'; + ptr2[middle] = 'x'; + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + kfree(ptr2); } -static void kmalloc_oob_krealloc_less(struct kunit *test) +static void krealloc_less_oob_helper(struct kunit *test, + size_t size1, size_t size2) { char *ptr1, *ptr2; - size_t size1 = 17; - size_t size2 = 15; + size_t middle; + + KUNIT_ASSERT_LT(test, size2, size1); + middle = size2 + (size1 - size2) / 2; ptr1 = kmalloc(size1, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); @@ -286,10 +305,60 @@ static void kmalloc_oob_krealloc_less(struct kunit *test) ptr2 = krealloc(ptr1, size2, GFP_KERNEL); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2); - KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2 + OOB_TAG_OFF] = 'x'); + /* Must be accessible for all modes. */ + ptr2[size2 - 1] = 'x'; + + /* Generic mode is precise, so unaligned size2 must be inaccessible. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size2] = 'x'); + + /* For all modes first aligned offset after size2 must be inaccessible. */ + KUNIT_EXPECT_KASAN_FAIL(test, + ptr2[round_up(size2, KASAN_GRANULE_SIZE)] = 'x'); + + /* + * For all modes both middle and size1 should land in separate granules + * and thus be inaccessible. + */ + KUNIT_EXPECT_LE(test, round_up(size2, KASAN_GRANULE_SIZE), + round_down(middle, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_LE(test, round_up(middle, KASAN_GRANULE_SIZE), + round_down(size1, KASAN_GRANULE_SIZE)); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[middle] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1 - 1] = 'x'); + KUNIT_EXPECT_KASAN_FAIL(test, ptr2[size1] = 'x'); + kfree(ptr2); } +static void krealloc_more_oob(struct kunit *test) +{ + krealloc_more_oob_helper(test, 201, 235); +} + +static void krealloc_less_oob(struct kunit *test) +{ + krealloc_less_oob_helper(test, 235, 201); +} + +static void krealloc_pagealloc_more_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_more_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 201, + KMALLOC_MAX_CACHE_SIZE + 235); +} + +static void krealloc_pagealloc_less_oob(struct kunit *test) +{ + /* page_alloc fallback in only implemented for SLUB. */ + KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_SLUB); + + krealloc_less_oob_helper(test, KMALLOC_MAX_CACHE_SIZE + 235, + KMALLOC_MAX_CACHE_SIZE + 201); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -983,8 +1052,10 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(pagealloc_oob_right), KUNIT_CASE(pagealloc_uaf), KUNIT_CASE(kmalloc_large_oob_right), - KUNIT_CASE(kmalloc_oob_krealloc_more), - KUNIT_CASE(kmalloc_oob_krealloc_less), + KUNIT_CASE(krealloc_more_oob), + KUNIT_CASE(krealloc_less_oob), + KUNIT_CASE(krealloc_pagealloc_more_oob), + KUNIT_CASE(krealloc_pagealloc_less_oob), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), From patchwork Mon Feb 1 19:43:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A10E7C433DB for ; Mon, 1 Feb 2021 19:44:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4879F64EA8 for ; Mon, 1 Feb 2021 19:44:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4879F64EA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9EDB16B0089; Mon, 1 Feb 2021 14:43:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 94E5D6B008A; Mon, 1 Feb 2021 14:43:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 759A16B008C; Mon, 1 Feb 2021 14:43:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id 557B06B0089 for ; Mon, 1 Feb 2021 14:43:58 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1CED28249980 for ; Mon, 1 Feb 2021 19:43:58 +0000 (UTC) X-FDA: 77770724556.05.knee31_2513d47275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id EFDE81802BC8B for ; Mon, 1 Feb 2021 19:43:57 +0000 (UTC) X-HE-Tag: knee31_2513d47275c4 X-Filterd-Recvd-Size: 5730 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:57 +0000 (UTC) Received: by mail-wm1-f73.google.com with SMTP id z25so394520wml.0 for ; Mon, 01 Feb 2021 11:43:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=yMQgrLxhHIKq4G0a2NCbXbKyDrvusdwY4yeazjCtuyk=; b=ivESTfLfXADjygdQiFNXCX+wPXeMmmyDL7MhSMwYXQFbjxX6om64VWAa9DB0UHgjky FA0IkQ9/NQQQM1j6Es5XovdS1v0MO0KZldXf+YBmCCZn47B20vO09U3YN5ZVtPh9slfU V0695NxHTqBxob7ObPmJFRiD3DbtZI77vvKRyogyBNt+B0g1mlTTZfcvclio6T9srFgt DeT+ymo2vjail6bFhgkunaozgr3fW+qjwQ7TMJkpHXXpDU0/3qtwo9c7+nWPQ+JBZxK0 m6EaK0Hm5acOgUfe9nl3oqyOCXvObbHtXRzj+nbFuJQYf48336Ff8i+FpKXIIffMt9oQ NOjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yMQgrLxhHIKq4G0a2NCbXbKyDrvusdwY4yeazjCtuyk=; b=hZOTe/p3IA4VpLnRugB1xbFFe+JaWTudZoLDqlOh+HIj0mwek3/VcYZxgIhfmaxLyT RiVGeysi+gjAB7Eyvw9RxbMGkL9r13MhWgjghPRcrV3ca7Owd/JsGo26eIIXj8pf8lz1 0CjXJQR8utvCfTqTZ9wVCiW3CZzI1yoeCusVyuRX1IQSh40XRDtJg0n/kTYL1gqyiJJF Svw2hTi7EH0mPO9uMpiSoQny3x3Jn4/Atl9nEqOhpWadt8bMfGcNKNxlQa8m2wqYIN2T yRq4vwsNm8HCSVqyXoZfI9s6YW90OXl9iDwVeplhaG+I+YRwECrPcFyHsXL+W+yF3tEx mr0w== X-Gm-Message-State: AOAM533Br9052f6q/Bfy1+pJXiPhuKJrvERLVclVwZyIvsJzUJx98trf C42yIU+OQZ1je7Igw6rj681TG9S/7OCIXywy X-Google-Smtp-Source: ABdhPJzOa+LccFfH9kBnsOeOGuqeG8GyRvo7cSnKb8yeZh67FwLtjenT5WmmmRbIZ1nG1AOskfgvEC22yE1+sKw/ X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a1c:7711:: with SMTP id t17mr441041wmi.64.1612208636543; Mon, 01 Feb 2021 11:43:56 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:31 +0100 In-Reply-To: Message-Id: <884e37ddff31b671725f4d83106111c7dcf8fb9b.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 07/12] kasan, mm: remove krealloc side-effect From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, if krealloc() is called on a freed object with KASAN enabled, it allocates and returns a new object, but doesn't copy any memory from the old one as ksize() returns 0. This makes a caller believe that krealloc() succeeded (KASAN report is printed though). This patch adds an accessibility check into __do_krealloc(). If the check fails, krealloc() returns NULL. This check duplicates the one in ksize(); this is fixed in the following patch. This patch also adds a KASAN-KUnit test to check krealloc() behaviour when it's called on a freed object. Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- lib/test_kasan.c | 20 ++++++++++++++++++++ mm/slab_common.c | 3 +++ 2 files changed, 23 insertions(+) diff --git a/lib/test_kasan.c b/lib/test_kasan.c index 2bb52853f341..61bc894d9f7e 100644 --- a/lib/test_kasan.c +++ b/lib/test_kasan.c @@ -359,6 +359,25 @@ static void krealloc_pagealloc_less_oob(struct kunit *test) KMALLOC_MAX_CACHE_SIZE + 201); } +/* + * Check that krealloc() detects a use-after-free, returns NULL, + * and doesn't unpoison the freed object. + */ +static void krealloc_uaf(struct kunit *test) +{ + char *ptr1, *ptr2; + int size1 = 201; + int size2 = 235; + + ptr1 = kmalloc(size1, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1); + kfree(ptr1); + + KUNIT_EXPECT_KASAN_FAIL(test, ptr2 = krealloc(ptr1, size2, GFP_KERNEL)); + KUNIT_ASSERT_PTR_EQ(test, (void *)ptr2, NULL); + KUNIT_EXPECT_KASAN_FAIL(test, *(volatile char *)ptr1); +} + static void kmalloc_oob_16(struct kunit *test) { struct { @@ -1056,6 +1075,7 @@ static struct kunit_case kasan_kunit_test_cases[] = { KUNIT_CASE(krealloc_less_oob), KUNIT_CASE(krealloc_pagealloc_more_oob), KUNIT_CASE(krealloc_pagealloc_less_oob), + KUNIT_CASE(krealloc_uaf), KUNIT_CASE(kmalloc_oob_16), KUNIT_CASE(kmalloc_uaf_16), KUNIT_CASE(kmalloc_oob_in_memset), diff --git a/mm/slab_common.c b/mm/slab_common.c index 39d1a8ff9bb8..dad70239b54c 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,6 +1140,9 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; + if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) + return NULL; + ks = ksize(p); if (ks >= new_size) { From patchwork Mon Feb 1 19:43:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16E0DC433DB for ; Mon, 1 Feb 2021 19:44:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9E25C64DBA for ; Mon, 1 Feb 2021 19:44:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E25C64DBA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 903DB6B008A; Mon, 1 Feb 2021 14:44:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F36D6B008C; Mon, 1 Feb 2021 14:44:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61A916B0092; Mon, 1 Feb 2021 14:44:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id 4A01B6B008A for ; Mon, 1 Feb 2021 14:44:00 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0FA7A181AEF1A for ; Mon, 1 Feb 2021 19:44:00 +0000 (UTC) X-FDA: 77770724640.04.bears68_5006aeb275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id E3B4C8007DFB for ; Mon, 1 Feb 2021 19:43:59 +0000 (UTC) X-HE-Tag: bears68_5006aeb275c4 X-Filterd-Recvd-Size: 6534 Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:43:59 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id a75so14097237qkg.16 for ; Mon, 01 Feb 2021 11:43:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=wTS7rGEWy/lmgEwVXVBE+5vzcvue/e9GmeYKwLI3/GI=; b=NM/EWWXfIO5X0CDVwGsXDCTqh8MrojXHR19eXM6kgDQheLb3UgxElTkZafjlwvJVMh hRX5SjP8nw46ZrfNSlgGwyvk51E5mCcJjGRICuKoVrBb2YwSa9i4QDptcJtNxcTGLJrj CeY1HpjU4xlSLRVgBGfO5+GLBbY3Lp0oNfGToUpZKWmAgO6jg6PmK+eLPiDFfbniYtjj 0oFFptb75/P6D059sJVp3xp5O4JXmkbYqtDH8y1wuU6wR8rq4N02//J4aewnw8CWdPfK zE529RUGZuEtVrAuesBdVfQ/DsQqo1DTN6iIqW3q4M7STnfb18Ldteve0sFZw56OEXzt nmdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wTS7rGEWy/lmgEwVXVBE+5vzcvue/e9GmeYKwLI3/GI=; b=nRxzV5GLchrfnt6WAqjJg5aTy5jZ1zJQ+JIKalapA2bC8P1UwoOOSF1Xdz/ese/m8M FvaP6sUGyywalb3JhUJE6gFCNo5Wfi5qj6EQ68rewanIdRdnp6mL1fvd/rYE10h9N3i5 S4QCdeuS4C8SPDJBKXWrYP2FLAhBuscsCSIoRlJ4BDjtN61dMzxayV+kP26GCH2xkZUh kcTJhLTwv/roCwcGA1yrRNnOOmw4Z7eFcqEKZdgISqFN+kAis6DPpb00gOFlqY7Js+qh DfoL/yV/L+j6P9voF1Eus9U6t6yCiiUVYISfzs16H9i1HhlgWMh1YpWgH9tibB3Yx3Kx h5Yw== X-Gm-Message-State: AOAM530r9IXzapq0ur/7GVjbj+qheicO4faxM4ggse3Av/Wo5G0V9UVh Zi9vQqJCDMb3Ytu8bDw78+f0NRWv6EXVqsvM X-Google-Smtp-Source: ABdhPJzxW58GoVDg9DXe3VlIIu6Zz+FBlV301B0hIass2XkT/B7p+K717pXSiFgUHOf4CH1xC9QxDtyYCzLn2SR8 X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:1110:: with SMTP id e16mr16696575qvs.62.1612208638752; Mon, 01 Feb 2021 11:43:58 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:32 +0100 In-Reply-To: Message-Id: <431c6cfa0ac8fb2b33d7ab561a64aa84c844d1a0.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 08/12] kasan, mm: optimize krealloc poisoning From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, krealloc() always calls ksize(), which unpoisons the whole object including the redzone. This is inefficient, as kasan_krealloc() repoisons the redzone for objects that fit into the same buffer. This patch changes krealloc() instrumentation to use uninstrumented __ksize() that doesn't unpoison the memory. Instead, kasan_kreallos() is changed to unpoison the memory excluding the redzone. For objects that don't fit into the old allocation, this patch disables KASAN accessibility checks when copying memory into a new object instead of unpoisoning it. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 12 ++++++++++-- mm/slab_common.c | 20 ++++++++++++++------ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 9c64a00bbf9c..a51d6ea580b0 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -476,7 +476,7 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* * The object has already been unpoisoned by kasan_slab_alloc() for - * kmalloc() or by ksize() for krealloc(). + * kmalloc() or by kasan_krealloc() for krealloc(). */ /* @@ -526,7 +526,7 @@ void * __must_check __kasan_kmalloc_large(const void *ptr, size_t size, /* * The object has already been unpoisoned by kasan_alloc_pages() for - * alloc_pages() or by ksize() for krealloc(). + * alloc_pages() or by kasan_krealloc() for krealloc(). */ /* @@ -554,8 +554,16 @@ void * __must_check __kasan_krealloc(const void *object, size_t size, gfp_t flag if (unlikely(object == ZERO_SIZE_PTR)) return (void *)object; + /* + * Unpoison the object's data. + * Part of it might already have been unpoisoned, but it's unknown + * how big that part is. + */ + kasan_unpoison(object, size); + page = virt_to_head_page(object); + /* Piggy-back on kmalloc() instrumentation to poison the redzone. */ if (unlikely(!PageSlab(page))) return __kasan_kmalloc_large(object, size, flags); else diff --git a/mm/slab_common.c b/mm/slab_common.c index dad70239b54c..821f657d38b5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1140,19 +1140,27 @@ static __always_inline void *__do_krealloc(const void *p, size_t new_size, void *ret; size_t ks; - if (likely(!ZERO_OR_NULL_PTR(p)) && !kasan_check_byte(p)) - return NULL; - - ks = ksize(p); + /* Don't use instrumented ksize to allow precise KASAN poisoning. */ + if (likely(!ZERO_OR_NULL_PTR(p))) { + if (!kasan_check_byte(p)) + return NULL; + ks = __ksize(p); + } else + ks = 0; + /* If the object still fits, repoison it precisely. */ if (ks >= new_size) { p = kasan_krealloc((void *)p, new_size, flags); return (void *)p; } ret = kmalloc_track_caller(new_size, flags); - if (ret && p) - memcpy(ret, p, ks); + if (ret && p) { + /* Disable KASAN checks as the object's redzone is accessed. */ + kasan_disable_current(); + memcpy(ret, kasan_reset_tag(p), ks); + kasan_enable_current(); + } return ret; } From patchwork Mon Feb 1 19:43:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CDE2C433E0 for ; Mon, 1 Feb 2021 19:44:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A39264EA8 for ; Mon, 1 Feb 2021 19:44:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A39264EA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 113976B008C; Mon, 1 Feb 2021 14:44:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A2CE6B0096; Mon, 1 Feb 2021 14:44:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE4C46B0095; Mon, 1 Feb 2021 14:44:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id BA2A46B008C for ; Mon, 1 Feb 2021 14:44:02 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 80B051EE6 for ; Mon, 1 Feb 2021 19:44:02 +0000 (UTC) X-FDA: 77770724724.11.bells78_0d0eae8275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 5A803180F8B81 for ; Mon, 1 Feb 2021 19:44:02 +0000 (UTC) X-HE-Tag: bells78_0d0eae8275c4 X-Filterd-Recvd-Size: 10974 Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:44:01 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id c63so8800721qkd.1 for ; Mon, 01 Feb 2021 11:44:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=yIaCjepZTR5qWD3z/y90ARyLWJ/KolY2qLhEkXBJOWM=; b=LzK81n8jRDSKSrwC/tHWoCW1SthbpuTfYnfZFmsFHmvHJTYsV4KqUATa9ZGrPrvBBc oB529UpqdDeoRp4v/vVhBEGgfPBT+1NXWdYX5Xvn/kvC+ORem0DCxpCa3jI7hJSgzYEc v8XlwQYe5HeTYBz2VJ9kbwXxPh5dTuaEPFLq+BlD0tk6wRDHhS/uZ3D7uynGWY5820LH nWpqqdOVnH/n1NoSsn+bx/H1xMABqaI0urla3+6oBMVUxG+A9o43btOQxjxE8/PeEhVu mmEmr74grVtfW5johenPyPm1luTjFhX1AtLwj0Cydk2Rb6p+inUgV1CkAFZX/D9xmjfV 3vZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yIaCjepZTR5qWD3z/y90ARyLWJ/KolY2qLhEkXBJOWM=; b=DhWoD43YHhmk/mtnkCt3k0RruKW3ltBa4tct3Wlr9AFL9NvGT3kqN+YoIwkU3dhJsa 8/E/anQV0iG2BIvqPleFw5fPKK3OJjMnY2+9mI5+TZgtDwWH0ODnhDSRMSXmi/syUvWc QIeGjw2zslWFehhCRlRR6ROJ2u5iPDhrwpZL26f+dyB9JNxpbGhW7RQkzKPxSxrjJQ5N 8wfIDY8qOVfms/6BlwrH/KWAzbrffCrncbj/Ja6OJsl4cBetVsK1Qtx6WcpPv7J7vFY8 SUCdgXmONoiyZwB/3u/dCdQIJ9Y5kauwU2QCdWBF/HAam6xnC8vCDbK6iiN6th1eM1No 7Sww== X-Gm-Message-State: AOAM5332q81NgyW7OaUXBVe5MQS2A0wlMMHqFNiVa0aCdB6m9KocN5Nu l8VTyWFS0bViVgZOdzztEwEHcQapKr440B6T X-Google-Smtp-Source: ABdhPJw6YkwwBeDFx/5q0zc31oTDJ4pRWKYTc7OTHoCBPXE314XQpal0iqmdPiewIdDNyJd1sipvAehw/0Vpjy2b X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a0c:efd2:: with SMTP id a18mr16703061qvt.7.1612208641117; Mon, 01 Feb 2021 11:44:01 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:33 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 09/12] kasan: ensure poisoning size alignment From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A previous changes d99f6a10c161 ("kasan: don't round_up too much") attempted to simplify the code by adding a round_up(size) call into kasan_poison(). While this allows to have less round_up() calls around the code, this results in round_up() being called multiple times. This patch removes round_up() of size from kasan_poison() and ensures that all callers round_up() the size explicitly. This patch also adds WARN_ON() alignment checks for address and size to kasan_poison() and kasan_unpoison(). Signed-off-by: Andrey Konovalov Reviewed-by: Marco Elver --- mm/kasan/common.c | 9 ++++++--- mm/kasan/kasan.h | 33 ++++++++++++++++++++------------- mm/kasan/shadow.c | 37 ++++++++++++++++++++++--------------- 3 files changed, 48 insertions(+), 31 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index a51d6ea580b0..5691cca69397 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -261,7 +261,8 @@ void __kasan_unpoison_object_data(struct kmem_cache *cache, void *object) void __kasan_poison_object_data(struct kmem_cache *cache, void *object) { - kasan_poison(object, cache->object_size, KASAN_KMALLOC_REDZONE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_REDZONE); } /* @@ -348,7 +349,8 @@ static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, return true; } - kasan_poison(object, cache->object_size, KASAN_KMALLOC_FREE); + kasan_poison(object, round_up(cache->object_size, KASAN_GRANULE_SIZE), + KASAN_KMALLOC_FREE); if ((IS_ENABLED(CONFIG_KASAN_GENERIC) && !quarantine)) return false; @@ -490,7 +492,8 @@ static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, /* Poison the aligned part of the redzone. */ redzone_start = round_up((unsigned long)(object + size), KASAN_GRANULE_SIZE); - redzone_end = (unsigned long)object + cache->object_size; + redzone_end = round_up((unsigned long)(object + cache->object_size), + KASAN_GRANULE_SIZE); kasan_poison((void *)redzone_start, redzone_end - redzone_start, KASAN_KMALLOC_REDZONE); diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 6a2882997f23..2f7400a3412f 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -321,30 +321,37 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *address, size_t size, u8 value) +static inline void kasan_poison(const void *addr, size_t size, u8 value) { - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), value); + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + hw_set_mem_tag_range((void *)addr, size, value); } -static inline void kasan_unpoison(const void *address, size_t size) +static inline void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - hw_set_mem_tag_range((void *)address, - round_up(size, KASAN_GRANULE_SIZE), tag); + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) + return; + size = round_up(size, KASAN_GRANULE_SIZE); + + hw_set_mem_tag_range((void *)addr, size, tag); } static inline bool kasan_byte_accessible(const void *addr) @@ -361,7 +368,7 @@ static inline bool kasan_byte_accessible(const void *addr) /** * kasan_poison - mark the memory range as unaccessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, must be aligned to KASAN_GRANULE_SIZE * @value - value that's written to metadata for the range * * The size gets aligned to KASAN_GRANULE_SIZE before marking the range. @@ -371,7 +378,7 @@ void kasan_poison(const void *addr, size_t size, u8 value); /** * kasan_unpoison - mark the memory range as accessible * @addr - range start address, must be aligned to KASAN_GRANULE_SIZE - * @size - range size + * @size - range size, can be unaligned * * For the tag-based modes, the @size gets aligned to KASAN_GRANULE_SIZE before * marking the range. diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 1ed7817e4ee6..c97f51c557ea 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -69,7 +69,7 @@ void *memcpy(void *dest, const void *src, size_t len) return __memcpy(dest, src, len); } -void kasan_poison(const void *address, size_t size, u8 value) +void kasan_poison(const void *addr, size_t size, u8 value) { void *shadow_start, *shadow_end; @@ -78,55 +78,62 @@ void kasan_poison(const void *address, size_t size, u8 value) * some of the callers (e.g. kasan_poison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* Skip KFENCE memory if called explicitly outside of sl*b. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) return; - size = round_up(size, KASAN_GRANULE_SIZE); - shadow_start = kasan_mem_to_shadow(address); - shadow_end = kasan_mem_to_shadow(address + size); + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) + return; + if (WARN_ON(size & KASAN_GRANULE_MASK)) + return; + + shadow_start = kasan_mem_to_shadow(addr); + shadow_end = kasan_mem_to_shadow(addr + size); __memset(shadow_start, value, shadow_end - shadow_start); } EXPORT_SYMBOL(kasan_poison); #ifdef CONFIG_KASAN_GENERIC -void kasan_poison_last_granule(const void *address, size_t size) +void kasan_poison_last_granule(const void *addr, size_t size) { if (size & KASAN_GRANULE_MASK) { - u8 *shadow = (u8 *)kasan_mem_to_shadow(address + size); + u8 *shadow = (u8 *)kasan_mem_to_shadow(addr + size); *shadow = size & KASAN_GRANULE_MASK; } } #endif -void kasan_unpoison(const void *address, size_t size) +void kasan_unpoison(const void *addr, size_t size) { - u8 tag = get_tag(address); + u8 tag = get_tag(addr); /* * Perform shadow offset calculation based on untagged address, as * some of the callers (e.g. kasan_unpoison_object_data) pass tagged * addresses to this function. */ - address = kasan_reset_tag(address); + addr = kasan_reset_tag(addr); /* * Skip KFENCE memory if called explicitly outside of sl*b. Also note * that calls to ksize(), where size is not a multiple of machine-word * size, would otherwise poison the invalid portion of the word. */ - if (is_kfence_address(address)) + if (is_kfence_address(addr)) + return; + + if (WARN_ON((u64)addr & KASAN_GRANULE_MASK)) return; - /* Unpoison round_up(size, KASAN_GRANULE_SIZE) bytes. */ - kasan_poison(address, size, tag); + /* Unpoison all granules that cover the object. */ + kasan_poison(addr, round_up(size, KASAN_GRANULE_SIZE), tag); /* Partially poison the last granule for the generic mode. */ if (IS_ENABLED(CONFIG_KASAN_GENERIC)) - kasan_poison_last_granule(address, size); + kasan_poison_last_granule(addr, size); } #ifdef CONFIG_MEMORY_HOTPLUG From patchwork Mon Feb 1 19:43:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E840BC433DB for ; Mon, 1 Feb 2021 19:44:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 867C464EA8 for ; Mon, 1 Feb 2021 19:44:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 867C464EA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 582AE6B0095; Mon, 1 Feb 2021 14:44:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 492616B0096; Mon, 1 Feb 2021 14:44:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 339E66B0098; Mon, 1 Feb 2021 14:44:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id 17B376B0095 for ; Mon, 1 Feb 2021 14:44:05 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CE69E1EE6 for ; Mon, 1 Feb 2021 19:44:04 +0000 (UTC) X-FDA: 77770724808.17.noise54_2515be1275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id AD53018000BEA for ; Mon, 1 Feb 2021 19:44:04 +0000 (UTC) X-HE-Tag: noise54_2515be1275c4 X-Filterd-Recvd-Size: 10859 Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:44:04 +0000 (UTC) Received: by mail-qv1-f73.google.com with SMTP id e17so12074293qvo.9 for ; Mon, 01 Feb 2021 11:44:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=Odkzh1LG/DFkpV7FBjHfcEglIWvDLvMk2053ZF0TmS8=; b=lF7topMIZZyUAMPDwbNSQ4IokD79ZbQnP4p85aPkPWNYrYdUGnhVGkX1XlIlJlGN/v 60POwouAVmOoJZqP7l9CLS7BUvwybx/oFs2L/ne7iJ39ReFFa18AhXXpmHqQ/lI8hjPc iXC1G5mymCrA4y+k73BKjkX8gg0SLhGqD8Rtk/oVQk2XwtoF5FWSKcFvtQeRG5y1myfE D95+/cYum+MKroXLFIEhxiEAlmVrQDvpBa0/n6jWJNk7idpFTz+UA5n7JVGyofYwMfGI fnAv0FHmNT5TiBdf4sStkK01pblTRh0kbJmlaH6Ku+BQXmRXqzXsOfihkmt1xIJJigCi tv/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Odkzh1LG/DFkpV7FBjHfcEglIWvDLvMk2053ZF0TmS8=; b=mOsmRlZrNeziEumuWWcVwoe1TFx+bOx0EZUveC9Blprzc+uc7pdX1zuh5W7fLNaHQo sQFdNuZ2UWYXVc4o3+I39GKL9izv007edDlBXTj19VxJXRpf9bIAlwR8jFMSwRUWtOW+ XSQlM7A1FAye07eVrNZ7MTHinjdCAX2MsG8soOm6R08U0AwHsAjooLDmGVdJ8iKvoAHC nPGrek3ZsF7nmKjl7UDcVx1etuE7+IQEUUWSm7060dzMI1uUSB4oDeUo78vW6PzAhizs htZ6eVdVYGXD4Be6u7vrRjABVxn5BJIZpxidDqvkRziZ8nddt5acSujpUHIeDTdXEucv UfhA== X-Gm-Message-State: AOAM5310urnrBjXY30laTI8ZstCWbmBuyiUqdj9CIfxH5Aao9XQpB3A5 OFBBOSPO9M3Y3qT5RGXgr6Rl8Wh/AV/G7jXT X-Google-Smtp-Source: ABdhPJwwy203Wfi9k9SJwbsVg87njWeKNMJTs2w+lAuwP90F/9wvY20yPqJOEsyJ0pie5F8Rd3dN+mutFSzO0a2d X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:ad4:5110:: with SMTP id g16mr16935423qvp.5.1612208643527; Mon, 01 Feb 2021 11:44:03 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:34 +0100 In-Reply-To: Message-Id: <17d6bef698d193f5fe0d8baee0e232a351e23a32.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 10/12] arm64: kasan: simplify and inline MTE functions From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This change provides a simpler implementation of mte_get_mem_tag(), mte_get_random_tag(), and mte_set_mem_tag_range(). Simplifications include removing system_supports_mte() checks as these functions are onlye called from KASAN runtime that had already checked system_supports_mte(). Besides that, size and address alignment checks are removed from mte_set_mem_tag_range(), as KASAN now does those. This change also moves these functions into the asm/mte-kasan.h header and implements mte_set_mem_tag_range() via inline assembly to avoid unnecessary functions calls. Co-developed-by: Vincenzo Frascino Signed-off-by: Andrey Konovalov Reviewed-by: Catalin Marinas Signed-off-by: Vincenzo Frascino --- arch/arm64/include/asm/cache.h | 1 - arch/arm64/include/asm/kasan.h | 1 + arch/arm64/include/asm/mte-def.h | 2 + arch/arm64/include/asm/mte-kasan.h | 64 ++++++++++++++++++++++++++---- arch/arm64/include/asm/mte.h | 2 - arch/arm64/kernel/mte.c | 46 --------------------- arch/arm64/lib/mte.S | 16 -------- 7 files changed, 60 insertions(+), 72 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index 77cbbe3625f2..a074459f8f2f 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -6,7 +6,6 @@ #define __ASM_CACHE_H #include -#include #define CTR_L1IP_SHIFT 14 #define CTR_L1IP_MASK 3 diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 0aaf9044cd6a..12d5f47f7dbe 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -6,6 +6,7 @@ #include #include +#include #include #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) diff --git a/arch/arm64/include/asm/mte-def.h b/arch/arm64/include/asm/mte-def.h index 2d73a1612f09..cf241b0f0a42 100644 --- a/arch/arm64/include/asm/mte-def.h +++ b/arch/arm64/include/asm/mte-def.h @@ -11,4 +11,6 @@ #define MTE_TAG_SIZE 4 #define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT) +#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" + #endif /* __ASM_MTE_DEF_H */ diff --git a/arch/arm64/include/asm/mte-kasan.h b/arch/arm64/include/asm/mte-kasan.h index 8ad981069afb..1f090beda7e6 100644 --- a/arch/arm64/include/asm/mte-kasan.h +++ b/arch/arm64/include/asm/mte-kasan.h @@ -11,13 +11,16 @@ #include +#ifdef CONFIG_ARM64_MTE + /* - * The functions below are meant to be used only for the - * KASAN_HW_TAGS interface defined in asm/memory.h. + * These functions are meant to be only used from KASAN runtime through + * the arch_*() interface defined in asm/memory.h. + * These functions don't include system_supports_mte() checks, + * as KASAN only calls them when MTE is supported and enabled. */ -#ifdef CONFIG_ARM64_MTE -static inline u8 mte_get_ptr_tag(void *ptr) +static __always_inline u8 mte_get_ptr_tag(void *ptr) { /* Note: The format of KASAN tags is 0xF */ u8 tag = 0xF0 | (u8)(((u64)(ptr)) >> MTE_TAG_SHIFT); @@ -25,9 +28,54 @@ static inline u8 mte_get_ptr_tag(void *ptr) return tag; } -u8 mte_get_mem_tag(void *addr); -u8 mte_get_random_tag(void); -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag); +/* Get allocation tag for the address. */ +static __always_inline u8 mte_get_mem_tag(void *addr) +{ + asm(__MTE_PREAMBLE "ldg %0, [%0]" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* Generate a random tag. */ +static __always_inline u8 mte_get_random_tag(void) +{ + void *addr; + + asm(__MTE_PREAMBLE "irg %0, %0" + : "+r" (addr)); + + return mte_get_ptr_tag(addr); +} + +/* + * Assign allocation tags for a region of memory based on the pointer tag. + * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and + * size must be non-zero and MTE_GRANULE_SIZE aligned. + */ +static __always_inline void mte_set_mem_tag_range(void *addr, size_t size, u8 tag) +{ + u64 curr, end; + + if (!size) + return; + + curr = (u64)__tag_set(addr, tag); + end = curr + size; + + do { + /* + * 'asm volatile' is required to prevent the compiler to move + * the statement outside of the loop. + */ + asm volatile(__MTE_PREAMBLE "stg %0, [%0]" + : + : "r" (curr) + : "memory"); + + curr += MTE_GRANULE_SIZE; + } while (curr != end); +} void mte_enable_kernel_sync(void); void mte_enable_kernel_async(void); @@ -47,10 +95,12 @@ static inline u8 mte_get_mem_tag(void *addr) { return 0xFF; } + static inline u8 mte_get_random_tag(void) { return 0xFF; } + static inline void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) { return addr; diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 237bb2f7309d..43169b978cd3 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -8,8 +8,6 @@ #include #include -#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" - #ifndef __ASSEMBLY__ #include diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 7763ac1f2917..8b27b70e1aac 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -19,7 +19,6 @@ #include #include #include -#include #include #include @@ -88,51 +87,6 @@ int memcmp_pages(struct page *page1, struct page *page2) return ret; } -u8 mte_get_mem_tag(void *addr) -{ - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "ldg %0, [%0]" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -u8 mte_get_random_tag(void) -{ - void *addr; - - if (!system_supports_mte()) - return 0xFF; - - asm(__MTE_PREAMBLE "irg %0, %0" - : "+r" (addr)); - - return mte_get_ptr_tag(addr); -} - -void *mte_set_mem_tag_range(void *addr, size_t size, u8 tag) -{ - void *ptr = addr; - - if ((!system_supports_mte()) || (size == 0)) - return addr; - - /* Make sure that size is MTE granule aligned. */ - WARN_ON(size & (MTE_GRANULE_SIZE - 1)); - - /* Make sure that the address is MTE granule aligned. */ - WARN_ON((u64)addr & (MTE_GRANULE_SIZE - 1)); - - tag = 0xF0 | tag; - ptr = (void *)__tag_set(ptr, tag); - - mte_assign_mem_tag_range(ptr, size); - - return ptr; -} - void mte_init_tags(u64 max_tag) { static bool gcr_kernel_excl_initialized; diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 9e1a12e10053..351537c12f36 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -149,19 +149,3 @@ SYM_FUNC_START(mte_restore_page_tags) ret SYM_FUNC_END(mte_restore_page_tags) - -/* - * Assign allocation tags for a region of memory based on the pointer tag - * x0 - source pointer - * x1 - size - * - * Note: The address must be non-NULL and MTE_GRANULE_SIZE aligned and - * size must be non-zero and MTE_GRANULE_SIZE aligned. - */ -SYM_FUNC_START(mte_assign_mem_tag_range) -1: stg x0, [x0] - add x0, x0, #MTE_GRANULE_SIZE - subs x1, x1, #MTE_GRANULE_SIZE - b.gt 1b - ret -SYM_FUNC_END(mte_assign_mem_tag_range) From patchwork Mon Feb 1 19:43:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96CDBC433DB for ; Mon, 1 Feb 2021 19:44:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3725764EA8 for ; Mon, 1 Feb 2021 19:44:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3725764EA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9F1EA6B0096; Mon, 1 Feb 2021 14:44:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 927776B0099; Mon, 1 Feb 2021 14:44:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CA316B009A; Mon, 1 Feb 2021 14:44:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id 601906B0096 for ; Mon, 1 Feb 2021 14:44:07 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1F32382499A8 for ; Mon, 1 Feb 2021 19:44:07 +0000 (UTC) X-FDA: 77770724934.19.quilt30_34179ec275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id E91F31AB9E5 for ; Mon, 1 Feb 2021 19:44:06 +0000 (UTC) X-HE-Tag: quilt30_34179ec275c4 X-Filterd-Recvd-Size: 6725 Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:44:06 +0000 (UTC) Received: by mail-qv1-f73.google.com with SMTP id k20so12022813qvm.16 for ; Mon, 01 Feb 2021 11:44:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=NVeqfy8PnyvgWAdKHeCgVfqveCjCMDJegzzrq5UVB6U=; b=b1lXS5TDAM0t8/OObzI88t9G9V6LITLxK5/HlZBBUXX7GfRsMrQIJFL7iZyApowvVV +V+FY2IUUWYkNTWeXZxdzf34OrHTRKwVqQcNL6fICTVHh3TpEujGh1tnocawM4hwAb4b TtuC53IMGp9hqR2Xewo3b/F1Q4iYHl+9mIZ2/lzggJuzIJcI4NBXgM0WrryaLtGR0m2a sq8jarjd69t2LKd1B5QHWJUUseYqkrYYJ/nB6l/GCPk5F1j7V1sJb7Xohp/yNGB1c8xk Doov5OEX30RUucXGXr1esBwDjbnsH9P77ukc8FOoY5A7aAkf1kP4LapMQoBp2hVkGH5G FQ2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NVeqfy8PnyvgWAdKHeCgVfqveCjCMDJegzzrq5UVB6U=; b=Ce2DlU6FAGTqOGXJXO5zgOhiwS0MXiYT+A9scw3lkNO8EJLZQppOaD2OIu2bAvsnnm umNDbSXnt324Zu4I8nam1K0Ds85jih15OxNL6oaH0vK+lAiaVZEu4ka+5RZ4oc5JMOHr TC3W2vh1I0rh+5zR7jZnEk7Ne185se6WquB6eHBWVhqD1FrDJUwBBBgWN13sFu/ApMsP /TD0By0sgQ5C1hWONvme6k4JZCm4dhsNdXVuZdrhyEblb0vX2t0kMvJDcCUcuynXL4bN foop0fz5Q8m7MxCgsMOaPSlRIIT0Bt0SV9h9oIn8KAFStV4rTdZivVK7LqcCt1gbplMX daww== X-Gm-Message-State: AOAM5316eqEVmNM5rB0+1w3V9xBTXIV55hJnIvpQWc/OF72f50lzW/Da 8hoLHIGOTQr4JyMjxpPOPcjbxilEYh8VZB7K X-Google-Smtp-Source: ABdhPJy3LMcMO1RRr1wTMaZvcYIHJ2lGN1Rm3XmR+DCe/2CyvjEerR8l4ufrmsSLp/ooBcba4JiSkTlvY93N5sqV X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:725:: with SMTP id c5mr16904134qvz.27.1612208645795; Mon, 01 Feb 2021 11:44:05 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:35 +0100 In-Reply-To: Message-Id: <05a45017b4cb15344395650e880bbab0fe6ba3e4.1612208222.git.andreyknvl@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 11/12] kasan: always inline HW_TAGS helper functions From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mark all static functions in common.c and kasan.h that are used for hardware tag-based KASAN as __always_inline to avoid unnecessary function calls. Signed-off-by: Andrey Konovalov --- mm/kasan/common.c | 13 +++++++------ mm/kasan/kasan.h | 6 +++--- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 5691cca69397..2004ecd6e43c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -279,7 +279,8 @@ void __kasan_poison_object_data(struct kmem_cache *cache, void *object) * based on objects indexes, so that objects that are next to each other * get different tags. */ -static u8 assign_tag(struct kmem_cache *cache, const void *object, bool init) +static __always_inline u8 assign_tag(struct kmem_cache *cache, + const void *object, bool init) { if (IS_ENABLED(CONFIG_KASAN_GENERIC)) return 0xff; @@ -321,8 +322,8 @@ void * __must_check __kasan_init_slab_obj(struct kmem_cache *cache, return (void *)object; } -static bool ____kasan_slab_free(struct kmem_cache *cache, void *object, - unsigned long ip, bool quarantine) +static __always_inline bool ____kasan_slab_free(struct kmem_cache *cache, + void *object, unsigned long ip, bool quarantine) { u8 tag; void *tagged_object; @@ -366,7 +367,7 @@ bool __kasan_slab_free(struct kmem_cache *cache, void *object, unsigned long ip) return ____kasan_slab_free(cache, object, ip, true); } -static bool ____kasan_kfree_large(void *ptr, unsigned long ip) +static __always_inline bool ____kasan_kfree_large(void *ptr, unsigned long ip) { if (ptr != page_address(virt_to_head_page(ptr))) { kasan_report_invalid_free(ptr, ip); @@ -461,8 +462,8 @@ void * __must_check __kasan_slab_alloc(struct kmem_cache *cache, return tagged_object; } -static void *____kasan_kmalloc(struct kmem_cache *cache, const void *object, - size_t size, gfp_t flags) +static __always_inline void *____kasan_kmalloc(struct kmem_cache *cache, + const void *object, size_t size, gfp_t flags) { unsigned long redzone_start; unsigned long redzone_end; diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 2f7400a3412f..d5fe72747a53 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -321,7 +321,7 @@ static inline u8 kasan_random_tag(void) { return 0; } #ifdef CONFIG_KASAN_HW_TAGS -static inline void kasan_poison(const void *addr, size_t size, u8 value) +static __always_inline void kasan_poison(const void *addr, size_t size, u8 value) { addr = kasan_reset_tag(addr); @@ -337,7 +337,7 @@ static inline void kasan_poison(const void *addr, size_t size, u8 value) hw_set_mem_tag_range((void *)addr, size, value); } -static inline void kasan_unpoison(const void *addr, size_t size) +static __always_inline void kasan_unpoison(const void *addr, size_t size) { u8 tag = get_tag(addr); @@ -354,7 +354,7 @@ static inline void kasan_unpoison(const void *addr, size_t size) hw_set_mem_tag_range((void *)addr, size, tag); } -static inline bool kasan_byte_accessible(const void *addr) +static __always_inline bool kasan_byte_accessible(const void *addr) { u8 ptr_tag = get_tag(addr); u8 mem_tag = hw_get_mem_tag((void *)addr); From patchwork Mon Feb 1 19:43:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12059913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 869E7C433E0 for ; Mon, 1 Feb 2021 19:44:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 24BDB64EA8 for ; Mon, 1 Feb 2021 19:44:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 24BDB64EA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2E4A96B009A; Mon, 1 Feb 2021 14:44:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26EF46B009B; Mon, 1 Feb 2021 14:44:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09BEA6B009C; Mon, 1 Feb 2021 14:44:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0051.hostedemail.com [216.40.44.51]) by kanga.kvack.org (Postfix) with ESMTP id D83C66B009A for ; Mon, 1 Feb 2021 14:44:09 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8E2553631 for ; Mon, 1 Feb 2021 19:44:09 +0000 (UTC) X-FDA: 77770725018.14.badge42_4707056275c4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 4EA4018229837 for ; Mon, 1 Feb 2021 19:44:09 +0000 (UTC) X-HE-Tag: badge42_4707056275c4 X-Filterd-Recvd-Size: 4307 Received: from mail-qv1-f73.google.com (mail-qv1-f73.google.com [209.85.219.73]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Feb 2021 19:44:08 +0000 (UTC) Received: by mail-qv1-f73.google.com with SMTP id a12so11208683qvo.3 for ; Mon, 01 Feb 2021 11:44:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=9ElhXeWA9hrtQUSa92BHDv89rLolOBWVaEOaHObQHiY=; b=scCt97wjEnPKthR1OEsfg6XHzgS7EdRFiyeoCwu46hVwq5SF3m83dULHz0MWmmLGZl yY1Mcb/oHxqrO0CQyzz2RZvmAnQ0bHwXgancVwJ8Kf9cil1iktYwcALcUpt8omkBBHc3 kkd4KIyCpQULEzuHjoILesZuKjI3HCZjm6xFhHFB/41fKMgY+HzZ0fmHsvzGSo5PJQRd 1qzDE52vdQilfVBIZo6KxqHC32etk1l7O9n0C3A7cR60R+7EJMBESajOp8+nVQnDVT7t X0qHdKFBkQ62nf38KxTN6Kq6kGBIJL7HtiqGEajvgyY3xCf6RvAxjkhQOoK7XuBRQGSn QsVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9ElhXeWA9hrtQUSa92BHDv89rLolOBWVaEOaHObQHiY=; b=bqkJ3jkLwntEhd1UL33xYF7VF2dp9fCO/ofVnx0ywropzlVwk4zcLtfXRoTdYLJUOX 2Tus171oOW/d4xTGSE+xj4c3Hq6jP3NCHg4xDztNwwk8/faocFbA21v66UvfnzybDm4o 9GLc5778oZSX2bH3xKX7pZNvL3z6qD2q13MhOV/MTJ0Q9HbeLw0nmQw6wgXRZle/dFnh hduJImCIsdt8aXY3wU1UtKGA27JmDdfFTTTrIjO2aoWDcP9d+Yl+9tJcEF2WvX7JoC6d DRPOtt66hfucbtO/EoMQKajiVKb5yjxpqOQU0c9QQI1RSk+IxdG8/t9PlX9ThE3o3FBy NT/g== X-Gm-Message-State: AOAM530RK2g1jGKXBB1Hk1+AjtsIh1eE2UdgAwD5xKwn6XxClX1p3aL/ UFUvxktBBjArRSwRCwxYaYYo/1lZiLBCQOdT X-Google-Smtp-Source: ABdhPJyGsScV89G9MAWMKyK7rBzUxlUepSEM9x8u78kzqVlPK95BXbpBATuTGNJidQ0HrKqL6DgEfGQHE22rUIOh X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:7220:84ff:fe09:7e9d]) (user=andreyknvl job=sendgmr) by 2002:a05:6214:11ab:: with SMTP id u11mr16841497qvv.17.1612208648056; Mon, 01 Feb 2021 11:44:08 -0800 (PST) Date: Mon, 1 Feb 2021 20:43:36 +0100 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog Subject: [PATCH 12/12] arm64: kasan: export MTE symbols for KASAN tests From: Andrey Konovalov To: Catalin Marinas , Vincenzo Frascino , Dmitry Vyukov , Alexander Potapenko , Marco Elver Cc: Andrew Morton , Will Deacon , Andrey Ryabinin , Peter Collingbourne , Evgenii Stepanov , Branislav Rankov , Kevin Brodsky , kasan-dev@googlegroups.com, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Export mte_enable_kernel() and mte_set_report_once() to fix: ERROR: modpost: "mte_enable_kernel" [lib/test_kasan.ko] undefined! ERROR: modpost: "mte_set_report_once" [lib/test_kasan.ko] undefined! Signed-off-by: Andrey Konovalov Acked-by: Catalin Marinas --- arch/arm64/kernel/mte.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 8b27b70e1aac..2c91bd288ea4 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -120,6 +120,7 @@ void mte_enable_kernel_sync(void) { __mte_enable_kernel("synchronous", SCTLR_ELx_TCF_SYNC); } +EXPORT_SYMBOL(mte_enable_kernel_sync); void mte_enable_kernel_async(void) { @@ -130,6 +131,7 @@ void mte_set_report_once(bool state) { WRITE_ONCE(report_fault_once, state); } +EXPORT_SYMBOL(mte_set_report_once); bool mte_report_once(void) {