From patchwork Tue Mar 28 09:58:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1ED9AC761A6 for ; Tue, 28 Mar 2023 09:58:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D47A6B0074; Tue, 28 Mar 2023 05:58:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8369D6B0075; Tue, 28 Mar 2023 05:58:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 689126B0078; Tue, 28 Mar 2023 05:58:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5784A6B0074 for ; Tue, 28 Mar 2023 05:58:36 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 25DC5C0774 for ; Tue, 28 Mar 2023 09:58:36 +0000 (UTC) X-FDA: 80617857432.11.B260B63 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf21.hostedemail.com (Postfix) with ESMTP id 517831C0014 for ; Tue, 28 Mar 2023 09:58:34 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="GNYV/SpQ"; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oXReBzDsI5mUppZ3rGyjpCu82GIXuKJKMV0lZiD0JNk=; b=oNKgZxdMDOHd3ACbJGwIUtNgHQDuQ4uoC5d2cgeBaZUpTKOs32CIG1+45AQz1RhsUITQIm 4V6qZdWMmAKtuJchZztUpQaQQgrRjcCy+bI8QH4wPRibKP55HjBkIrkPaBQVtBTmAzDx0L miAnh3sjlR38JTl8GOHqeaJrBmY4aBM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="GNYV/SpQ"; spf=pass (imf21.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997514; a=rsa-sha256; cv=none; b=A+n0Ksw/oJtNhVCRQwVVSf3SYjRE6WYIfq0luiAQxcP9/OZ6xVZxxC9raRNWd1b6gvpmkD SV9WXUVUjBdkKHys1elkk3XwShMV8hcWwecW8n2U7DU3S7ty8Cfas9qtdTcM2WEigtJm3H MB1sGhnKEYOG5Ty4MqlvP+JzifDMNw0= Received: by mail-pg1-f176.google.com with SMTP id y19so6849491pgk.5 for ; Tue, 28 Mar 2023 02:58:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997513; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oXReBzDsI5mUppZ3rGyjpCu82GIXuKJKMV0lZiD0JNk=; b=GNYV/SpQ4PF1Rk9guHwx0aYyiVbzxvCe8LgWKY4fKwXFWvJ0gHYKR9uI9Hyt7SuIba lqZhGe+iNAZBDttHfVPfrJtzXlSY4W0BwXLoi8+mDBLpZrOaqvyR3cnJuUvQfnijfFWF DgXTjFXJbvdNpSpbmeMZZq9T6E47bY/Bfun3K0PrAfzuydbRgjCh+tcBC+yQ0FegohJ2 LDusHWD4zsYHXyzzOsDLqMZyFzXOZExMZ6+6Nz5aE4HYnhqn7ndtS3fNHvJhBx1KBdK6 QyPuMZMYaMgxyNGf7ut9mP8hd0RslzKwLhfQJ46QWZOfUXnZ+YLqil4EchVkeDWoIrO7 Af0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997513; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oXReBzDsI5mUppZ3rGyjpCu82GIXuKJKMV0lZiD0JNk=; b=qaT4P1SysDd6NY/owuAo9Yx87cZ9GTmSDPt3q/bH+GYKw3jUGlXmHgQXLMlgf9S5LJ 6Bt0TJkFIP5ZlOJhoAYpny4RrzdltHUrx8x57cmGZl2PuJE1ptUy57/RAY2n/s+EsyOf PCJK9vIQn1wU/k2wnmG5BfZZOMy+gBlVqslxOChAgP2EjTjjIAyEk16V7afBdpxh50dU oguUSe81dWZ165nBDcBjshuCn6gLdxV9YI/+O9hLmoToKCDFUPxDFKmYBsVF9pJesUq2 /vlvKnx29Dnv4RLV2/pPGDPfxRbfE8u01jJ0OE7PZixDwVNIXEqEBJET4xmSOjWTp0yj yLcw== X-Gm-Message-State: AAQBX9ffHLV7ckRaZE7H3XvPinZ6rtxUgKfwBdIilQfoJ48/GvyIZSkV gaz2j1djKY134XTo7i3iQzwGyw== X-Google-Smtp-Source: AKy350a8z+tgtFIvSjXB4Jv/VaYEvsSRl5/1SyRdomnZTLXT1oaUTne7LXl5YF+58ZBqG0o4zrBD6g== X-Received: by 2002:aa7:96f8:0:b0:600:cc40:2589 with SMTP id i24-20020aa796f8000000b00600cc402589mr3361407pfq.3.1679997512966; Tue, 28 Mar 2023 02:58:32 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:32 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 1/6] mm: kfence: simplify kfence pool initialization Date: Tue, 28 Mar 2023 17:58:02 +0800 Message-Id: <20230328095807.7014-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 517831C0014 X-Stat-Signature: d7qhqaiq7ghwqosxyybnzez9u519wub6 X-HE-Tag: 1679997514-737210 X-HE-Meta: U2FsdGVkX19+jm6DKZApb3QIOiaqfISIkUyKHine1rcuwYtxcknoTzzkRkCA31jHhENX5KblOyVETZSQ7TMKKcsb3OrmE9xXe3Dz72jxSeTmsS7fRDuAFWE4cQKxHRrWqhXE8cPZnOs6ebK8ERc22ywP9oORZ9btesY/9jOaHNEDON9ss/RQYvTSLh7fP3foX1ChO9iVXwOWFtBWOSBAPxI2ND9ffVMD3ZkFFZoJ4zGFCzEeLfepnADu92LXYo2DDwQiflwjZI0kbmf8GYVmrUuPsQz22oI7683X7PUXavK/kWVdxdR9Bpuh9ZD0sQjVdNGbhCdgQquiER2End6AUT/fZ75XfrvNdQuKh7//QGzYYgCBR4wgzIk+inRS9djd1DV2Mcpy/E1QyeGgNuGupk43Bly/HzLoEYaURC3FwmXxsWSMu+uSe1q3cC2qfazwvaI1IcOz4UfybH3sE6hDnzynQx3sLZ2LaBSSh11wvWrqZdBMt1fv+kcjuJUSSnCwt+mXlKbs/LxcpmGpytY4RI+SAi2OSLedl6tBHyEIN7iTEG7DtToZU/qekP7gstldzzqMduvT6X7FTqLDpeq+/OWIVrryNhvpga8DIOv8RU9j/zuOus7tdl/hHt5t/Fa05hIFVj2gj7LQ0PYInS3UWEG6T3HMc9dfB0kD8txh3hOAYcu0Rv+BzJmgzO3KCePBWvoMMUn6apT4TZU7OsTE/Fy5FPkKlhLqCb+wm86ukTAvFbr9hf4b56iScWANshhfJODR4P3bro+q8p6SjV77y16svsStzYb87yXCiqE1DSDzR66WVTVen7ky9wsiehdW3qeUDGsuNzKsZkJu+WFyQpl/Cnyvb9fE0BrqSsBEzEup7gxU4P0J2X5uzxJ90ga9Ok+VsVWvYWT4JJ+HoqIFWFs3tMHSs+kIJ/FIBvtSJjsu/8OD2Zge8AeY+zHmraz6APeX7Zi+1TfBQ0wubUD 0D67s8mK gblGDuXWmVL0lEkvcF5MCQewFvCt12eGlQqmzuIuJS3ElO0uFzXu3FkmxVzPMswpCyzAy7h1kxWpL8s1oxUOxYg1aHJpAJ+BdQPqGq3pWM+8ptJa9f7AaAC2pvJ5el7pnaL27z/b7ovc9eScFteUff+SIAusH+Aa9FIzGOjEdy3TARNtJDP27aqmM84DEWIlDZ3Rdysi5G3W8Nm6LMjb3jMT5SG1iWeWfdFz6muHqd47CjT2/h9ZmGyb33ldYIcDOslmmxr7zDuTaPleV5fjISHPGph6iRvxv0uMzw17n0h8MtIUsUWsbokhWtuZ+ow18sIgTBzYYxCELokA8wZFm6IjpxzWeDwoh1poWfjxrjIkxsOVoGsPBEtbT7RD5+wpTy6hY/i7c5K2DYICiNeDeGbxtd2IMrzzCXK3jLUT/Q/4wny6ShBbpUq08zNVUiIm9pN0L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are three similar loops to initialize kfence pool, we could merge all of them into one loop to simplify the code and make code more efficient. Signed-off-by: Muchun Song Reviewed-by: Marco Elver --- mm/kfence/core.c | 47 ++++++----------------------------------------- 1 file changed, 6 insertions(+), 41 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 7d01a2c76e80..de62a84d4830 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -539,35 +539,10 @@ static void rcu_guarded_free(struct rcu_head *h) static unsigned long kfence_init_pool(void) { unsigned long addr = (unsigned long)__kfence_pool; - struct page *pages; int i; if (!arch_kfence_init_pool()) return addr; - - pages = virt_to_page(__kfence_pool); - - /* - * Set up object pages: they must have PG_slab set, to avoid freeing - * these as real pages. - * - * We also want to avoid inserting kfence_free() in the kfree() - * fast-path in SLUB, and therefore need to ensure kfree() correctly - * enters __slab_free() slow-path. - */ - for (i = 0; i < KFENCE_POOL_SIZE / PAGE_SIZE; i++) { - struct slab *slab = page_slab(nth_page(pages, i)); - - if (!i || (i % 2)) - continue; - - __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; -#endif - } - /* * Protect the first 2 pages. The first page is mostly unnecessary, and * merely serves as an extended guard page. However, adding one @@ -581,8 +556,9 @@ static unsigned long kfence_init_pool(void) addr += PAGE_SIZE; } - for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++) { + for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++, addr += 2 * PAGE_SIZE) { struct kfence_metadata *meta = &kfence_metadata[i]; + struct slab *slab = page_slab(virt_to_page(addr)); /* Initialize metadata. */ INIT_LIST_HEAD(&meta->list); @@ -593,26 +569,15 @@ static unsigned long kfence_init_pool(void) /* Protect the right redzone. */ if (unlikely(!kfence_protect(addr + PAGE_SIZE))) - goto reset_slab; - - addr += 2 * PAGE_SIZE; - } - - return 0; - -reset_slab: - for (i = 0; i < KFENCE_POOL_SIZE / PAGE_SIZE; i++) { - struct slab *slab = page_slab(nth_page(pages, i)); + return addr; - if (!i || (i % 2)) - continue; + __folio_set_slab(slab_folio(slab)); #ifdef CONFIG_MEMCG - slab->memcg_data = 0; + slab->memcg_data = (unsigned long)&meta->objcg | MEMCG_DATA_OBJCGS; #endif - __folio_clear_slab(slab_folio(slab)); } - return addr; + return 0; } static bool __init kfence_init_pool_early(void) From patchwork Tue Mar 28 09:58:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA520C76195 for ; Tue, 28 Mar 2023 09:58:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 578E86B0075; Tue, 28 Mar 2023 05:58:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5023C6B0078; Tue, 28 Mar 2023 05:58:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37D8E6B007B; Tue, 28 Mar 2023 05:58:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 22D016B0075 for ; Tue, 28 Mar 2023 05:58:41 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E72E6408A5 for ; Tue, 28 Mar 2023 09:58:40 +0000 (UTC) X-FDA: 80617857600.10.AF14137 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf02.hostedemail.com (Postfix) with ESMTP id 169BD8000C for ; Tue, 28 Mar 2023 09:58:38 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=HhbQB+dg; spf=pass (imf02.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997519; a=rsa-sha256; cv=none; b=znaGGLZ3/RXKdKskdbfu29B+M/f4MkNzD4Na8GjLeu+Nw5RPZJgTW60pkuVZ2hlEcDfLpm Erwf1se1TOkmNHnBDvvqMkwObVMSqv3OXctwSKZPCzugEasuvTlMYLeXnmM41czmaaxrWo 7KOIiV+JcNW1Nh5MOvY4ZCmN1NbHFLM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=HhbQB+dg; spf=pass (imf02.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lBD4yaWYkYDbLkNhZA944C9ObF80BX09B3zGBKkIydI=; b=vJAEXbFXJjYUnwl3VfX39FeI8SfexICQ5FkBdyzp7eKCsfrNEswC7OUA3TtQn6lEVazNy3 XgcFJsabcyWEkSaDRtaufTCyy5dJs/eOAFzyIPd641gaCG+yDMRVtB4IXJx8LgGJaZdvgp FXjAkNS7xXCabBCD3J+vddV3DbqIrrk= Received: by mail-pf1-f174.google.com with SMTP id u38so7565085pfg.10 for ; Tue, 28 Mar 2023 02:58:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997517; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lBD4yaWYkYDbLkNhZA944C9ObF80BX09B3zGBKkIydI=; b=HhbQB+dgFelVX19imcjsNLuo23G94LeWSU/HGZgUiVNfjyKVW+HiToJUnagtKSthBY 5kq+NvO/ctnPo47Twm8reDS0d5f0bC6BH3c9qB+Q9Gjw3RaCZ+yyNCqRN0jzkzO4hh9Z MgZCQ7XZwY1fjVVJrlrFKYiY+51dScHVJrHlEvymXRrBLu2hClZ3UbSwxLjhj/YFxX64 vMWId0i1jE82E/qg7Sr38pLdxec55XLsUoXO4nMH1EQgjTwoGGcRX4UGDsco7eCfVB8d cLv4R1G7yh9yGkneEds787t4phoeg1nJ0GlxivunqwTSMcQ5LysGeIuwdCvjaRqL97IT maTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997517; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lBD4yaWYkYDbLkNhZA944C9ObF80BX09B3zGBKkIydI=; b=xo+h8fG84SYO2KTw/zSUzagga2H0WWyrJVHkxk6jo4LmHiOJ4zIxQN8cKKTZv9QVXU ENANQC0kYbV8pPE+wTzXPAT+KiaTwikhwXKHOtnC48fV9wjDaH6acHqDMX8iHS7gy4tR Ro9LoiiucxEXbXe6HtTszW/ONWxDe8vI6VNas/x6nBQhU9NqsuCCsVazQA9BY4pLXqjh jb/NKpRCLw/IZ7Lg6k5DZjGqJyO52iM6vzHVgnDwvZxnIkb8F6WKft9QR/7dMm7crSFn qMNvFamRxQ0XgO9lBvADTT7AfhNUZksw+s7lQgF5cAIvVl0/7OmlQU44KFR93Vtu03ge effw== X-Gm-Message-State: AAQBX9dvaCnE9C3wC9xAnCsyRCO01GbBc396Ln+KzXkdhBy8Cj8CfeHL zouQOHx9AGKibOzejJ8jODtXQg== X-Google-Smtp-Source: AKy350aBkCEiteWTRB8arAPPUAGcyuriJ2lhLAbvPCn/eGNEdVUaJTvoxyZvHgGLTc4pYAyj0IqVBw== X-Received: by 2002:a62:1dca:0:b0:627:de2e:f1a5 with SMTP id d193-20020a621dca000000b00627de2ef1a5mr13507673pfd.4.1679997517751; Tue, 28 Mar 2023 02:58:37 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:37 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 2/6] mm: kfence: check kfence pool size at building time Date: Tue, 28 Mar 2023 17:58:03 +0800 Message-Id: <20230328095807.7014-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 169BD8000C X-Rspamd-Server: rspam01 X-Stat-Signature: t9pqfuw3jo9h4wnyzjzcotfyesb394so X-HE-Tag: 1679997518-401485 X-HE-Meta: U2FsdGVkX18E6fukNfJPleolD+P/fBJ0r809bS6gpaJZ7M6q3Iz45Vb0BbswQKaN9A1dWhIpT5NBalBYOWUcvv9ir2XviwrZV3SHrelNMkMDIyTfVcTHx9VQ0jA+iREDWEhZfF/N2sYJ8k5quIdMzkYimcGtGzE6VqOcMpyxCDSWeluQ9Qe/OItD4U7cjDvZt0XgbzVM51nmocxzF7W4gqrzoLyD1Fr0GmCduGJWxYyrjXIAHiotj1d33xsTd0z3sFl6wypA8CZ6pQRjvy8azp2JvYfolEa/k4QqkSfHEqc9XntqiINf9x1zlxuU8Fu5EamkSbOAV/xqfBdqF4jiSiML/i6XNHShB7LEnT5aeV6tUHTbCb1ekEcWd2JoS0AsMs4GS1MAwDxu0bOW1GhxRYbY1O28oysfH12+kGu6prebBKYO06JZq/YQvF27fEXO1OzxTtFgZRfpd0ux7GOkf6pS5D1LzY+z+n9k9J7h4iviQZtODx7LTnhhQyVsqO4gRA4xcPkw8iKYga/W78ENvhUudaCYeBDcYwKhZ0ZitzKguZDg6JX78LUjXe59uMcynvk/VRZp642guTNoS8q/OFbzfnKBCXpz98+haX6DOE9XOP/f8QdEHusOxn51JmKhsci2Cs5XViWsFAkdFKLsRuNLW3UrirZbdMHSwqLxIh3yJJeyMSYzp1r3pN7IG10lKUj95ke2ZfR9QrPDUACq8Ja4wKCFgXtR5xG0qgjLI6Ay/hO6ZPnKGsacP2QrlAEOLvZGlp6RwkwSGwRJx5YOMyN/kfJ01PjoeMO4aWxPRSE7Ueydfm/TWIiele8ogvaseRCetXTTOcEhZNwjbbrFlTV7SDmOxh0+eTwgXM9mJlDDQb2qnJGTUeGuO3oTY8AIlxUOdT4hzm8hlwLQm2J1Hy0MC7eO2jbSJ43Q8QMh4IG0qbqw/vchQrONjY69ZAAryA+/r8oXsf3NSxeJAdu BIdbNAD+ qS0Ld4vJUN2dn4tfEj5BJB0eklg1ouKn+RrmHN2sJncNdn7VsjJjUPqPJJQVPmp7ttXzrcNCrxY7N+7EcUgCzVKW+w2THUchOYniivKxaZ90Q30AWrkFHyX5tgCAJFvbw5nu6jZV5TjzH40QWRrO0eGyZmVSC+2KCQJ46YpYOsRIrjPEuyCVQ/mMxDScyEawmxJJOKxoZPChh+8vhM9ooo3q8YSnbAhT0IHO1tlNRfAV+xmCK0WGCQUTK3Ido0ssGveyzJ+w4BiZqc221y1NIbVhji3g0uem6KAuM9LhKY00UldbkYdHfrgs9zNmYyeQjwDVz8+uC3RvDlbpdB3tBO08IFO2iILwNI/0FWQMACplrmzv7RG7c2kAvb6WEqwkPKGp8LbMhS2uy611VP5auXg6ZdUY9FVXqH69O+qAfHauSdI+NJLTTWQK0CA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Check kfence pool size at building time to expose problem ASAP. Signed-off-by: Muchun Song --- mm/kfence/core.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index de62a84d4830..6781af1dfa66 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -841,10 +841,9 @@ static int kfence_init_late(void) return -ENOMEM; __kfence_pool = page_to_virt(pages); #else - if (nr_pages > MAX_ORDER_NR_PAGES) { - pr_warn("KFENCE_NUM_OBJECTS too large for buddy allocator\n"); - return -EINVAL; - } + BUILD_BUG_ON_MSG(get_order(KFENCE_POOL_SIZE) > MAX_ORDER, + "CONFIG_KFENCE_NUM_OBJECTS is too large for buddy allocator"); + __kfence_pool = alloc_pages_exact(KFENCE_POOL_SIZE, GFP_KERNEL); if (!__kfence_pool) return -ENOMEM; From patchwork Tue Mar 28 09:58:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C49B2C76195 for ; Tue, 28 Mar 2023 09:58:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59F7B6B0078; Tue, 28 Mar 2023 05:58:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 529416B007B; Tue, 28 Mar 2023 05:58:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A3C06B007D; Tue, 28 Mar 2023 05:58:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 25FDD6B0078 for ; Tue, 28 Mar 2023 05:58:46 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DD4061A0172 for ; Tue, 28 Mar 2023 09:58:45 +0000 (UTC) X-FDA: 80617857810.10.4648C83 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf26.hostedemail.com (Postfix) with ESMTP id 110C4140008 for ; Tue, 28 Mar 2023 09:58:43 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RU+saPvg; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997524; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hUuhgY917ceHjuJWRr2UIN+zKxCzAXrecq/+Z8gmroA=; b=AgHT4o1oVKlAlCp5prGTqlID+0AVOJasG5z710yxRvlyi4HUFiqI7DkHFrUe9BW2zVM0ZU R3s635dWnSZ1VOW6KjKeznNxOfQp6qR7q4N9X3DwpFd7LFKzCrrPiJlzdYUTOIYEA3EEkT zQDbFet8CQbn3M/Ayn0tpWPENE4jAsY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RU+saPvg; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf26.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997524; a=rsa-sha256; cv=none; b=3fvU2BjSQaessgOp45zgg2jiNpP90bcQaFRgESkZZ1zSQnfHQWDFxPc03qmys6vU3X07WW 0hphCo5IhNblLluVTmp3oYH9ZiqSkXkrttAhifyNOPQOiCVn1Q9SGTtO+F6wtyiFou8j3y fevCK9gNIVio6Vft43yOBK6m+03zw5s= Received: by mail-pl1-f169.google.com with SMTP id iw3so11130600plb.6 for ; Tue, 28 Mar 2023 02:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hUuhgY917ceHjuJWRr2UIN+zKxCzAXrecq/+Z8gmroA=; b=RU+saPvgKQNlmWTq+bdeYrfwyFkXqc8uO9OS/o1Zx1EI0DNYBWRSf0QGSblvehP0sW RM+07ZqnzFzu4DjBduQxRxeI/bnGLwWGrG69+xCnQrjxWyHaTi5IbLa4s50piAEl1g8Z FlyBovcoVrO8jQLazmCKoXrZOtd8V+0q/bPp446qMiJO2zBeSdxk4Cj7Dpz9gEJmYkFC 45eHsrxKLK73f5ZeYjUihiNtWuQZSNL7GMdA5wqC0iRWUxlzED66XOPpFiqLt0C/pO7D XgLxesijy3b0waONiSKt0aj5vxTHkOjCcUzKulf750LwwtxWInWbOsnCElj5hV/LB0dL z6oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hUuhgY917ceHjuJWRr2UIN+zKxCzAXrecq/+Z8gmroA=; b=LUDIMmKwbXYkVZAIdqU+xmbghRqYcFPiurMutOCaR9wte4sDqaX2DXDOlI/1+aHTx2 mUSNfCIiKf+CAbtZWnqLttXsxRvkqCuRwN2N1XJfp9R6+Up7Jq7wYN2UVtKHNgYLMQ8A hi+/r/lO54I/oxFEgBUxHnHzHKa/IXsfDm7v4Ci2/M/ay1FRz/iwQxknc5Pwqth+/EHF ya/79Pz3Rqa4Q5R6MIYUSPjxuRSPZusCCkGgGvmziTWHemS9ewIln+nVXe5Shr9cXKfu 40DraIc15OpUjenjybYCu7UIVdmllV2E8BGVdTUh8dakOBXmxBJ0y9m7SDLDVdFE1zfN svhg== X-Gm-Message-State: AO0yUKURd0RVm9U3NWTwfqQrSm39c54N4hsqUNz8gP7Klz9gudedlcKq VaR8vf4ahqBymrPyoNEciIB3uw== X-Google-Smtp-Source: AK7set8K3WCn7qGv5XrCwVU77b7YcNrErbvROMxOijirnK/PKs0Os7YyvQj1wcB+W7SDvFzggqBDvg== X-Received: by 2002:a05:6a20:c119:b0:d4:77a6:156f with SMTP id bh25-20020a056a20c11900b000d477a6156fmr12759375pzb.53.1679997522898; Tue, 28 Mar 2023 02:58:42 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:42 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 3/6] mm: kfence: make kfence_protect_page() void Date: Tue, 28 Mar 2023 17:58:04 +0800 Message-Id: <20230328095807.7014-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 110C4140008 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: rr7yj3xbzdafjeaz5gposafhm3bzbaq9 X-HE-Tag: 1679997523-745824 X-HE-Meta: U2FsdGVkX1+Ubr6IySAWYDbUl3uQ62Dgu+TfClSiqQv6VuVFJLFGDLAwlNuUuIZIB+sVvZVu3wEyrrK4CCbVnTFZsJdYtysIA2Xyes17Wd+nFmoiBUfJ3nnhhiG0olMqme8tN2+Nap1HcKpJVEax0hiU+kAtraBzDe8mjeMS8tNANryBzq59syIxgP0RQQ4X6C2rKETpz5RCFvCKH6ezQaudd/NZsjBWxYvTmd7KCJ8s7x7xn3REmSG32BosmDI0WCuYkRY2W7WFL8XVy6d1w1hQ5nAn6sgukjN9H3qcjd1HKY8TdahcQkYC2/s/FA4CTz50o6MCdXJp4f827TX+SkJnXL2SbV4HSzbjko+A0r9xwLvcYIIj7n5dvhNe4wGs/cvxy+6VBm7BQ/EIi1UUortfYOMwyTP7urGA6TjelM2rtYUJSfBIaNWZLA2p4ZRWqcyGt8P6pIfoUuKH2D62RZWizflp6pMVcT9fAv5w+gdmT2YBu4c5ItkMhmr59oF7qOVeHovjYYjxLn5Uh5NixbbGHaueGkJJI3agaH7BHCLY/QF9elhpll5PThHWbHr7g47lqkM26x0vXVm/XZ7CpgePwzZ78nwMe0KjxD3vPYDi71XxSaL9sB4tRcFtrKqnSL9Rp+vLsUMzw4Cews9Y3Xvt9hXqJnxZe7mXplmPdAYT0rESH0+QXhhUmbSglr/F0LJQjBuBBlw6iuEHZHK7KVhFfl7LIsDKWMu791our9NrHzk9mAAa2gXldJsXeLaQ+2+fN8Cc3Gb9izz84P6bjzjBnHx9ggxk3nc5u5P7hp6hzcjcvRQkailyimQdw0996qgyX/0kNMnuuqhvsYEZu9ieK2kxNlyvTTaVvowBt1824x1najwXHUeua+sJI+fp/ng5Z/q+9UL7+rjKatFFvnblYFQQpGyB07ckox6esfqKmekOyB7QaunBJVJpDWoWL9C4thEXlizJFaQux+B eTNDx+3A N6yEVKhK/ICuYTJ5+twOneWnFuSVL5TL8zLzYNBzOfhjB2Wx0QGD9NGiAo7S2bT24fyt8SocY+RYtorstWIvMZqnkQHhgkyG1buefLW+ComJYrHcipv7XBbTs4ufr3H+wM9bM/TWpgUKiTrR2R7dVF+s0AVPkQWjHC670VdfmGOHOkPW68EspeMOk278ZTEG0Yazp34dct234mbt9fP46nLy1CrpZPxne1sxU78UpatJL2uWgmephdKo5tftnAMu6Y7Wjc0Hw+jtduR0WmkbPIb5vXxCFqtuS0TF3Tjvft99WPfLize0LvmHrJKqAQMo+YzakD3vtil1s2pX3N2uH+QFGGgYay2eSAhIiH9lFqNHawJvjdqiPSYNlFngTCnT9UbbrYydbXCorLH3FpPtMgnAcGu8tdpz5HbBhjzvy+eyCEfeZ1LcnuMkGkHsoNfK0N3oxQFgBl8C9bKliln2dnfYgKICwF2MhY9pW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The arch_kfence_init_pool() make sure kfence pool is mapped with base page size (e.g. 4KB), so the following PTE lookup in kfence_protect_page() will always succeed. Then there is no way to stop kfence_protect_page() always returning true, so make it void to simplify the code. Signed-off-by: Muchun Song --- arch/arm/include/asm/kfence.h | 4 +- arch/arm64/include/asm/kfence.h | 4 +- arch/parisc/include/asm/kfence.h | 7 +- arch/powerpc/include/asm/kfence.h | 8 +-- arch/riscv/include/asm/kfence.h | 4 +- arch/s390/include/asm/kfence.h | 3 +- arch/x86/include/asm/kfence.h | 9 +-- mm/kfence/core.c | 142 +++++++++++++++++--------------------- 8 files changed, 73 insertions(+), 108 deletions(-) diff --git a/arch/arm/include/asm/kfence.h b/arch/arm/include/asm/kfence.h index 7980d0f2271f..c30a5f8125e8 100644 --- a/arch/arm/include/asm/kfence.h +++ b/arch/arm/include/asm/kfence.h @@ -43,11 +43,9 @@ static inline bool arch_kfence_init_pool(void) return true; } -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { set_memory_valid(addr, 1, !protect); - - return true; } #endif /* __ASM_ARM_KFENCE_H */ diff --git a/arch/arm64/include/asm/kfence.h b/arch/arm64/include/asm/kfence.h index a81937fae9f6..7717c6d98b6f 100644 --- a/arch/arm64/include/asm/kfence.h +++ b/arch/arm64/include/asm/kfence.h @@ -12,11 +12,9 @@ static inline bool arch_kfence_init_pool(void) { return true; } -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { set_memory_valid(addr, 1, !protect); - - return true; } #ifdef CONFIG_KFENCE diff --git a/arch/parisc/include/asm/kfence.h b/arch/parisc/include/asm/kfence.h index 6259e5ac1fea..290792009315 100644 --- a/arch/parisc/include/asm/kfence.h +++ b/arch/parisc/include/asm/kfence.h @@ -19,13 +19,10 @@ static inline bool arch_kfence_init_pool(void) } /* Protect the given page and flush TLB. */ -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { pte_t *pte = virt_to_kpte(addr); - if (WARN_ON(!pte)) - return false; - /* * We need to avoid IPIs, as we may get KFENCE allocations or faults * with interrupts disabled. @@ -37,8 +34,6 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); - - return true; } #endif /* _ASM_PARISC_KFENCE_H */ diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index 6fd2b4d486c5..9d8502a7d0a4 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -21,16 +21,14 @@ static inline bool arch_kfence_init_pool(void) } #ifdef CONFIG_PPC64 -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { struct page *page = virt_to_page(addr); __kernel_map_pages(page, 1, !protect); - - return true; } #else -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { pte_t *kpte = virt_to_kpte(addr); @@ -40,8 +38,6 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) } else { pte_update(&init_mm, addr, kpte, 0, _PAGE_PRESENT, 0); } - - return true; } #endif diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index d887a54042aa..1299f47170b5 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -46,7 +46,7 @@ static inline bool arch_kfence_init_pool(void) return true; } -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { pte_t *pte = virt_to_kpte(addr); @@ -56,8 +56,6 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) set_pte(pte, __pte(pte_val(*pte) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); - - return true; } #endif /* _ASM_RISCV_KFENCE_H */ diff --git a/arch/s390/include/asm/kfence.h b/arch/s390/include/asm/kfence.h index d55ba878378b..6d7b3632d79c 100644 --- a/arch/s390/include/asm/kfence.h +++ b/arch/s390/include/asm/kfence.h @@ -33,10 +33,9 @@ static __always_inline void kfence_split_mapping(void) #endif } -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { __kernel_map_pages(virt_to_page(addr), 1, !protect); - return true; } #endif /* _ASM_S390_KFENCE_H */ diff --git a/arch/x86/include/asm/kfence.h b/arch/x86/include/asm/kfence.h index ff5c7134a37a..6ffd4a078a71 100644 --- a/arch/x86/include/asm/kfence.h +++ b/arch/x86/include/asm/kfence.h @@ -38,13 +38,9 @@ static inline bool arch_kfence_init_pool(void) } /* Protect the given page and flush TLB. */ -static inline bool kfence_protect_page(unsigned long addr, bool protect) +static inline void kfence_protect_page(unsigned long addr, bool protect) { - unsigned int level; - pte_t *pte = lookup_address(addr, &level); - - if (WARN_ON(!pte || level != PG_LEVEL_4K)) - return false; + pte_t *pte = virt_to_kpte(addr); /* * We need to avoid IPIs, as we may get KFENCE allocations or faults @@ -65,7 +61,6 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) preempt_disable(); flush_tlb_one_kernel(addr); preempt_enable(); - return true; } #endif /* !MODULE */ diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 6781af1dfa66..5726bf2ae13c 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -229,14 +229,14 @@ static bool alloc_covered_contains(u32 alloc_stack_hash) return true; } -static bool kfence_protect(unsigned long addr) +static inline void kfence_protect(unsigned long addr) { - return !KFENCE_WARN_ON(!kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), true)); + kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), true); } -static bool kfence_unprotect(unsigned long addr) +static inline void kfence_unprotect(unsigned long addr) { - return !KFENCE_WARN_ON(!kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), false)); + kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), false); } static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *meta) @@ -531,30 +531,19 @@ static void rcu_guarded_free(struct rcu_head *h) kfence_guarded_free((void *)meta->addr, meta, false); } -/* - * Initialization of the KFENCE pool after its allocation. - * Returns 0 on success; otherwise returns the address up to - * which partial initialization succeeded. - */ -static unsigned long kfence_init_pool(void) +static void kfence_init_pool(void) { unsigned long addr = (unsigned long)__kfence_pool; int i; - if (!arch_kfence_init_pool()) - return addr; /* * Protect the first 2 pages. The first page is mostly unnecessary, and * merely serves as an extended guard page. However, adding one * additional page in the beginning gives us an even number of pages, * which simplifies the mapping of address to metadata index. */ - for (i = 0; i < 2; i++) { - if (unlikely(!kfence_protect(addr))) - return addr; - - addr += PAGE_SIZE; - } + for (i = 0; i < 2; i++, addr += PAGE_SIZE) + kfence_protect(addr); for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++, addr += 2 * PAGE_SIZE) { struct kfence_metadata *meta = &kfence_metadata[i]; @@ -568,38 +557,33 @@ static unsigned long kfence_init_pool(void) list_add_tail(&meta->list, &kfence_freelist); /* Protect the right redzone. */ - if (unlikely(!kfence_protect(addr + PAGE_SIZE))) - return addr; + kfence_protect(addr + PAGE_SIZE); __folio_set_slab(slab_folio(slab)); #ifdef CONFIG_MEMCG slab->memcg_data = (unsigned long)&meta->objcg | MEMCG_DATA_OBJCGS; #endif } - - return 0; } static bool __init kfence_init_pool_early(void) { - unsigned long addr; - if (!__kfence_pool) return false; - addr = kfence_init_pool(); - - if (!addr) { - /* - * The pool is live and will never be deallocated from this point on. - * Ignore the pool object from the kmemleak phys object tree, as it would - * otherwise overlap with allocations returned by kfence_alloc(), which - * are registered with kmemleak through the slab post-alloc hook. - */ - kmemleak_ignore_phys(__pa(__kfence_pool)); - return true; - } + if (!arch_kfence_init_pool()) + goto free; + kfence_init_pool(); + /* + * The pool is live and will never be deallocated from this point on. + * Ignore the pool object from the kmemleak phys object tree, as it would + * otherwise overlap with allocations returned by kfence_alloc(), which + * are registered with kmemleak through the slab post-alloc hook. + */ + kmemleak_ignore_phys(__pa(__kfence_pool)); + return true; +free: /* * Only release unprotected pages, and do not try to go back and change * page attributes due to risk of failing to do so as well. If changing @@ -607,27 +591,7 @@ static bool __init kfence_init_pool_early(void) * fails for the first page, and therefore expect addr==__kfence_pool in * most failure cases. */ - memblock_free_late(__pa(addr), KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool)); - __kfence_pool = NULL; - return false; -} - -static bool kfence_init_pool_late(void) -{ - unsigned long addr, free_size; - - addr = kfence_init_pool(); - - if (!addr) - return true; - - /* Same as above. */ - free_size = KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool); -#ifdef CONFIG_CONTIG_ALLOC - free_contig_range(page_to_pfn(virt_to_page((void *)addr)), free_size / PAGE_SIZE); -#else - free_pages_exact((void *)addr, free_size); -#endif + memblock_free_late(__pa(__kfence_pool), KFENCE_POOL_SIZE); __kfence_pool = NULL; return false; } @@ -830,30 +794,50 @@ void __init kfence_init(void) kfence_init_enable(); } -static int kfence_init_late(void) -{ - const unsigned long nr_pages = KFENCE_POOL_SIZE / PAGE_SIZE; #ifdef CONFIG_CONTIG_ALLOC - struct page *pages; +static inline void *kfence_pool_alloc(void) +{ + struct page *page = alloc_contig_pages(KFENCE_POOL_SIZE / PAGE_SIZE, + GFP_KERNEL, first_online_node, NULL); - pages = alloc_contig_pages(nr_pages, GFP_KERNEL, first_online_node, NULL); - if (!pages) - return -ENOMEM; - __kfence_pool = page_to_virt(pages); + return page ? page_to_virt(page) : NULL; +} + +static inline void kfence_pool_free(const void *ptr) +{ + free_contig_range(page_to_pfn(virt_to_page(ptr)), KFENCE_POOL_SIZE / PAGE_SIZE); +} #else +static inline void *kfence_pool_alloc(void) +{ BUILD_BUG_ON_MSG(get_order(KFENCE_POOL_SIZE) > MAX_ORDER, "CONFIG_KFENCE_NUM_OBJECTS is too large for buddy allocator"); - __kfence_pool = alloc_pages_exact(KFENCE_POOL_SIZE, GFP_KERNEL); + return alloc_pages_exact(KFENCE_POOL_SIZE, GFP_KERNEL); +} + +static inline void kfence_pool_free(const void *ptr) +{ + free_pages_exact(virt_to_page(ptr), KFENCE_POOL_SIZE); +} +#endif + +static int kfence_init_late(void) +{ + if (__kfence_pool) + return 0; + + __kfence_pool = kfence_pool_alloc(); if (!__kfence_pool) return -ENOMEM; -#endif - if (!kfence_init_pool_late()) { - pr_err("%s failed\n", __func__); + if (!arch_kfence_init_pool()) { + kfence_pool_free(__kfence_pool); + __kfence_pool = NULL; return -EBUSY; } + kfence_init_pool(); kfence_init_enable(); kfence_debugfs_init(); @@ -862,8 +846,8 @@ static int kfence_init_late(void) static int kfence_enable_late(void) { - if (!__kfence_pool) - return kfence_init_late(); + if (kfence_init_late()) + return -ENOMEM; WRITE_ONCE(kfence_enabled, true); queue_delayed_work(system_unbound_wq, &kfence_timer, 0); @@ -1054,8 +1038,9 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs if (!is_kfence_address((void *)addr)) return false; - if (!READ_ONCE(kfence_enabled)) /* If disabled at runtime ... */ - return kfence_unprotect(addr); /* ... unprotect and proceed. */ + /* If disabled at runtime ... unprotect and proceed. */ + if (!READ_ONCE(kfence_enabled)) + goto out; atomic_long_inc(&counters[KFENCE_COUNTER_BUGS]); @@ -1079,7 +1064,7 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs } if (!to_report) - goto out; + goto report; raw_spin_lock_irqsave(&to_report->lock, flags); to_report->unprotected_page = addr; @@ -1093,7 +1078,7 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs } else { to_report = addr_to_metadata(addr); if (!to_report) - goto out; + goto report; raw_spin_lock_irqsave(&to_report->lock, flags); error_type = KFENCE_ERROR_UAF; @@ -1105,7 +1090,7 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs */ } -out: +report: if (to_report) { kfence_report_error(addr, is_write, regs, to_report, error_type); raw_spin_unlock_irqrestore(&to_report->lock, flags); @@ -1113,6 +1098,7 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs /* This may be a UAF or OOB access, but we can't be sure. */ kfence_report_error(addr, is_write, regs, NULL, KFENCE_ERROR_INVALID); } - - return kfence_unprotect(addr); /* Unprotect and let access proceed. */ +out: + kfence_unprotect(addr); + return true; } From patchwork Tue Mar 28 09:58:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1833BC76195 for ; Tue, 28 Mar 2023 09:58:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACFB16B007B; Tue, 28 Mar 2023 05:58:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A58D46B007D; Tue, 28 Mar 2023 05:58:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D2C46B007E; Tue, 28 Mar 2023 05:58:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7A7076B007B for ; Tue, 28 Mar 2023 05:58:50 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 47AAD1A015B for ; Tue, 28 Mar 2023 09:58:50 +0000 (UTC) X-FDA: 80617858020.02.7C2E7ED Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 8504540004 for ; Tue, 28 Mar 2023 09:58:48 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Qx3+v0sZ; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997528; a=rsa-sha256; cv=none; b=y91xWq1A5UQxj9UuKbLxlsyfNtEpH+CTcOuThSXaBZMZEQRsVnKxbyNUtS1ywXesl4dJRM TPkRavACedfjngsW+LUiKIfAdj2Fak4F5v8rovbeV2DVARJxPCahnyriYaayVS2WtVb1Nq eeCVTBFm6lhzvrCDI//ARYKywl/4lRU= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=Qx3+v0sZ; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=15Z/PJf0Etf+mCvL+KJkEExIR5UO6fpjH+VUxXpLOps=; b=HvRVIt2DYcGQVQcq4qOA2dpgzmyBNWUAhW62ZBIeajTaYpnfA0fcG5czJPvouiCv7uWfnS AMw1ohGDiBd8jzxPG14DYU1i/eqFP57aBq7HO1L4QxBHvRvfmP/69Bhg+cTT0ekNLdS6Hm e2IgA9CPj4j59FEMUQoUJDT0q3Ls8B8= Received: by mail-pf1-f172.google.com with SMTP id fd25so7592631pfb.1 for ; Tue, 28 Mar 2023 02:58:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=15Z/PJf0Etf+mCvL+KJkEExIR5UO6fpjH+VUxXpLOps=; b=Qx3+v0sZWmUR+ce5l5MqQpd0SAUJEWb0aMDPuqaDGWTo2nZ3Wt+/EWNMN0CiEAmd00 oTzaOuBYvp5IpyTjML80vgl9ojlZu6X7RxGSvyffWJFXCt43KPzhXd/iS8yeRgsejWrP FB+3T6fofiINoekHWqN2WApQUWGvX7bhd+zsJ0LEfvQMmZC7J/Z58o6Yu8Fg0YJgA5dD U5S/kAP6t560+N8wYkN1uqz4IRP232N9dBKfyi+Pr0qHtZeRD4S7DIfi+nbFhJk6P5UR 6HnQZc7vHWimyHei7ItyH6oDI/3Q67JEsR5mXPYeO9JRjRx5a5nwt5lMF5XG/eFNzRa0 KjZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=15Z/PJf0Etf+mCvL+KJkEExIR5UO6fpjH+VUxXpLOps=; b=0qzazXjFdqZGmsSdk/5tVmOuNLkir6ApbMWKXOiC/Vi1GuanWdeQH0eQBXu5amcL9J JFuv6A3/M8cJL1XCH+iP6jAIFU9T863+OwAiOp37VlvDiLVbF7UtIsTjxwl12WGt59TO lKMBRvtMnfcfl1KxQuuhumhCpBHh4VH3jWQhLQLbXWJ8zKdvlPX7CmbDQ1hWVJtUE5FB pk5mGUMe0XvsI/kkje+SOmAg7nFdGwGsGhIvVSV2kiyB6BOJB45pLu7ItGj3BVt+sQtd 61ir+as/8OqBAFm+oiOr0WOeTUT0MHfs2ZVImkuHMupzgp1UOW7dhReYDX/w6nnZKbPU 2yDg== X-Gm-Message-State: AAQBX9d4awA/Nqr+gCh6PEjibgeA92QHc8+LUD5r0ebPj+xBkWcVi4/0 DMbXg/OetkNLnY1n84oX8LTU7A== X-Google-Smtp-Source: AKy350ZjS30LuwRfFP7DkSp3XNPM4Idyf1sfqofA7Lh6rHryfmWu8BfaGo3V4Xaq7lv0dFHBxL1eMg== X-Received: by 2002:a62:6454:0:b0:5a8:b2bf:26ac with SMTP id y81-20020a626454000000b005a8b2bf26acmr14066890pfb.20.1679997527531; Tue, 28 Mar 2023 02:58:47 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:46 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 4/6] mm: kfence: remove useless check for CONFIG_KFENCE_NUM_OBJECTS Date: Tue, 28 Mar 2023 17:58:05 +0800 Message-Id: <20230328095807.7014-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 8504540004 X-Rspamd-Server: rspam01 X-Stat-Signature: d48g8nwob5gm81iqmcdt14cf5x6hq1st X-HE-Tag: 1679997528-418966 X-HE-Meta: U2FsdGVkX18k8uYsUdKVLrZRAPtfaYBFNLYRELLlkOZK4tLXDFRNJDBfKHcz3PtdC4rXfIlvppp+vd7/zaUswqJbGUEKyX5rJtVYEkIfKThroU/CXB3c2Ude5mgRTPhlcFAPz01BLEBullzP9ctdKVtC7WBrjPhqwcX675902pnrGaXM9nXEaVbr8w2+GfCFRwFlAORwtORljz6APSoSJl0a3tqw14JDMjLRxttbthwHFb/Ww40kWFDcykA9MgsXlZIaKyDOnR4bXmD+tuDD8z7kjT97TkbIOHcTPtvvgeS9EJAQPQOMTYskzjC1E9pa7OlTOMO4TcF+cJaI76GHJnxd/JbLfIWARcrug1UWm7r95JapJh2qj4633VZ+vKLwVxmSmXkH5KouVuBmwcOZmloiff5RoOFqG7+6lHaz5piM8dZMrrP5zaOE6UERAoM59c5wKmNLK9pGp4vk3RcZ5Hqr1XB1cp2vG19SmusX4pWoxF7OD67yP+K91XqomL+Yj0Xm8jHMP3hIf7zR0mxDHscZ02K+BCDiZ2F3x4xVpcBhW1L6Zfqy5/FsZsh38pOQ7rHc4a2O9n61fkFq3SXHR982scWUqrLcaD9YHIDYWxMjAnc+8PRX+/orP4EjW+nOj7PD4BgtKrTsvROJYyfiO2zL6/++u0OerTVEGR8FbeZTdzHUvNXq2uTaB5vYghWLHavu6WX7c07g1Q0BjhWpr7SeDU+V1F6FaxPmg0UbSGo1BVfntbxOcCbh+z2VWUpyE7OLKrjqCnnVpk0j7Io5tgyNSDeP2NbwKqr2smNB92N9oBET7s2EQXNaXp7tMUXpzFvHWpyCDkMvb5Wuyekh7N4DwW/yyr3Sce8HtjjZ7esovr/7IbQp0qGQu7LtpD5O5vFM2FwIb0DACHcoZ2E/S5Nj5vcAdP3Mv9K9YiWTYnNXOG1sLD6FtLACB5ZvIkOrhy4OKZMD+o29rinUEbc ZNNhjgmi O7e4qcsh2aOY2NmPJC8/IWjPglz6XCARb4ZC8dqmTAXToxqKzXz8z6mcO8H2KM5D32tjUloQrSznItQSax0sjuTfqQ3aJNcE1vc3cbqf3TWVgGF5aBgT2OlObo/ktCjnXgBkfPp8fZzEYjkzClDa2E70Dl2Be0crpzvq8Su3cfxMSFFVPqByjowE+bEchLII1K+Z2pU0gCNYt/1kGCC+iDp0h3YKtnrM7mlFZ5O7H/brJY8JhyzS0XLeBn9CGalGvZ//8KF/pLuG5XCZm/cqTLgaqdwoxdCidbWORyKaJ9dFW1pteRDFpeajc1h4+7LzIP6YUAP/sRYpSg/7l8wrKC8gbL+ob9w9Oa37dLpOaB5h2Gp9Emg5dZ3HJSM9xcyvfoI3V2+qEZbvPd5UdNs/CKmL6Z53PXq+JwcwwNSBkdsAMJ4bH+sLgwZhW4M/L886ehrRaX/dE38Z7kpwS7EaboJrihDA7SEx4h6rB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The CONFIG_KFENCE_NUM_OBJECTS is limited by kconfig and vary from 1 to 65535, so CONFIG_KFENCE_NUM_OBJECTS cannot be equabl to or smaller than 0. Removing it to simplify code. Signed-off-by: Muchun Song Reviewed-by: Alexander Potapenko --- mm/kfence/core.c | 1 - 1 file changed, 1 deletion(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 5726bf2ae13c..41befcb3b069 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -115,7 +115,6 @@ EXPORT_SYMBOL(__kfence_pool); /* Export for test modules. */ * Per-object metadata, with one-to-one mapping of object metadata to * backing pages (in __kfence_pool). */ -static_assert(CONFIG_KFENCE_NUM_OBJECTS > 0); struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS]; /* Freelist with available objects. */ From patchwork Tue Mar 28 09:58:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 742F2C76195 for ; Tue, 28 Mar 2023 09:58:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E4C06B007D; Tue, 28 Mar 2023 05:58:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06FDE6B007E; Tue, 28 Mar 2023 05:58:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2A366B0080; Tue, 28 Mar 2023 05:58:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D1E966B007D for ; Tue, 28 Mar 2023 05:58:57 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A620E1608F8 for ; Tue, 28 Mar 2023 09:58:57 +0000 (UTC) X-FDA: 80617858314.11.13C19FA Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf13.hostedemail.com (Postfix) with ESMTP id DBE0420008 for ; Tue, 28 Mar 2023 09:58:55 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cRycGiJJ; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IMHBZhrpM5z4k2ZtEf9hxQYrO1Qyd30oWmty+49rOpw=; b=W3uVAaas25PnBvku/a8moBzH+WDCsIaC80KkeLMQs0T9bn7OS1gPZb8IRqsq/f3EQO9tki 8YMkQKHPGmJHzYvpz57aipJEpp+rQIqET/+KuRxw4p+D5phfeRZglqS6kY5dTuK9gsqPbh LMBr0X2Fngk+bW+vaYKM98rWC6zpGe4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=cRycGiJJ; spf=pass (imf13.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997535; a=rsa-sha256; cv=none; b=TYpRIpHHcboEE4MvptDmWuUhG03Lrg/qn49+7EO6S63FDRx4ZETVwWeea1EvqAOGczZAuH JWtbYxXrd4pvzGkOyEDtX04O/D/zcP2uLSSWnZ6bLhq5Ezuqde/QQh6/k+KdpEF7QdmaTo AEXLLMPy0zptDSYyxsY1NjJKSWqZLL4= Received: by mail-pl1-f175.google.com with SMTP id ja10so11134819plb.5 for ; Tue, 28 Mar 2023 02:58:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IMHBZhrpM5z4k2ZtEf9hxQYrO1Qyd30oWmty+49rOpw=; b=cRycGiJJnkVXyhfrk/dJRNSCMoFpZNGGzl7dazDlUieFTT+oQ+yoOI0svPNXNgtJqi ueJI4DHM5wT7gQ5RfXmNTIxsoHKaTtsJl7K7gn59uLOroQOiZT59BCAtb8NqmrT1Mh3k Gmp9KRHfZU2HK7LFnjWVQlIsu8uap2Y3Occ0QtUuXKCCiQ/RNy7ku8loyA64bCWAh8nu YarRVEK3zgJVrD1NAyG+Rw6Yp31+UQjj/8IL6k6ldDxLShPxNG7chryjPAjYipTl6RtD e5Kw7dStq3e/3/leQwjmSZgKAQc77VNntEv25vM1Y9n+FcT/YOolm+LYWL6kjyf2byr7 cH1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997535; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IMHBZhrpM5z4k2ZtEf9hxQYrO1Qyd30oWmty+49rOpw=; b=yo9b8baILrvf2rfdNAJGPWURcTckVXL74IRa0ZMWgYc41usrAZKWzkNruHwUWN+F62 h5ZEodRPg/0arY+lYGpqf/yUFHgVJBWr22Cwx5DznyXvrWFnyvOqhhA8etvoY0KguVVL RLJP8+5Zmo4OMZOc4jRerjOAEgy8c9L8rX6/zMuBCsvQ8I+fUURKlW5sF81pnfV+Xbm2 4E1axldcgyq/+9rCkTMvup6qzPI6GnKLj25zb/R4Se20AQRBEx72+kZ7lDVyGUMKovW6 VoUhsPEKOwDRDXKyA6r5HLBFO6OFNL/LoSI+0SpH6RCa2+3DzXhM/AMZGBOQAkbCFfUI s2/g== X-Gm-Message-State: AO0yUKX2CnHM9N0lynuSv+/VTvK7otblx767AruCP4eiYyuht+9iCq0B JV3Lu0eqRc8n+sA0+CJX1UG0/A== X-Google-Smtp-Source: AK7set932jr2dvDlZkZ6I1j9l5nVAPzrKihlTnIBzV515yMv5dukdFImzzt3Wkm14B5fhfxAnCRdyA== X-Received: by 2002:a05:6a20:6baf:b0:da:1e1:3f46 with SMTP id bu47-20020a056a206baf00b000da01e13f46mr13145721pzb.31.1679997534793; Tue, 28 Mar 2023 02:58:54 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:54 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 5/6] mm: kfence: change kfence pool page layout Date: Tue, 28 Mar 2023 17:58:06 +0800 Message-Id: <20230328095807.7014-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DBE0420008 X-Rspam-User: X-Stat-Signature: hm3yrdu8rg8i8hec7myakooczr5gd8ka X-HE-Tag: 1679997535-482058 X-HE-Meta: U2FsdGVkX18uhV1EEiIMjFkeDq99mqUnoW7r5CK4AickNZydhaQs1hMKS7mnTcZhgQCh7FEd2q+ke70ai1EjNEL6XPkRBC8qa9iUZMdUGJD8e2mKK/Un78F8hpgPARLtBXoxb5pHFU9yvqGrqq3OvVsVyC+vMsQZPwl3JQZ1UMT+H/2FBEhyHP03Q2JHF9llwuO+Ui5Ho+Fx7ekJY3qc3rTYP4UQIC/ISpc1TDru+DuYIbqAsLnUw6/OmqYqpHRjs7wtnGbOvnaEX2P9lnNJceKpH/NQFGd57vZt5Gc2bVhTF9HvIl90uxd6sBgHvy41T6OdZBJ6jIf7vZKvrVMBNOF3xX0w+byo46E0+dcPWWvzB8Nw+YZBekBcfr2Vw1UMTLtBRxMiqojfofZD2UU6WzjWGgrD7TMMQFb4Mw8R5LhS5VTN3xXN9XudJ32jg7J2UQuAnIQQrWI/gNEI4gINcpGcBITguNwdzZ1NkjIo9hTt0HQYk+r5Ws6A/SNRHpL4gqeHIg0frTsBk0fnmfuAC29VX2TvimdafzoNtbv89he1StjAjT0V6shCcUsRb4rlyds+YtHYJ4hG05vx/fnq3rs+MMxqFPhbp37cqkcSVpBNge19JyctEr5iHzMC0d8Di/bz6/05GbXVFocsu7nMV1Yl3npMnYAeNBvNatKrcW2VIb39ObvShVyYC0/Rvdl7M2UfMVdHjZzoSxbkwLBIkQN4M24+ZuspruCof4qUMgZN11BJi/tnMGy3aKTY5UA0i1kGzZOgo9fg7t4c3nIT+ttOJI4C2OHSLKHVlbC2i2eRbrJSiUmG2jChneUk+QDR70GJl5tk2nuzgkteV1ZI6yo52Cm5g1jQg4G81mX4vD0mDKjI18EuwoGnrkJErh4275CF1JpxZV/TnHa+7zXouajmty5QWYibiDs1txyh0u30vh9/dTKx2SkI0NarFMnNuJR8udTVFO5rnkp0J+2 5ZhM2OwO z0c7MvNOD6u9vtyE9YUFTGfcTxIe/pDmy9toyS/E1tbrInfeylNlNwP6tDmcwBuebMiRzFbCHgpo5SGhtdftZjUfE0/Z7ilZiRcLsFiaEHgsdZksumcfCfCjwDdbXbtDeHVd3s3ml92MU13YYlnHXyqo/jxQhRMBnkVdmQS2+zGzNcccgyUkwf9XZjLqA4CKvjbBkCv4zOy2po8rDBkgduwmrTcEUMRiOE90lDV/Je3B3AeBc31PS/mINO+k6vwBxN9Z8iEB7mSsil7jeRW3Q0ljgOppHb7NG7ZLcFNokoyGH9uCoWpiwMvlFj5sWtbc0NIIXC4jGH0JOmy8PPkHGRQ33RRy4ZDrsrULKHbO0P8vj9HZXqJjfvYbAQcgbFqkJkRWN6PtCITXcUzk+wAzSoX8zEwWEyUAQbeKcT5Y8u+LnUEzBEeP09OsAgQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The original kfence pool layout (Given a layout with 2 objects): +------------+------------+------------+------------+------------+------------+ | guard page | guard page | object | guard page | object | guard page | +------------+------------+------------+------------+------------+------------+ | | | +----kfence_metadata[0]---+----kfence_metadata[1]---+ The comment says "the additional page in the beginning gives us an even number of pages, which simplifies the mapping of address to metadata index". However, removing the additional page does not complicate any mapping calculations. So changing it to the new layout to save a page. And remmove the KFENCE_ERROR_INVALID test since we cannot test this case easily. The new kfence pool layout (Given a layout with 2 objects): +------------+------------+------------+------------+------------+ | guard page | object | guard page | object | guard page | +------------+------------+------------+------------+------------+ | | | +----kfence_metadata[0]---+----kfence_metadata[1]---+ Signed-off-by: Muchun Song --- include/linux/kfence.h | 8 ++------ mm/kfence/core.c | 40 ++++++++-------------------------------- mm/kfence/kfence.h | 2 +- mm/kfence/kfence_test.c | 14 -------------- 4 files changed, 11 insertions(+), 53 deletions(-) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 726857a4b680..25b13a892717 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -19,12 +19,8 @@ extern unsigned long kfence_sample_interval; -/* - * We allocate an even number of pages, as it simplifies calculations to map - * address to metadata indices; effectively, the very first page serves as an - * extended guard page, but otherwise has no special purpose. - */ -#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE) +/* The last page serves as an extended guard page. */ +#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS * 2 + 1) * PAGE_SIZE) extern char *__kfence_pool; DECLARE_STATIC_KEY_FALSE(kfence_allocation_key); diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 41befcb3b069..f205b860f460 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -240,24 +240,7 @@ static inline void kfence_unprotect(unsigned long addr) static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *meta) { - unsigned long offset = (meta - kfence_metadata + 1) * PAGE_SIZE * 2; - unsigned long pageaddr = (unsigned long)&__kfence_pool[offset]; - - /* The checks do not affect performance; only called from slow-paths. */ - - /* Only call with a pointer into kfence_metadata. */ - if (KFENCE_WARN_ON(meta < kfence_metadata || - meta >= kfence_metadata + CONFIG_KFENCE_NUM_OBJECTS)) - return 0; - - /* - * This metadata object only ever maps to 1 page; verify that the stored - * address is in the expected range. - */ - if (KFENCE_WARN_ON(ALIGN_DOWN(meta->addr, PAGE_SIZE) != pageaddr)) - return 0; - - return pageaddr; + return ALIGN_DOWN(meta->addr, PAGE_SIZE); } /* @@ -535,34 +518,27 @@ static void kfence_init_pool(void) unsigned long addr = (unsigned long)__kfence_pool; int i; - /* - * Protect the first 2 pages. The first page is mostly unnecessary, and - * merely serves as an extended guard page. However, adding one - * additional page in the beginning gives us an even number of pages, - * which simplifies the mapping of address to metadata index. - */ - for (i = 0; i < 2; i++, addr += PAGE_SIZE) - kfence_protect(addr); - for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++, addr += 2 * PAGE_SIZE) { struct kfence_metadata *meta = &kfence_metadata[i]; - struct slab *slab = page_slab(virt_to_page(addr)); + struct slab *slab = page_slab(virt_to_page(addr + PAGE_SIZE)); /* Initialize metadata. */ INIT_LIST_HEAD(&meta->list); raw_spin_lock_init(&meta->lock); meta->state = KFENCE_OBJECT_UNUSED; - meta->addr = addr; /* Initialize for validation in metadata_to_pageaddr(). */ + meta->addr = addr + PAGE_SIZE; list_add_tail(&meta->list, &kfence_freelist); - /* Protect the right redzone. */ - kfence_protect(addr + PAGE_SIZE); + /* Protect the left redzone. */ + kfence_protect(addr); __folio_set_slab(slab_folio(slab)); #ifdef CONFIG_MEMCG slab->memcg_data = (unsigned long)&meta->objcg | MEMCG_DATA_OBJCGS; #endif } + + kfence_protect(addr); } static bool __init kfence_init_pool_early(void) @@ -1043,7 +1019,7 @@ bool kfence_handle_page_fault(unsigned long addr, bool is_write, struct pt_regs atomic_long_inc(&counters[KFENCE_COUNTER_BUGS]); - if (page_index % 2) { + if (page_index % 2 == 0) { /* This is a redzone, report a buffer overflow. */ struct kfence_metadata *meta; int distance = 0; diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index 600f2e2431d6..249d420100a7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -110,7 +110,7 @@ static inline struct kfence_metadata *addr_to_metadata(unsigned long addr) * __kfence_pool, in which case we would report an "invalid access" * error. */ - index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2) - 1; + index = (addr - (unsigned long)__kfence_pool) / (PAGE_SIZE * 2); if (index < 0 || index >= CONFIG_KFENCE_NUM_OBJECTS) return NULL; diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c index b5d66a69200d..d479f9c8afb1 100644 --- a/mm/kfence/kfence_test.c +++ b/mm/kfence/kfence_test.c @@ -637,19 +637,6 @@ static void test_gfpzero(struct kunit *test) KUNIT_EXPECT_FALSE(test, report_available()); } -static void test_invalid_access(struct kunit *test) -{ - const struct expect_report expect = { - .type = KFENCE_ERROR_INVALID, - .fn = test_invalid_access, - .addr = &__kfence_pool[10], - .is_write = false, - }; - - READ_ONCE(__kfence_pool[10]); - KUNIT_EXPECT_TRUE(test, report_matches(&expect)); -} - /* Test SLAB_TYPESAFE_BY_RCU works. */ static void test_memcache_typesafe_by_rcu(struct kunit *test) { @@ -787,7 +774,6 @@ static struct kunit_case kfence_test_cases[] = { KUNIT_CASE(test_kmalloc_aligned_oob_write), KUNIT_CASE(test_shrink_memcache), KUNIT_CASE(test_memcache_ctor), - KUNIT_CASE(test_invalid_access), KUNIT_CASE(test_gfpzero), KUNIT_CASE(test_memcache_typesafe_by_rcu), KUNIT_CASE(test_krealloc), From patchwork Tue Mar 28 09:58:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 13190792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68E46C761A6 for ; Tue, 28 Mar 2023 09:59:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E4936B007E; Tue, 28 Mar 2023 05:59:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06EE36B0080; Tue, 28 Mar 2023 05:59:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E296E6B0081; Tue, 28 Mar 2023 05:59:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D2E756B007E for ; Tue, 28 Mar 2023 05:59:02 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 971D41C319D for ; Tue, 28 Mar 2023 09:59:02 +0000 (UTC) X-FDA: 80617858524.29.23559C0 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf23.hostedemail.com (Postfix) with ESMTP id BC84B14000C for ; Tue, 28 Mar 2023 09:59:00 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VsNWH6wb; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf23.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679997540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a9z+2OKezgBl6Xq8s4/uft3FH4lHgs0F/FglMYKuxZE=; b=eT5XWJtRSIij+Gmq8AalfrLotkG4OuIoZQDSrPSlVRZbzvRXgBr2r5Mnvn8I6C0a4HjBFf LXY43U/Ehw4SgfjyQ08vNTzCGab17BERlfLW4fWBMgYSv28NfAS3+YoJZ4b8AZY2ZBrMNa oFaKVAh8bTpeZ+i1c32SumgFHyJXxUg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=VsNWH6wb; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf23.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679997540; a=rsa-sha256; cv=none; b=hD3YDhfDlehmxSBmAPB5jTL/JJI2wROLR4DHi8gs/6WVIbl5y3YHfx/Pt+8HQQMCPZ0GcG h7EeNpz8h1ZautIqWzYqemdJiv5S1DfNVD6dzDQhDeGHbksZwGIgFPKVD2AG/3CKdl2JcH kFFd0dtQ7JVeCBMs6LUnV3M3RcWQSYw= Received: by mail-pj1-f47.google.com with SMTP id mp3-20020a17090b190300b0023fcc8ce113so14560189pjb.4 for ; Tue, 28 Mar 2023 02:59:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679997539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a9z+2OKezgBl6Xq8s4/uft3FH4lHgs0F/FglMYKuxZE=; b=VsNWH6wbabZBW09E5p1rywOaT5p8dwv3DVQF4JxvKy8pAkOwOWTmSR3v5YYbHdOVIH o9LpPW7UeEmwFT8bwKYV0uifRm0n7p6iEYBMeiwyTvcS/TasidJuyj7P88eL57v1KTDd tI7u4PDJStX8H8gJLJrbqJ2Kof9xiRkHbs+wEsWSrPnUYYMPFzysTbud6J24HgyFWNpP x/6rvB/nHTaJWJCevltlgD8XQOccc/yfD7VHMVUS6y71oqY16doN1wfGwwkfO74aIK1P TXfWcZDvSIRMhEA4BilI3L6ew7QAaPregsJwYg77jDT/Rie9uRJd+pu1GfBpPZf028JJ BL6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679997539; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a9z+2OKezgBl6Xq8s4/uft3FH4lHgs0F/FglMYKuxZE=; b=hE5ON2QLzNz2nlZVMVNawYCpB7U5kndfeRuDW8pn9I/f5CZaPZYK2qIIRPiecfcixI G6P4dBywY0yAs8sRom63p7WerJ4UrbDN/osFtX3a42t7HTDFTNkHQF5hjdVDfCf0At/U xCQICbKbTqo2TgN2T5CxGFc9beBYhIaXL7waEvDz0Rb8mn3Fk2U1j3XFNUM4DJY+TSJj 2ewJz7jzME9qSSdTMZ31DGofPQ1ZKB/73WdyzyxlwOQ7NSV11u2GxJwx1XsQgttLSWvP b2VBN7lzpTl1GMuhC05EfgIZwajgwaN4q+tdqBMAvBOslwWKpD65hqOZ9C9+dJ3frvje ishQ== X-Gm-Message-State: AO0yUKUmcFLvSk+xBI5c/m9MKqoxdU7ZX0eZb4jIvTncE0eoKVCR8T9z dTsne2P7tsS+uaXGQG1y+5TD7w== X-Google-Smtp-Source: AK7set+sSOqvm7BhrGFyb5OKdu0ZTvBWUBOT2Bfsk23ttc/ateaVqwe3xWklthmOX72MppMUmuO5pA== X-Received: by 2002:a05:6a20:6aa0:b0:d9:2d4e:c08c with SMTP id bi32-20020a056a206aa000b000d92d4ec08cmr12797492pzb.61.1679997539764; Tue, 28 Mar 2023 02:58:59 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([139.177.225.236]) by smtp.gmail.com with ESMTPSA id m26-20020aa78a1a000000b005a8a5be96b2sm17207556pfa.104.2023.03.28.02.58.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 28 Mar 2023 02:58:59 -0700 (PDT) From: Muchun Song To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, jannh@google.com, sjpark@amazon.de, muchun.song@linux.dev Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH 6/6] mm: kfence: replace ALIGN_DOWN(x, PAGE_SIZE) with PAGE_ALIGN_DOWN(x) Date: Tue, 28 Mar 2023 17:58:07 +0800 Message-Id: <20230328095807.7014-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230328095807.7014-1-songmuchun@bytedance.com> References: <20230328095807.7014-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BC84B14000C X-Stat-Signature: fdibim8i7mip3pcs4iq16kjnhafs3i7s X-HE-Tag: 1679997540-426727 X-HE-Meta: U2FsdGVkX19e90WuUUk0vHIVi/ksVClmFfxqX4Jxoggg9nBB/+ToxI+gDKRQwinPxPQ265rkd6mKjKSMmZgFwvi0zmBcwRbaeeJu2NIMNx8bZdbKuodHL639Shkxk+tk8s3pyRyoxX2rUnW3PEvdBnr8MgQXNnJ2XQwiisQwHKUj05Pvlq5OwjvEaXrCx0Ic6xkUXKnDyGZuGx8ENZ8aSEWwcdqxXpEjN/9qJlPHyNqRaABwBtn1edP4wV8ywidklC6e+wWAG+sil+MaSSfaNFBJ8jPPlNjZh1nfW79R93z6fZQF5WJk/HgWrATIYhrVearrVsgELWFd9C+f8CSGLNr8WfwfRcQh9WyG+7JA5p8ikE0EZ+cHH8OstJHVk0Rgq2Amax8nZJvMAFoJA5UJiGFduxrUJI85b7iFdkDwz4S8RpMBpEg5ytNGvBzCEnFZY+DR2Ct+OrFRSbDgoehAuqz0q8c2e+J/7CdKZho8lfzUMGTL9YXUOpHp4Nm6RKcQPEMWMYIcOKBbqNKv4XTqI0Yl8dsR0pdaLxdLARgckUUHgZUB54k/YVubkJh1pc9CBjTiihIPZyg2ecl056z/dpxBXm3owhocBQGFQJR6cChTXsU3bBCGrVPAjo8fBLazO9CkMzuIEwo7Akdo8v5YzDd9FQVBFktChcMxU+p4d/LnVZ3WYoqIwQJc05I+pMUS5ZNwLz5DOFLFbA4pHivaPyLkfUNkrYtb8/lhtnwORkswNMx+SzGNZdZKazRPNQ4UQW97K6Yhq70NLf69/PJcrvSMcELvCwQSS34YBpLuIeOXb1E+JHUHNWZIZcEYOpjXMGg3h5FmN78TPd0RqJro01HLb+CYtiOd0e/kpsEfHDq3nIqcRb1AWyjDz78Mj/x6CrZPI+tyznzh+WWjyQa/ixiPIIw+Vks9gQrWUSG9j++Z5bMExTx4JEvCWZvBXwxHlDP2ePhn+330oZ922tM lQhOqfQw D7c9YxLe/LcYQeCQ3TBBH3/4pt32wbR6B0zPFCrX+dgg8yoItEfYohvgWQl/70xwlHiNGko6X2bddAu4Y3k5isBeV4G7vQD/X1t3/HRwgKjo8yRNn5FufnGKd+eagUQtCm/aYQMp/wZa55kHC4YAuZhyVsIH4Cs6kDNG7/Pifeixlvm3q3L3+p4FI7Y/pRSAv4F1Yblrd2wHJBN1YYCmkDQR6C/awpRGaok3UrfEAxbnx4hn05I0F1W93pc7NKO3OGMgbX5iTQouCuUTPP1Xx1djAdD4vtN4xxRT63fHQXnCj3p2rvlqCYavi2rKDVg3E+j066Oa9ATv48u6fFWqMN2uw9k9WyrfKgwaN+Pkel+MxOoCoMIh4KOMcv0EFpoBmcryy3W8pl6Oyq2CfUeCQCEXWnyhz/0/L3RexccbvgbL/s4Sw4eG0BNIN7pCTmajv83v/VolN4aSBkiqGgLKBWUHen8bg9eBWZl0u X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace ALIGN_DOWN(x, PAGE_SIZE) with PAGE_ALIGN_DOWN(x) to simplify the code a bit. Signed-off-by: Muchun Song Reviewed-by: Marco Elver --- mm/kfence/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/kfence/core.c b/mm/kfence/core.c index f205b860f460..dbfb79a4d624 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -230,17 +230,17 @@ static bool alloc_covered_contains(u32 alloc_stack_hash) static inline void kfence_protect(unsigned long addr) { - kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), true); + kfence_protect_page(PAGE_ALIGN_DOWN(addr), true); } static inline void kfence_unprotect(unsigned long addr) { - kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), false); + kfence_protect_page(PAGE_ALIGN_DOWN(addr), false); } static inline unsigned long metadata_to_pageaddr(const struct kfence_metadata *meta) { - return ALIGN_DOWN(meta->addr, PAGE_SIZE); + return PAGE_ALIGN_DOWN(meta->addr); } /* @@ -308,7 +308,7 @@ static inline bool check_canary_byte(u8 *addr) /* __always_inline this to ensure we won't do an indirect call to fn. */ static __always_inline void for_each_canary(const struct kfence_metadata *meta, bool (*fn)(u8 *)) { - const unsigned long pageaddr = ALIGN_DOWN(meta->addr, PAGE_SIZE); + const unsigned long pageaddr = PAGE_ALIGN_DOWN(meta->addr); unsigned long addr; /* @@ -455,7 +455,7 @@ static void kfence_guarded_free(void *addr, struct kfence_metadata *meta, bool z } /* Detect racy use-after-free, or incorrect reallocation of this page by KFENCE. */ - kcsan_begin_scoped_access((void *)ALIGN_DOWN((unsigned long)addr, PAGE_SIZE), PAGE_SIZE, + kcsan_begin_scoped_access((void *)PAGE_ALIGN_DOWN((unsigned long)addr), PAGE_SIZE, KCSAN_ACCESS_SCOPED | KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT, &assert_page_exclusive); @@ -464,7 +464,7 @@ static void kfence_guarded_free(void *addr, struct kfence_metadata *meta, bool z /* Restore page protection if there was an OOB access. */ if (meta->unprotected_page) { - memzero_explicit((void *)ALIGN_DOWN(meta->unprotected_page, PAGE_SIZE), PAGE_SIZE); + memzero_explicit((void *)PAGE_ALIGN_DOWN(meta->unprotected_page), PAGE_SIZE); kfence_protect(meta->unprotected_page); meta->unprotected_page = 0; }