From patchwork Thu Mar 16 08:50:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenhua Huang X-Patchwork-Id: 13177238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 98B5DC6FD19 for ; Thu, 16 Mar 2023 08:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:CC :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=Hohi3qr0itFI4MRqVjlPXrhOKNANzCTQeox/Gt1K1kY=; b=3Zgoz4jELOZ2vW xJtX7TCE7LN9V3LOgI8+roVJRrATU3sAfHHur36ll2ZWQl3kgKu+sPL5+W8cqxMkbOj8iP/8rO9E5 DxoHmd3MJKpRdctBSl5Fj4QzrUKBu0HvTWMwwnQay6Jxx//hhs1cQRAh7aEG/gE3lq6DQ9lkg7yEE oGoxzi+d3HLViBl9k8XQdlB/Mv2NWN89vrn0ev1AKi9Cm4178BeI7DMfHXqcaSROixmxnSa1mTyBU E4Rvonk9J+ZBMDfiFU4DFxHcfxPwtj9ViQsp8Xh9l6hiU2rP8F+h4ikF1lcxi0QNdcnOI5Cep/g40 H3p715J/gwiOBtLxUkZg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pcjK9-00FluQ-1q; Thu, 16 Mar 2023 08:51:01 +0000 Received: from mx0b-0031df01.pphosted.com ([205.220.180.131]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pcjK6-00Flsw-0F for linux-arm-kernel@lists.infradead.org; Thu, 16 Mar 2023 08:51:00 +0000 Received: from pps.filterd (m0279873.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32G6I5GD029854; Thu, 16 Mar 2023 08:50:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=qcppdkim1; bh=e82hxHn52yqSiK6HiEB47+42nkEcrdxP6ydPxrg8nW8=; b=NSFDRT5Fd4Qvpg3nFILSr00slArVf+NOOGA85kObBI5dTt4XOwkBXhqs3f+pl1NNCTCG BzvudPl4qtKaaXKj+0lztIgQp8Se3VQPCMFsK6c03p8Kq4EuvlfDGA4BSlnfkq2KVgRl 5EowwRO5uklN1nw83sZsy9UrRnDG85wi5UVwT8rKXYPqz3mNGYGErNOGuwAn80BBlYZw z2gQOQzxySqSai1LBgR/gIKn+km2v7Md1f4Zo6KAxvtbDWTgu9cLuIG4e2dDlUQBmNGh 3ZsVqgWSSMDeZTZysFUc0fWAoq0CdzSbzFkP9JVKmF5y+oMcLZDfeVJwjCxOMbyrsNed ug== Received: from nalasppmta03.qualcomm.com (Global_NAT1.qualcomm.com [129.46.96.20]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3pbpy3h8a3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Mar 2023 08:50:41 +0000 Received: from nalasex01a.na.qualcomm.com (nalasex01a.na.qualcomm.com [10.47.209.196]) by NALASPPMTA03.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 32G8oe0m000605 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 16 Mar 2023 08:50:40 GMT Received: from zhenhuah-gv.qualcomm.com (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 16 Mar 2023 01:50:35 -0700 From: Zhenhua Huang To: , , , , , , , , , , CC: Zhenhua Huang , , , , , , Subject: [PATCH v9] mm,kfence: decouple kfence from page granularity mapping judgement Date: Thu, 16 Mar 2023 16:50:20 +0800 Message-ID: <1678956620-26103-1-git-send-email-quic_zhenhuah@quicinc.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nalasex01a.na.qualcomm.com (10.47.209.196) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 0HFkjlROkqOfuMDTe0_Xwv9AnTTKv4bV X-Proofpoint-ORIG-GUID: 0HFkjlROkqOfuMDTe0_Xwv9AnTTKv4bV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-16_06,2023-03-15_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 bulkscore=0 mlxlogscore=999 impostorscore=0 lowpriorityscore=0 suspectscore=0 phishscore=0 malwarescore=0 adultscore=0 clxscore=1015 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303150002 definitions=main-2303160074 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230316_015058_253133_887B21E3 X-CRM114-Status: GOOD ( 26.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Kfence only needs its pool to be mapped as page granularity, if it is inited early. Previous judgement was a bit over protected. From [1], Mark suggested to "just map the KFENCE region a page granularity". So I decouple it from judgement and do page granularity mapping for kfence pool only. Need to be noticed that late init of kfence pool still requires page granularity mapping. Page granularity mapping in theory cost more(2M per 1GB) memory on arm64 platform. Like what I've tested on QEMU(emulated 1GB RAM) with gki_defconfig, also turning off rodata protection: Before: [root@liebao ]# cat /proc/meminfo MemTotal: 999484 kB After: [root@liebao ]# cat /proc/meminfo MemTotal: 1001480 kB To implement this, also relocate the kfence pool allocation before the linear mapping setting up, arm64_kfence_alloc_pool is to allocate phys addr, __kfence_pool is to be set after linear mapping set up. LINK: [1] https://lore.kernel.org/linux-arm-kernel/Y+IsdrvDNILA59UN@FVFF77S0Q05N/ Suggested-by: Mark Rutland Signed-off-by: Zhenhua Huang --- arch/arm64/include/asm/kfence.h | 16 +++++++++++ arch/arm64/mm/mmu.c | 59 +++++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 9 +++++-- include/linux/kfence.h | 1 + mm/kfence/core.c | 4 +++ 5 files changed, 87 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kfence.h b/arch/arm64/include/asm/kfence.h index aa855c6..8143c91 100644 --- a/arch/arm64/include/asm/kfence.h +++ b/arch/arm64/include/asm/kfence.h @@ -10,6 +10,22 @@ #include +extern phys_addr_t early_kfence_pool; + +#ifdef CONFIG_KFENCE + +extern char *__kfence_pool; +static inline void kfence_set_pool(phys_addr_t addr) +{ + __kfence_pool = phys_to_virt(addr); +} + +#else + +static inline void kfence_set_pool(phys_addr_t addr) { } + +#endif + static inline bool arch_kfence_init_pool(void) { return true; } static inline bool kfence_protect_page(unsigned long addr, bool protect) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6f9d889..61944c70 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -38,6 +39,7 @@ #include #include #include +#include #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) @@ -525,6 +527,48 @@ static int __init enable_crash_mem_map(char *arg) } early_param("crashkernel", enable_crash_mem_map); +#ifdef CONFIG_KFENCE + +static bool kfence_early_init __initdata = !!CONFIG_KFENCE_SAMPLE_INTERVAL; +/* + * early_param can be parsed before linear mapping + * set up + */ +static int __init parse_kfence_early_init(char *p) +{ + int val; + + if (get_option(&p, &val)) + kfence_early_init = !!val; + return 0; +} +early_param("kfence.sample_interval", parse_kfence_early_init); + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + phys_addr_t kfence_pool; + + if (!kfence_early_init) + return 0; + + kfence_pool = memblock_phys_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); + if (!kfence_pool) + pr_err("failed to allocate kfence pool\n"); + + return kfence_pool; +} + +#else + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + return 0; +} + +#endif + +phys_addr_t early_kfence_pool; + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN); @@ -543,6 +587,10 @@ static void __init map_mem(pgd_t *pgdp) */ BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + early_kfence_pool = arm64_kfence_alloc_pool(); + if (early_kfence_pool) + memblock_mark_nomap(early_kfence_pool, KFENCE_POOL_SIZE); + if (can_set_direct_map()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; @@ -608,6 +656,17 @@ static void __init map_mem(pgd_t *pgdp) } } #endif + + /* Kfence pool needs page-level mapping */ + if (early_kfence_pool) { + __map_memblock(pgdp, early_kfence_pool, + early_kfence_pool + KFENCE_POOL_SIZE, + pgprot_tagged(PAGE_KERNEL), + NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); + memblock_clear_nomap(early_kfence_pool, KFENCE_POOL_SIZE); + /* kfence_pool really mapped now */ + kfence_set_pool(early_kfence_pool); + } } void mark_rodata_ro(void) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 79dd201..7ce5295 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -7,10 +7,12 @@ #include #include #include +#include #include #include #include +#include struct page_change_data { pgprot_t set_mask; @@ -22,12 +24,15 @@ bool rodata_full __ro_after_init = IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED bool can_set_direct_map(void) { /* - * rodata_full, DEBUG_PAGEALLOC and KFENCE require linear map to be + * rodata_full and DEBUG_PAGEALLOC require linear map to be * mapped at page granularity, so that it is possible to * protect/unprotect single pages. + * + * Kfence pool requires page granularity mapping also if we init it + * late. */ return (rodata_enabled && rodata_full) || debug_pagealloc_enabled() || - IS_ENABLED(CONFIG_KFENCE); + (IS_ENABLED(CONFIG_KFENCE) && !early_kfence_pool); } static int change_page_range(pte_t *ptep, unsigned long addr, void *data) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 726857a..91cbcc9 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -222,6 +222,7 @@ bool __kfence_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *sla #else /* CONFIG_KFENCE */ +#define KFENCE_POOL_SIZE 0 static inline bool is_kfence_address(const void *addr) { return false; } static inline void kfence_alloc_pool(void) { } static inline void kfence_init(void) { } diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 5349c37..e05ccf1 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -814,6 +814,10 @@ void __init kfence_alloc_pool(void) if (!kfence_sample_interval) return; + /* if the pool has already been initialized by arch, skip the below */ + if (__kfence_pool) + return; + __kfence_pool = memblock_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); if (!__kfence_pool)