From patchwork Fri Nov 11 17:11:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87D74C43217 for ; Fri, 11 Nov 2022 17:13:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cDn18vatDb2H1qrhqhWP7UbjaKJEhARle9P19URvnTw=; b=e5ej49r73pA1K4 69UGi6qH3yuhWiqmcfVQv4TxEGmAWH110oTSY8RzmccLGEqu3QZFUHiJC8ud3mgGY9n8GYLZbTwEb sv1p0btlrKOsyLO/2sfm2DB6F99jTIFOZ0Pc6WVHjNgsKz9hXpPgzReeCw1pr3m8MFne/3w7kqeBK 3ZkPSQ0oHHmzYQ/Ors/QvwqvEx8y04HD6KCgB8m6adnrK2Hr5xLv3GP94eKxNF9qJ+B58a8SdlXCq E1rx5mgzvFmEyM97oWRhk+jdIVW2kiuCmi7j1XCot6XwcwVwNUfTTHBUvd5lv33Z0ZuW5MQkaYago KlOd7/EbUG+riQd9w3CA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZp-00H7Uf-P6; Fri, 11 Nov 2022 17:12:26 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZm-00H7S1-0J for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:23 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5DD2362066; Fri, 11 Nov 2022 17:12:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E69A3C43470; Fri, 11 Nov 2022 17:12:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186740; bh=J++SBGWUY6FVLhysDTR5H/9o1FHWzVTfhLMhqjMESfM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FVHka5vxBw/Ui67mfLdp3Vl6o+ignFj807EZuvG3SBYeEFqUKwKe/imREtduaOi7B tT/c0yzsjJAtq0lbcZEuKp5SxzlwiRgzReeXB2yp7z2L7Fp1/+Nb5xrr29dQ+GfniH 5lfCPacql2bARaTE1v326YrZNvKJ1hxXtlEYmdcRySNAqXNTCoQlSK1KceQ6N+8DKX r0o6kGwLTm/XvvQfXnuY1ddXBABIIETp3ZkDyasi1HH/+1a4izEB5ijlFcHsiWb/H6 eOJ55YTV2m6rwMkL2sq7mK3+H9kw5b4w/nd2s3SP3tcTfnNW4NhqRvIMNHWzrlTfzl uWOMWvWRGsT9g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 01/33] arm64: mm: Avoid SWAPPER_BLOCK_xxx constants in FDT fixmap logic Date: Fri, 11 Nov 2022 18:11:29 +0100 Message-Id: <20221111171201.2088501-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4747; i=ardb@kernel.org; h=from:subject; bh=J++SBGWUY6FVLhysDTR5H/9o1FHWzVTfhLMhqjMESfM=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboIq+zZ9FL9wP02x8xraMZ1LSQZx9Eszwi1VJDk1 ot94QeuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CKgAKCRDDTyI5ktmPJN7KDA CaeO1xRNCZqquTHymhy4cz+oWANlsYFEo4Ga3tJl2LZBifb8s17gP1lBSXicRqQ6k0wq3IQRb+5OAD 4Gc2H9fIvVi9C6+Ni7cVIHneZSqSJJKjBC+McIB2+T80ZCjXaAt2oBwa3rE/Cr5r1TkMzPX+QwqWod Q8ePZcKN74UYtNqMt22RRU9I44bii3Mu8U9csF+/uPwyJ9yEydqht9peKaMi/HrHnUgK4nVtiCGwyV EY1MBuNDzeEevFdSBUI+me+DZLsemPb5ZKDn8oGgV8Lxe4Qtx6H1B0OQcLVeEeYxo68NKa0EE/kVhi dgxzpn9iMAyRhOSMmbh4moFRrcFhhjQ4zblXYymj61pHcmI+UTlA4QQtDnLy/LGfJ7eUtHTvj0m2jo awe14nS0srGa6rvd5CXUlvzRFjdo9tWrlOL8ATGVDSLG9xZQjapcfZDom3JPv0srQcpFWR0F1EMi+k +Q2xDboPPOhmmxx3CA+ihUfjBTg2xYPWrEemOCwUHK2fc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091222_164495_82B886AF X-CRM114-Status: GOOD ( 26.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The FDT is permitted to be up to 2 MiB in size, and is mapped in the fixmap region using the same granularity as we use for the initial ID map, and up until recently, as we use for the preliminary mapping of the kernel in swapper_pg_dir. However, even though the constants are the same, the motivation is different: on 4k pagesize configurations, the fixmap region only has 2 MiB's worth of level 3 PTE slots to begin with, and so mapping the FDT down to pages in the fixmap would be wasteful, and this is why we use block mappings in this case. This is also documented in the boot protocol, i.e., adjacent regions must not be used by the platform in a away that could result in the need for memory attributes that conflict with the cacheable attributes used for the FDT block mapping. For larger page sizes, using block mappings is unnecessary, and given the potential issues caused by rounding, undesirable, so we use page mappings in that case. So to convey that this granularity is unrelated to the swapper block size, and to allow us to rename or remove the associated constants in a subsequent patch, use our own constants to define the granularity. No functional change intended, although the FDT fixmap virtual address will no longer be 2 MiB aligned on non-4k pages configurations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 5 ++-- arch/arm64/mm/mmu.c | 27 ++++++++++---------- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 71ed5fdf718bd0fd..d09654af5b1277c6 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -40,11 +40,12 @@ enum fixed_addresses { * maximum supported size, and put it at the top of the fixmap region. * The additional space ensures that any FDT that does not exceed * MAX_FDT_SIZE can be mapped regardless of whether it crosses any - * 2 MB alignment boundaries. + * 2 MB alignment boundaries on 4k pages configurations. * * Keep this at the top so it remains 2 MB aligned. */ -#define FIX_FDT_SIZE (MAX_FDT_SIZE + SZ_2M) +#define FIX_FDT_BSIZE (MAX_FDT_SIZE >= PMD_SIZE ? PMD_SIZE : PAGE_SIZE) +#define FIX_FDT_SIZE (MAX_FDT_SIZE + FIX_FDT_BSIZE) FIX_FDT_END, FIX_FDT = FIX_FDT_END + FIX_FDT_SIZE / PAGE_SIZE - 1, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9a7c38965154081e..757c2fe54d2e99f0 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1373,22 +1373,23 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) * allocate additional translation table pages, so that it is safe * to call create_mapping_noalloc() this early. * - * On 64k pages, the FDT will be mapped using PTEs, so we need to - * be in the same PMD as the rest of the fixmap. - * On 4k pages, we'll use section mappings for the FDT so we only - * have to be in the same PUD. + * On 4k pages, the entire level 3 fixmap only covers 2 MiB, so we'll + * need to use section mappings for the FDT, and these must be covered + * by the same statically allocated PUD (bm_pud). Otherwise, the FDT + * will be mapped using PTEs, so the entire mappings needs to fit into + * a single PMD (bm_pmd). */ - BUILD_BUG_ON(dt_virt_base % SZ_2M); + BUILD_BUG_ON(dt_virt_base % FIX_FDT_BSIZE); - BUILD_BUG_ON(__fix_to_virt(FIX_FDT_END) >> SWAPPER_TABLE_SHIFT != - __fix_to_virt(FIX_BTMAP_BEGIN) >> SWAPPER_TABLE_SHIFT); + BUILD_BUG_ON((__fix_to_virt(FIX_FDT_END) ^ __fix_to_virt(FIX_BTMAP_BEGIN)) + & ~((FIX_FDT_BSIZE << (PAGE_SHIFT - 3)) - 1)); - offset = dt_phys % SWAPPER_BLOCK_SIZE; + offset = dt_phys % FIX_FDT_BSIZE; dt_virt = (void *)dt_virt_base + offset; /* map the first chunk so we can read the size from the header */ - create_mapping_noalloc(round_down(dt_phys, SWAPPER_BLOCK_SIZE), - dt_virt_base, SWAPPER_BLOCK_SIZE, prot); + create_mapping_noalloc(round_down(dt_phys, FIX_FDT_BSIZE), + dt_virt_base, FIX_FDT_BSIZE, prot); if (fdt_magic(dt_virt) != FDT_MAGIC) return NULL; @@ -1397,9 +1398,9 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) if (*size > MAX_FDT_SIZE) return NULL; - if (offset + *size > SWAPPER_BLOCK_SIZE) - create_mapping_noalloc(round_down(dt_phys, SWAPPER_BLOCK_SIZE), dt_virt_base, - round_up(offset + *size, SWAPPER_BLOCK_SIZE), prot); + if (offset + *size > FIX_FDT_BSIZE) + create_mapping_noalloc(round_down(dt_phys, FIX_FDT_BSIZE), dt_virt_base, + round_up(offset + *size, FIX_FDT_BSIZE), prot); return dt_virt; } From patchwork Fri Nov 11 17:11:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E42FC4332F for ; Fri, 11 Nov 2022 17:14:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gZiZxkq3MB4SN3Kpyw74+Lad9LGVRvYX+/dBoV6zNik=; b=iqqPybsptlHZZO SU37YKAhfr7iv6KSNNfcyVR1t6dutkzufMzrczfpQ12517Yg/QI00xWXmaRKEBtVdTyh1h/nF3VFP UfXfbth1DyCH/KOSY+MgNXgYCuVieRv8YGZG3glr5xjlnl5m5foEShzQo8DL05C9PL7WkeUBrDXnR MxpLXADpRx0v7XQFNggOjU16UVqeeSV4QopCjJayUDp4PjpKZtsl+tgvjyZr7umgJSe8pAARpfWy+ HLIaa9Lq9TORqRxhD9KnZ3VimevNi9qh8OpoCEp/211G0S50S+KbIQgvRlDDRtKGczpoCEwWJO4Fs 9iEUUaLFfj8yF1sphlKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaN-00H7of-Cr; Fri, 11 Nov 2022 17:13:00 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZp-00H7Um-QC for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:27 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6B64FB8266E; Fri, 11 Nov 2022 17:12:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3C09CC43143; Fri, 11 Nov 2022 17:12:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186743; bh=gtt5pGHgRthVGPZRRIcJQzKvp+dB6EGtAB54jUbejCw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OwabbIkfduAipKggnrpEJZjr8C+HwjoThN9rumzg2HfMUz0ByZAKeqN9u7d2nhA4N zbEww8cm3f0FRIgiIqNSQHYcAGQckGOjUFbMkttyLNLrZb/4+MkDEa/Utlxe9Lu0zu 2uoTkU8uHGi6qu4yLmN9ldHQQikwZx2eAAwKgOaQ42DuOq+A3nsCmOXpyZyB6mXFHk IzctRzyfak9PrAzv5Q/JJRE8CRTPfCGlLwTOSf8Ph+PQq6yieGT7rKqQ9bYodXGP/c zUmNc1B2vD6MBG3rJ1ebrfrZ6X09OE5/51MYRT0ozLoQ4VQcT7zpGQPrqsct8bztf3 sxmHwlL2HdBSg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 02/33] arm64: mm: Avoid swapper block size when choosing vmemmap granularity Date: Fri, 11 Nov 2022 18:11:30 +0100 Message-Id: <20221111171201.2088501-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1399; i=ardb@kernel.org; h=from:subject; bh=gtt5pGHgRthVGPZRRIcJQzKvp+dB6EGtAB54jUbejCw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboIrLR09qep9dTEu83lqqtQkTD3Slfq7j5BFz470 FqKNis2JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CKwAKCRDDTyI5ktmPJOSWC/ 9YeLc2A4jH1IuBRg1oMSimuElHyiTkJYH/JKPbI7MS/3A2TNzsyyIi2wjpETPxl3irEbvKY26ppQRG 0yBZL4daM+IyRpKhSp3Et13pTfHT+Ji68Qv7VtkrGZjT4I/VZ5lBpPXbqC/Pv18w/bChaE0EExmocO 4oHyfT6Ed4Vlrbp+0tadbIbeWJccYmPD9GVzXSypVsZhrGHL3vopjCJaBzYCieAkc0P1wyEjvIAZbY OPNhXS/9ciG+Wa4ZZTxr1xSVViGCKP3IDd9DBG1WqiB5xn7dvnM2aQk+Y/vQmI6qUov3ar/cHGuqDF naXOyRep/9fFrvINKPQxm3FPbyK4MALTO5bBS2WbEj5AI+Vs2+M5f42UhlGC5G0/9FxmfjfnztLOoJ 1J+6e9KRasMcGKI16lbhJTJVkEH0jKCAfkkcvholZKiGH+L9AsftS63btyxX/IZ4VIg2LGZjes7Kd/ krLIOxzzFbJ5TVG7Ek6oexRo8CzhHYgLFMTEuRQ1/lSqo= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091226_048661_9C0D7097 X-CRM114-Status: GOOD ( 16.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The logic to decide between PTE and PMD mappings in the vmemmap region is currently based on the granularity of the initial ID map but those things have little to do with each other. The reason we use PMDs here on 4k pagesize kernels is because a struct page array describing a single section of memory takes up at least the size described by a PMD, and so mapping down to pages is pointless. So use the correct conditional, and add a comment to clarify it. This allows us to remove or rename the swapper block size related constants in the future. Signed-off-by: Ard Biesheuvel Reviewed-by: Anshuman Khandual --- arch/arm64/mm/mmu.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 757c2fe54d2e99f0..0c35e1f195678695 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1196,7 +1196,12 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, WARN_ON((start < VMEMMAP_START) || (end > VMEMMAP_END)); - if (!ARM64_KERNEL_USES_PMD_MAPS) + /* + * Use page mappings for the vmemmap region if the area taken up by a + * struct page array covering a single section is smaller than the area + * covered by a PMD. + */ + if (SECTION_SIZE_BITS - VMEMMAP_SHIFT < PMD_SHIFT) return vmemmap_populate_basepages(start, end, node, altmap); do { From patchwork Fri Nov 11 17:11:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A7E7EC433FE for ; Fri, 11 Nov 2022 17:14:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U/XGl5Brtkfz/dd75rCHildcbBwjlVfAd++6VyxcXZg=; b=fN0EXRJ3E5Cajz eQ2VqcYaDY2Df7wuOzUje6XhWk0GXF+INv7Ske/8IlZ4bpIzMqKTBlUxSWaQGu19aXxOROkcJ2POa kSbeh1+jskD3XUCM+lygHevoI0+hd3e1IXU2VVhfbLiXUJozD65UDk9VJK/Za2WWOSvCa1Bv9kyCx Icdf8ThZpabRFjvVcPYWY4yge9b5uqUF96/3emr2YJLqftFgLaguRl/WLroRKwgvZtRcV/pSESGr+ 3lJpgIQ7v1S5ycV6RNOMbVxmwOEhFJcq6lrIyu7W1Kio6DKPy+nXofYHVvmMiNKFfuKS0SLsZ4UZS ABrwtq4+3tqhF6G1BGxA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXb6-00H8Ee-6C; Fri, 11 Nov 2022 17:13:45 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZr-00H7Vq-U9 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:29 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id AB194B82680; Fri, 11 Nov 2022 17:12:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83C1CC4347C; Fri, 11 Nov 2022 17:12:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186745; bh=rtWfJTnP56vx45KlZMlZ84CRl/GSmeWPFWoZF91SgKU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VZYQ/ZT4/tjejSpLEaOutbaH+jNzRjDBkaPSHN70s0DK6zOkQdAMF3pDceyoOoHwG PGmsGOsvIWIgOBCzeeIZTK0HLH0pDw3jtPpE91qP6Sx5l+8+jvAP2PkERXKstbgUjA HRX2cLEaRNZVREZl01CYqYY5DkMH6MBpoViLXjSVsxemTmc457iVa9rERPJkY9zCVa KB20DW77PN/mHjpaIC5uLBkumh98yPCIEXBX0Y1yRYdZ6hswFOzr44weAmO5iRGGty FvahS36aE+eeqdOoY23ixi/C+qGUj2cr1sLhcQGlqI/mPx679uvf87zvJUnZjDm6d9 8ZQDc3NHOsSAQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 03/33] arm64: kaslr: don't pretend KASLR is enabled if offset < MIN_KIMG_ALIGN Date: Fri, 11 Nov 2022 18:11:31 +0100 Message-Id: <20221111171201.2088501-4-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2489; i=ardb@kernel.org; h=from:subject; bh=rtWfJTnP56vx45KlZMlZ84CRl/GSmeWPFWoZF91SgKU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboItzobCf7wyIOvNDz4K+zMMTFk3ZHQngQxTSu7U Lv3WMaqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CLQAKCRDDTyI5ktmPJL4QC/ 4yTJjuNNzgBwdGVqJCJV0IUJ7CmJQ22bdiMOYrs6p7f5xNZ3Wzj6F4Dhjq4U2OfMuyFL+uJF8BmDO9 SjlBqsNzKJb9HZ3ZLiNTuteKxCAr7+VyU8pvTRRKVofjbj3ZQqiqkhgus6jcXBfhuHJEpGHOMysEgR p6ECiX/9sL/NYNXNDjw/TpFhn0kilu60luvssHQmaIAlg+eq/hwipYTdwMbMcMRONTdcwx+QUGyP9n pzHGcSiMQcYMm+HCUNSRTEWCcsrF/xEWYT6cOZAKRSwL0sMW8Nym/FEFQ8SERcBwGZKjnlbmwJjPmM jixlEmSxri6b9RcC9kSWiu6MdWP7gYA+MTuxTVIQ0hoafDAfD1yVdc8Vg2K1gNCetXXhsdyIagz/Y2 ecZM+ALnZiwQlan2X9fam5Zl2uKkZ5eSAKtjANePaMAD2mujDOZQiYXWrHMg3G2wb2b8mTqfqVEOHc HLN86v9XkB+tdeMNnBFKOK++e+Kp2EsECeGQjz+TGGbq8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091228_308607_F9D92323 X-CRM114-Status: GOOD ( 18.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Our virtual KASLR displacement consists of a fully randomized multiple of 2 MiB, combined with an offset that is equal to the physical placement modulo 2 MiB. This arrangement ensures that we can always use 2 MiB block mappings (or contiguous PTE mappings for 16k or 64k pages) to map the kernel. This means that a KASLR offset of less than 2 MiB is simply the product of this physical displacement, and no randomization has actually taken place. So let's avoid misreporting this case as 'KASLR enabled'. Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Brown --- arch/arm64/include/asm/memory.h | 11 +++++++++++ arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/kaslr.c | 2 +- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 9dd08cd339c3f028..78e5163836a0ab95 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -180,6 +180,7 @@ #include #include #include +#include #include #if VA_BITS > 48 @@ -203,6 +204,16 @@ static inline unsigned long kaslr_offset(void) return kimage_vaddr - KIMAGE_VADDR; } +static inline bool kaslr_enabled(void) +{ + /* + * The KASLR offset modulo MIN_KIMG_ALIGN is taken from the physical + * placement of the image rather than from the seed, so a displacement + * of less than MIN_KIMG_ALIGN means that no seed was provided. + */ + return kaslr_offset() >= MIN_KIMG_ALIGN; +} + /* * Allow all memory at the discovery stage. We will clip it later. */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b3f37e2209ad378f..ded7684b0a304edc 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1620,7 +1620,7 @@ bool kaslr_requires_kpti(void) return false; } - return kaslr_offset() > 0; + return kaslr_enabled(); } static bool __meltdown_safe = true; diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 325455d16dbcb31a..e7477f21a4c9d062 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -41,7 +41,7 @@ static int __init kaslr_init(void) return 0; } - if (!kaslr_offset()) { + if (!kaslr_enabled()) { pr_warn("KASLR disabled due to lack of seed\n"); return 0; } From patchwork Fri Nov 11 17:11:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3975EC4332F for ; Fri, 11 Nov 2022 17:15:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UsCXdbOofP24aJtO/vQswzAStycMRXlatyu8G1a7Fis=; b=PrzsUyJo36l2dW uuGNYweGgDSs1r/4ru+kwGK6jiXSO0Z7JhOB7d1BvlVnI5dz9+xCL1Ew15IqURw6EQ2I4orNA2Xwh BDJyVokXtxnUpaItWepmeZloQnRZ5/tadqL2jYQgR9vRm83ccj9kLCyCFgknwj4BNiAw/QANSan1x UkJpHnb0UGt6CcxZOOvkyjuMLoM2E1H3KmIcslAiBKHsTB/c38mykfKA8OlRGrWicU2cpV/FlkkVe uoeb+eoWdbsUrKBftatujpIT4MQSnz6G9k0PtpqimRh3ED+dnjNUHFaTmSG1JeOgEQoIEorFFeT2n NaZyJwpIJaImbL4Dqu1A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXbd-00H8WB-Po; Fri, 11 Nov 2022 17:14:18 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZu-00H7XM-80 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:31 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F1A77B8266E; Fri, 11 Nov 2022 17:12:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CABB8C433C1; Fri, 11 Nov 2022 17:12:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186747; bh=USSZOsJFDOvXgcBT4K5g9iMqI5O9MK9akV3y6MlIp1c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SJhZJMBFxfaqpCzbLEY+0c9ZHtF/Cl6ZpJD5tjaoW+X4ijb9lJpBcXWXVhDZyKDml OQPxdk0qgh5v8+t+KkkmVpvi8zl2CMoYsXzvSC2/4bmAXxhffzzrFiFuz2V+0XiuRo d9F6DKunCu7lxi1VlYdngVrEmU1P53yz+3ia9QW2OQiE2YLL50CPTmsnhv5b8zD4IS fc6T1RNB78c89GORAwYyd7KnEsTLyMkkK4qAyTm/A7ql0v9X1E51ZJMpt+TcFQfDAA L5OXNALCXW6gPSqzqX0umKdV6E2N+oI3bAtMFaldqj9ZPvWRkFoK1l6d6OpwhnmT7T OZq1/vS924/ug== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 04/33] arm64: kaslr: drop special case for ThunderX in kaslr_requires_kpti() Date: Fri, 11 Nov 2022 18:11:32 +0100 Message-Id: <20221111171201.2088501-5-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1384; i=ardb@kernel.org; h=from:subject; bh=USSZOsJFDOvXgcBT4K5g9iMqI5O9MK9akV3y6MlIp1c=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboIvz1I86NW1pYDTZJAk8iHtX1JB1uEOHQsVJ9p8 0avLSPaJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CLwAKCRDDTyI5ktmPJGxWC/ 9cfwPnCC7kW2WG8ZPhSVI/VG8V5lBGPC+porm+EMJ5NsvjBskVxiqHvAtP1h8WBY8pKKnvEfOICF12 p/+/SHcqMaLZpziRIRgL/nnTHflQEQ+C1nJ7qCPSBizWngRxmLiAI8GOr7K2+Azn9PdY9+EeeqYwmc VnShYCs8YGBo2nuAX0X1p5xK9AbLk0MD/g5pzsl4NBWbeF4sHA9kVvuj15Psd45WwnjZmUhFg3DLKg zWzQoNEzrLMMzxnLScMCubvI7Zz+QLKjVlu3dCk5dTsLLqhnZDEVWwO4WUt7XwUIIy+u2Eg7lBjAIK suP7ydiRAK8h8TG24CO/UR8eAr/w7ntQSCu7Yfj15iR7LWHIgQ4bxPSjGPldxU28zcMqQL4SROdsKj z9aYKeOx+NVNAZI/yLKqdVyl2jH6IM65byEI+9IeRhtVFhoKiccXNnejxieK8ZrGxG2Ck7PryY15W7 MDkyHSngphZwMvJzI3ihDR9lT4nwqRRxKvzI7EGKUSO+0= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091230_456508_D9A9CCE7 X-CRM114-Status: GOOD ( 13.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org ThunderX is an obsolete platform that shipped without support for the EFI_RNG_PROTOCOL in its firmware. Now that we no longer misidentify small KASLR offsets as randomization being enabled, we can drop the explicit check for ThunderX as well, given that KASLR is known to be unavailable. Note that we never enable KPTI on these systems, in spite of what this function returns. However, using non-global mappings for code regions is what tickles the erratum on these cores, regardless of whether KPTI is enabled or not, so non-global mappings should simply never be used here. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ded7684b0a304edc..fdbae2320b466d98 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1608,18 +1608,6 @@ bool kaslr_requires_kpti(void) return false; } - /* - * Systems affected by Cavium erratum 24756 are incompatible - * with KPTI. - */ - if (IS_ENABLED(CONFIG_CAVIUM_ERRATUM_27456)) { - extern const struct midr_range cavium_erratum_27456_cpus[]; - - if (is_midr_in_range_list(read_cpuid_id(), - cavium_erratum_27456_cpus)) - return false; - } - return kaslr_enabled(); } From patchwork Fri Nov 11 17:11:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0B8CCC433FE for ; Fri, 11 Nov 2022 17:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8iaQcNfQ6F8raKaCWyJkXYGzQX3Zd8RXUxIQmh1b0Pc=; b=Rj2dC/8AXQm0rR M5gsa38XA0ODW1o1BKx88U9ez0di/W8ltfueNoHJrpEDDICTONebCEZNLXKPSPBjGo9C6bqmyzaoG FA/+GJYI0wBdUtj/kPVsBxuYeVa8qPqdfszsWh556qItZ7YZjcUZ9EmmYFFd/GGmdp/bAuJu3GKgW aZHGnnNTQBsalhDsXqVX5Ni1MJR+qExvJduBi+QkMh2fe+csrFyH8VWmwQ50K8GWhf/BMpSJsEidh mR4yk0UvSTic/cVRPnXkiITB0c8iZphwMYZ038rjm4BnAUjNRQzL2RdCMdZmoA8da77c62y9y7aO6 v6BAV8xF0at2MfUu0fcg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXcH-00H8om-0C; Fri, 11 Nov 2022 17:14:57 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZv-00H7YK-6l for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:32 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8F35F618C9; Fri, 11 Nov 2022 17:12:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D70DC43143; Fri, 11 Nov 2022 17:12:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186750; bh=rZmP7BIoj1J491MBF4A3/sy3itpwJoiIMNgf5sj7MjI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HBju7epUAPaaaL5hzdtNN5stS038EvAfgde35I3WyVOJwNzOHRmMV7accMQ9ubgLr 8yxsx3os8kfAX0ptgvM3CqvEDAvPypKK7CuDdJvmBKbKVRdBPCAASB/+L/VkjCvHuk SnSNGvVILEFxYMra1vBQMjIFbzZ20CU+howeQTRNp9aKTPQXi+Fih52Qidr05GXfz8 qvBuAl1+lAuQrN1UNBX3B8CQO80lgQvn3ItOuCL6NUWPs8zn1xNMPdqmknAH7dB1Lp 5AQEB0jegSkRnzpoEXpqOFcUlnVKo0+lEdSnKiw6T9xCvvdpQvPzzV5hkNzebYlqM5 lUvNGk2uFp0RA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 05/33] arm64: kernel: Disable latent_entropy GCC plugin in early C runtime Date: Fri, 11 Nov 2022 18:11:33 +0100 Message-Id: <20221111171201.2088501-6-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=849; i=ardb@kernel.org; h=from:subject; bh=rZmP7BIoj1J491MBF4A3/sy3itpwJoiIMNgf5sj7MjI=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboIxNlQuGKin0p0gYlZsDdmLvROOtJgb3iz1VepF WjbfNzuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CMQAKCRDDTyI5ktmPJPaBC/ wJ9WUQBKnfYpN/76TQy7dYSwE4hUmj/KadHBfFe0ur+8+UGsD6c5ALSxWVM3NXGLb6TJcF7Sjn7Pok xlW7cf0k2Qk1JChFYK6y1jQwJHNbdRzIj+E60B8yxUtoVTnjapxHwONoT1Gv4FClwrCoEyNWRYlJEG yrPgJE/nj/BZsFb+IxYRUiQszWhEhqfHK+Z+X16U08XC7vW4eUhKBRD3Vk9eaoQRAZnebPf6SLjEdE GMy8PBqex2jFPNt6sEls+Yx5lm0CiEd+KuCfBjOreZ9xiIrY6fy9KVgN9fdTDOIvpQbxxOCVLNlmhY QrNOryhRN36nyAk0CBxsDFG9hsk2Xw6pSocte+mM9hXVpTlKnFvZHqzMWwZiJ4IJJttbQK3UBjmXdy o5kfTs7Pw71/vXRME0ZoXJJPRxe6yevUe/eOacan+gVrFysJZQgdSaAIAH3At0NR3obr6um8VrDLsO ETpcOp24vThbk4c5l+knIfHKwI/ysrgaCGl9NLbaSB2DE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091231_334453_7E95DE70 X-CRM114-Status: GOOD ( 10.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Avoid build issues in the early C code related to the latent_entropy GCC plugin, by incorporating the C flags fragment that disables it. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 4c0ea3cd4ea406b6..c844a0546d7f0e62 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -3,6 +3,7 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -Os -DDISABLE_BRANCH_PROFILING $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_LATENT_ENTROPY_PLUGIN) \ $(call cc-option,-mbranch-protection=none) \ -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ -include $(srctree)/include/linux/hidden.h \ From patchwork Fri Nov 11 17:11:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCABBC4332F for ; Fri, 11 Nov 2022 17:16:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CtlkyVy0/B7uLY0H0qdwLlf7Go/W2y7qqeHTfHh8t6Y=; b=lZfIqTYXyRQaz4 xMVsTvROwfyywpTY08NILBJ4bBLnK8dP1Pigh4JxdJos/0brULP8oYdnJHLV7PuSCbQncUj6oweoB gpKVDD/dAyFY0RX29PVn0bqiLRtHt85dZu2aA3/wZbcJctcy+gANZo/VG0OX3bPFP+UUgBDXbvIWp pPvjdlbx2qvkPwybtteQ8hAoLAvSjPGpNz1L5UMdTNdimIt4x6hAn1vqOEkt9QqaVs1i2FJ0PI46q +TUA4nWlx6gzUk6iIBm1teMPk2op8Zj3UVt/WkfVIlAWOhZonzTpawW1XCeNDFPrV6Zu/LJiWdbdG kHO4tb9pHOzTmtE0wv/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXcb-00H91m-Dd; Fri, 11 Nov 2022 17:15:17 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXZy-00H7aX-VE for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:36 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 98B4DB8267B; Fri, 11 Nov 2022 17:12:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6546BC43144; Fri, 11 Nov 2022 17:12:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186752; bh=2jU7X7NRPByZdbNZ3bzi7gLzJ6OfyCcumL153oY3F70=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RgfwcoOF4taEQI2BJ1fINBYidYjB/mhuhOfB1lNxrY7nn8i/WCD87XVbOjVvRobaX OQG6zgzRJ3aI332TxRXhdZD+QZ23DHUXCm2dqbjIJK8kYX+q+toqypHwX45duP7ZoK 9XGGC458kmJbbXXuGYEBkvdebMbeAl850aNqR4p8Pirti1WWw0F41WzAq0L1sZKkXE uAD3PC9z0YXt3/yqezhWHCR+0HoMAgtQDxSIkRHd8hUsLRR8zIGxwj4Nz+e+X/T9NY 2J+8ZqV05CsH6qfMOtT1j79KYl+TzEP7l25+tReyaCATiQeVWcHnY0gBNFiCKyp/3D P7WlY1B1Rpd2w== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 06/33] arm64: kernel: Add relocation check to code built under pi/ Date: Fri, 11 Nov 2022 18:11:34 +0100 Message-Id: <20221111171201.2088501-7-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4367; i=ardb@kernel.org; h=from:subject; bh=2jU7X7NRPByZdbNZ3bzi7gLzJ6OfyCcumL153oY3F70=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboIyMJnk4unNMOMK4fmGHAm3Zl3TOjD+rLjxKdW5 hidCPc2JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CMgAKCRDDTyI5ktmPJFYEDA CKFf9730UNrMceTDaML8/kCqML3ZKk3733fljheQrZBQlQjvsn4xYXy46mhSmLUaAKU9PjRjC+dnd3 2vduRIhL5Ne5ZJE45Oy/39uZsM+k7LLTbcEpCVbN10/NzXrhJy5Pe8xEd/tU6loqhN89rQIdsWOR5X Ox3VcmGRK/zJ4/VTRcZM2ryhCRWW7p9hyHJiTm+t8F6pYz0Amh3WD1q/pQa0SaJzmW9AOY64Dn4jM3 bAjIdRvAMmydQhH0VzmBzfxRZv+0xhZM8ZNM/EjmK7zLUkDN+xILhPvSLDSmX3q791Llpyv15NkhFW jY4nWDlbz7wWK1pdmKFjzVDeV01cVbE7BHyVWhl0/BffY4/vt3d3XQcE/C0yLo8X7YrR0ELFVtp9+B ujsLRDDatowwzpsf8dhmUlZp+M7biFIvVGY6Bcyld604Tx00nzVFhrOf9WgQdMDWODrYhh+s07TJ3R nG5UarV5DvDF1w9pN43zzyrLGB6PPjWFPsxOkT+3XvW3A= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091235_352041_8DC1A879 X-CRM114-Status: GOOD ( 20.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables. Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object that operate on data sections that are part of the executable image, and raising an error if any relocations of type R_AARCH64_ABS64 exist. Note that such relocations are permitted in other places (e.g., debug section) and can never occur in compiler generated code sections, so only check sections that have SHF_ALLOC set and SHF_EXECINSTR cleared. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/relacheck.c | 104 ++++++++++++++++++++ 2 files changed, 111 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index c844a0546d7f0e62..810fdae897601e88 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(obj)/relacheck $< && $(cmd_objcopy) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/relacheck.c b/arch/arm64/kernel/pi/relacheck.c new file mode 100644 index 0000000000000000..1039259360c735d2 --- /dev/null +++ b/arch/arm64/kernel/pi/relacheck.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 - Google LLC + * Author: Ard Biesheuvel + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define HOST_ORDER ELFDATA2LSB +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define HOST_ORDER ELFDATA2MSB +#endif + +static bool swap; + +static uint64_t swab_elfxword(uint64_t val) +{ + return swap ? __builtin_bswap64(val) : val; +} + +static Elf64_Ehdr *ehdr; +static Elf64_Shdr *shdr; + +static uint32_t swab_elfword(uint32_t val) +{ + return swap ? __builtin_bswap32(val) : val; +} + +static uint16_t swab_elfhword(uint16_t val) +{ + return swap ? __builtin_bswap16(val) : val; +} + +int main(int argc, char *argv[]) +{ + struct stat stat; + int fd, ret; + + if (argc < 2) { + fprintf(stderr, "file argument missing\n"); + exit(EXIT_FAILURE); + } + + fd = open(argv[1], O_RDWR); + if (fd < 0) { + fprintf(stderr, "failed to open %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ret = fstat(fd, &stat); + if (ret < 0) { + fprintf(stderr, "failed to stat() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + ehdr = mmap(0, stat.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (ehdr == MAP_FAILED) { + fprintf(stderr, "failed to mmap() %s\n", argv[1]); + exit(EXIT_FAILURE); + } + + swap = ehdr->e_ident[EI_DATA] != HOST_ORDER; + shdr = (void *)ehdr + swab_elfxword(ehdr->e_shoff); + + for (int i = 0; i < swab_elfhword(ehdr->e_shnum); i++) { + unsigned long info, flags; + const Elf64_Rela *rela; + int numrela; + + if (swab_elfword(shdr[i].sh_type) != SHT_RELA) + continue; + + /* only consider RELA sections operating on data */ + info = swab_elfword(shdr[i].sh_info); + flags = swab_elfxword(shdr[info].sh_flags); + if ((flags & (SHF_ALLOC | SHF_EXECINSTR)) != SHF_ALLOC) + continue; + + rela = (void *)ehdr + swab_elfxword(shdr[i].sh_offset); + numrela = swab_elfxword(shdr[i].sh_size) / sizeof(*rela); + + for (int j = 0; j < numrela; j++) { + uint64_t info = swab_elfxword(rela[j].r_info); + + if (ELF64_R_TYPE(info) == R_AARCH64_ABS64) { + fprintf(stderr, + "Absolute relocations detected in %s\n", + argv[1]); + exit(EXIT_FAILURE); + } + } + } + return 0; +} From patchwork Fri Nov 11 17:11:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1F60C4332F for ; Fri, 11 Nov 2022 17:16:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C0pbF1h7/iB2P7a5JjbQA4s5kqHT8mOLxldA8U0HWXU=; b=okH5V7gIwF9jrZ L/akxypNG3QZ2kZK4+oWiAJEP9Md/bmIwroycQJEljTf9aOImFnvkzTqeuDtZu1x8Ud2uoZSN0/WL 2dPu5FsxJyPMgH41Lw3R2PcZP+AIETuae6U98prWvhGgdIRw2PVYQfHv56f5YwtGm5U9FY/G0oYBm lJGQvT6hOs31whbgVlgiSkemcgB6yphP9bWylske5IEYzb9uM0IDu+aQR/J6KkTDjRedr0C6VtkE7 ujrBjo5SRFc199V/jL4IzeFFRGG22b7Hpe1G3RFLB/YHGixR5IsAkYV2xGEERbqpbjH4cfYrbrHg9 qOvV0hO/9iuQtMIBF/Jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXd2-00H9IL-TJ; Fri, 11 Nov 2022 17:15:45 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXa1-00H7bO-CS for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:39 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DB263B8266E; Fri, 11 Nov 2022 17:12:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC78FC4347C; Fri, 11 Nov 2022 17:12:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186754; bh=wcHghYDG8a1xGzC6B9oPbu4QgDzp6nE0raGQNLcDOmA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KqjjZjJ5oNf+wmSHfLkwZk0F+w6vmqLYZZWXUBmdduxEk/xWhjxEQuryhiADCrIm1 CnF4QIHaRE3a6ZR1yZG4Ovk86D3Np7D+wFYLfhF/CPZyYWwepDfPm+2lL8djn45rj7 yByUxr/wc/B88BlMzoWXG0gNlBPef1Ljr6MNYpxHsWFBcUZUN+ZP3zgJO35GmziQIq 3vb1OJZlZfZeiig4cBmJEfRzamV6Z+Ti/pK8+7TutVN4Hy/2Lu0EGFVPCwbvLi9V2V TwuRxOpwdVdJt9RMseeApaFQZyUNFm8v3S4ttfP1zR9D4cSb1JOg0PnV4ECuxg+8/T Dx9sPjAKsF2sQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 07/33] arm64: kernel: Don't rely on objcopy to make code under pi/ __init Date: Fri, 11 Nov 2022 18:11:35 +0100 Message-Id: <20221111171201.2088501-8-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3191; i=ardb@kernel.org; h=from:subject; bh=wcHghYDG8a1xGzC6B9oPbu4QgDzp6nE0raGQNLcDOmA=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI0oB7buSyROeyNn4l+9zAXktPfPP4iGyVDR3qM Ysr6NxqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CNAAKCRDDTyI5ktmPJHyiDA C6K1kep2IvhWjqB2tuKkwRFnfBt1FZwhzEqp1P8mnN6asEPOYGRF/64OKrbSQqbx1cZYmDFVuXYE+T SOfsnM393PTo7IbaXhkxul7ae6X6Lv7BiQeD83pOFk62Y3vEpxpNdExJltNSKM1UJsYgPbcR6xrZSE q1ZZzu/VHStmbzGYjFM+WWx6sW5wy0bKmXPh+Y5NcwT5+SQXbpyZU+UQqrjhal5itl8Up/DWtIwFXu AuSNmRYp2Q/zYwlSD03Q41lb0XDmYBqbKsFzFg8niRVCsscQDzmbcmsO3tFaUMxIz7lMD7FhQQM7AV 9SONCmZPisoeRER9yDR/BmgtT+hI7z0xYH0MWAeggxM3lOjsy6jeeepc7JMX73FEIg871WdwubELTq Y2JMA3YlcMAFcKfhPMSFjsuqiWTsU+Tq/hiqYeQzr8OXDdmPedsTv8eoYo228gP8VPZOTVkhZ8rYLU N6gG1mfZY03eia6BaKRWEEIdNX05H6819/L1FBvJ8AWXc= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091237_764204_2FD59073 X-CRM114-Status: GOOD ( 13.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will add some code under pi/ that contains global variables that should not end up in __initdata, as they will not be writable via the initial ID map. So only rely on objcopy for making the libfdt code __init, and use explicit annotations for the rest. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/Makefile | 6 ++++-- arch/arm64/kernel/pi/kaslr_early.c | 16 +++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 810fdae897601e88..8aaf7cbac359ecdb 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -28,11 +28,13 @@ quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) cmd_piobjcopy = $(obj)/relacheck $< && $(cmd_objcopy) $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ - --remove-section=.note.gnu.property \ - --prefix-alloc-sections=.init + --remove-section=.note.gnu.property $(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE $(call if_changed,piobjcopy) +# ensure that all the lib- code ends up as __init code and data +$(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init + $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 17bff6e399e46b0b..86ae0273c95016c6 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -16,7 +16,7 @@ #include /* taken from lib/string.c */ -static char *__strstr(const char *s1, const char *s2) +static char *__init __strstr(const char *s1, const char *s2) { size_t l1, l2; @@ -32,7 +32,7 @@ static char *__strstr(const char *s1, const char *s2) } return NULL; } -static bool cmdline_contains_nokaslr(const u8 *cmdline) +static bool __init cmdline_contains_nokaslr(const u8 *cmdline) { const u8 *str; @@ -40,7 +40,7 @@ static bool cmdline_contains_nokaslr(const u8 *cmdline) return str == cmdline || (str > cmdline && *(str - 1) == ' '); } -static bool is_kaslr_disabled_cmdline(void *fdt) +static bool __init is_kaslr_disabled_cmdline(void *fdt) { if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { int node; @@ -66,17 +66,19 @@ static bool is_kaslr_disabled_cmdline(void *fdt) return cmdline_contains_nokaslr(CONFIG_CMDLINE); } -static u64 get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt) { + static char const chosen_str[] __initconst = "chosen"; + static char const seed_str[] __initconst = "kaslr-seed"; int node, len; fdt64_t *prop; u64 ret; - node = fdt_path_offset(fdt, "/chosen"); + node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; - prop = fdt_getprop_w(fdt, node, "kaslr-seed", &len); + prop = fdt_getprop_w(fdt, node, seed_str, &len); if (!prop || len != sizeof(u64)) return 0; @@ -85,7 +87,7 @@ static u64 get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 kaslr_early_init(void *fdt) +asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed; From patchwork Fri Nov 11 17:11:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040628 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3ED94C433FE for ; Fri, 11 Nov 2022 17:17:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jJXu6j+vXJBYRce1PVgAyAX2Y3Od+daDBCvsWHuYTAY=; b=wMLai2M5Vysr+a xA+/5zjqgAquDtwR6FpVjGmlS+hsKYqWtBKwY+FvmBmEeHSM97UvQm1Nmb3O8IxbDPcUPJtqvRxEb cw/sDgnx0gX0hOLlHM/0tXNDL8On3aUq4Q9roEnE5Be4e3xQDgPczpu2wBGG5sQD4BsKhIprEtBab UGDp2DCIewYT/hYKZBBx6JvPpaWAACLOzlQr9lOAjis6/eXbNDzKKMumZdeOIg4FpDnwxzkXY0gEA Z67/kP+RREwLg5oQAI1Qxh+38qcKJSJ1suFaNOrvAuXc4fcu1B2p0MrfKQtzIikj8nXQANv+beWhw ugIpYg7RLjw3E672MrQw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXdQ-00H9Xl-7R; Fri, 11 Nov 2022 17:16:08 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXa1-00H7cR-Sv for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:41 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6E80C62068; Fri, 11 Nov 2022 17:12:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 00F0FC433C1; Fri, 11 Nov 2022 17:12:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186756; bh=nIB5MPg6Fo8fAH1sZHrQptxJ6lkWooOxq9R+Mr0Ms+4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JTjuyO95DQyDCcqvfzUUZ7KlC7tvTKtGRFwRoRl1K0wRT5Fs14xy5KFHQGBOdhfAk 6HuEqz9yzc5v7tISInith2xzQYKo9a75z8hMAzUs/1im45Vg4r/SHXLMD0pq5re102 fRjL4GPE6zoDL/jNH2ptbQfWFm/MqIQoCkovindKmepoLfwnUyxVUipjv06Xg91aZ+ 08enmLjlmWM8BhjOAvkrNGDy26Oc3ieCfalz+kFGydouR6CLtlSYe81WLca8e1QxAa aVgfLVcIGX9C+bNaZEGWuCsaUQN7jZxJGWWkJdzI6RVfiKkQtrXeoysd3LdZXyn4RN WevkOwjT7VJLg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 08/33] arm64: head: move relocation handling to C code Date: Fri, 11 Nov 2022 18:11:36 +0100 Message-Id: <20221111171201.2088501-9-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10061; i=ardb@kernel.org; h=from:subject; bh=nIB5MPg6Fo8fAH1sZHrQptxJ6lkWooOxq9R+Mr0Ms+4=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI2ZjsbLlWIxAQUSHjDLP9wC0JVBT+DcM+PaGji SCz1X4GJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CNgAKCRDDTyI5ktmPJN62C/ 9ZagmfeJ51bgw5yS6kjMUclC53cQ4Z0V8buZgWlwoyvyS/XnNSxbVGpuOm5TolIZVL4OltIWntnAym rRsgcDrBxHMjAuwWwteAhDh4aw32O5CdAmhQKoANs5Wz3mtfizI7RDN1UKAdq6oval7R1/qCsLlUpd qPLeLy7rqxoo/4rwqZHqMLd1rckRFunutmLdoKCf2/zuTqFLUaE39SKi2mSf2C4XM5x8zSkGEpoxbp O7hIVGgbGB3bO3180CSRGI/+iPT9ULf99yXd/AlLHwC84GmFPmTloaIJbH9yKSqJo3cPHlEgcZ1/Tu 1JcZJmrbMO6PkH9RKYbgFGZPpPjuLvCJsmMubfsnxbFcN2HK0/np+DvoXoKR5fVvAUh8XDNH7fx112 9/4P6bZbtJQSO671zIp5syUqcmSIjBpEVU6pgP559J9yrSdbgp5FR918gVfHj3cEhZwpf+QsF06uat MmXffTQEY+kpd9UUewB8BkBpsb/ZHt96/CQBwhJD2RmAk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091238_098838_843D83AB X-CRM114-Status: GOOD ( 28.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that we have a mini C runtime before the kernel mapping is up, we can move the non-trivial relocation processing code out of head.S and reimplement it in C. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 3 +- arch/arm64/kernel/head.S | 104 ++------------------ arch/arm64/kernel/pi/Makefile | 5 +- arch/arm64/kernel/pi/relocate.c | 61 ++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 12 ++- 5 files changed, 81 insertions(+), 104 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 8dd925f4a4c6d29e..a8717865fee5c296 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -65,7 +65,8 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ +obj-$(CONFIG_RELOCATABLE) += pi/ +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \ diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 952e17bd1c0b4f91..998a3e066b2fdf0a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,7 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset + * x23 __primary_switch() physical misalignment/KASLR offset * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register @@ -338,7 +338,7 @@ SYM_FUNC_START_LOCAL(create_idmap) /* Remap the kernel page tables r/w in the ID map */ adrp x1, _text adrp x2, init_pg_dir - adrp x3, init_pg_end + adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov x5, SWAPPER_RW_MMUFLAGS mov x6, #SWAPPER_BLOCK_SHIFT @@ -705,97 +705,6 @@ SYM_FUNC_START_LOCAL(__no_granule_support) b 1b SYM_FUNC_END(__no_granule_support) -#ifdef CONFIG_RELOCATABLE -SYM_FUNC_START_LOCAL(__relocate_kernel) - /* - * Iterate over each entry in the relocation table, and apply the - * relocations in place. - */ - adr_l x9, __rela_start - adr_l x10, __rela_end - mov_q x11, KIMAGE_VADDR // default virtual offset - add x11, x11, x23 // actual virtual offset - -0: cmp x9, x10 - b.hs 1f - ldp x12, x13, [x9], #24 - ldr x14, [x9, #-8] - cmp w13, #R_AARCH64_RELATIVE - b.ne 0b - add x14, x14, x23 // relocate - str x14, [x12, x23] - b 0b - -1: -#ifdef CONFIG_RELR - /* - * Apply RELR relocations. - * - * RELR is a compressed format for storing relative relocations. The - * encoded sequence of entries looks like: - * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] - * - * i.e. start with an address, followed by any number of bitmaps. The - * address entry encodes 1 relocation. The subsequent bitmap entries - * encode up to 63 relocations each, at subsequent offsets following - * the last address entry. - * - * The bitmap entries must have 1 in the least significant bit. The - * assumption here is that an address cannot have 1 in lsb. Odd - * addresses are not supported. Any odd addresses are stored in the RELA - * section, which is handled above. - * - * Excluding the least significant bit in the bitmap, each non-zero - * bit in the bitmap represents a relocation to be applied to - * a corresponding machine word that follows the base address - * word. The second least significant bit represents the machine - * word immediately following the initial address, and each bit - * that follows represents the next word, in linear order. As such, - * a single bitmap can encode up to 63 relocations in a 64-bit object. - * - * In this implementation we store the address of the next RELR table - * entry in x9, the address being relocated by the current address or - * bitmap entry in x13 and the address being relocated by the current - * bit in x14. - */ - adr_l x9, __relr_start - adr_l x10, __relr_end - -2: cmp x9, x10 - b.hs 7f - ldr x11, [x9], #8 - tbnz x11, #0, 3f // branch to handle bitmaps - add x13, x11, x23 - ldr x12, [x13] // relocate address entry - add x12, x12, x23 - str x12, [x13], #8 // adjust to start of bitmap - b 2b - -3: mov x14, x13 -4: lsr x11, x11, #1 - cbz x11, 6f - tbz x11, #0, 5f // skip bit if not set - ldr x12, [x14] // relocate bit - add x12, x12, x23 - str x12, [x14] - -5: add x14, x14, #8 // move to next bit's address - b 4b - -6: /* - * Move to the next bitmap's address. 8 is the word size, and 63 is the - * number of significant bits in a bitmap entry. - */ - add x13, x13, #(8 * 63) - b 2b - -7: -#endif - ret - -SYM_FUNC_END(__relocate_kernel) -#endif - SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir @@ -803,11 +712,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - adrp x1, init_pg_end + adrp x1, early_init_stack mov sp, x1 mov x29, xzr +#ifdef CONFIG_RANDOMIZE_BASE + mov x0, x22 bl __pi_kaslr_early_init and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 @@ -820,7 +729,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, init_pg_dir load_ttbr1 x1, x1, x2 #ifdef CONFIG_RELOCATABLE - bl __relocate_kernel + mov x0, x23 + bl __pi_relocate_kernel #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 8aaf7cbac359ecdb..e046c10606cb822e 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,5 +38,6 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c new file mode 100644 index 0000000000000000..c35cb918fa2a004a --- /dev/null +++ b/arch/arm64/kernel/pi/relocate.c @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2022 Google LLC +// Author: Ard Biesheuvel + +#include +#include +#include + +extern const Elf64_Rela rela_start[], rela_end[]; +extern const u64 relr_start[], relr_end[]; + +void __init relocate_kernel(u64 offset) +{ + u64 *place = NULL; + + for (const Elf64_Rela *rela = rela_start; rela < rela_end; rela++) { + if (ELF64_R_TYPE(rela->r_info) != R_AARCH64_RELATIVE) + continue; + *(u64 *)(rela->r_offset + offset) = rela->r_addend + offset; + } + + if (!IS_ENABLED(CONFIG_RELR) || !offset) + return; + + /* + * Apply RELR relocations. + * + * RELR is a compressed format for storing relative relocations. The + * encoded sequence of entries looks like: + * [ AAAAAAAA BBBBBBB1 BBBBBBB1 ... AAAAAAAA BBBBBB1 ... ] + * + * i.e. start with an address, followed by any number of bitmaps. The + * address entry encodes 1 relocation. The subsequent bitmap entries + * encode up to 63 relocations each, at subsequent offsets following + * the last address entry. + * + * The bitmap entries must have 1 in the least significant bit. The + * assumption here is that an address cannot have 1 in lsb. Odd + * addresses are not supported. Any odd addresses are stored in the + * RELA section, which is handled above. + * + * Excluding the least significant bit in the bitmap, each non-zero bit + * in the bitmap represents a relocation to be applied to a + * corresponding machine word that follows the base address word. The + * second least significant bit represents the machine word immediately + * following the initial address, and each bit that follows represents + * the next word, in linear order. As such, a single bitmap can encode + * up to 63 relocations in a 64-bit object. + */ + for (const u64 *relr = relr_start; relr < relr_end; relr++) { + if (!(*relr & 1)) { + place = (u64 *)(*relr + offset); + *place++ += offset; + } else { + for (u64 *p = place, r = *relr >> 1; r; p++, r >>= 1) + if (r & 1) + *p += offset; + place += 63; + } + } +} diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 4c13dafc98b8400f..bebb88daf4c52039 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -270,15 +270,15 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { - __rela_start = .; + __pi_rela_start = .; *(.rela .rela*) - __rela_end = .; + __pi_rela_end = .; } .relr.dyn : ALIGN(8) { - __relr_start = .; + __pi_relr_start = .; *(.relr.dyn) - __relr_end = .; + __pi_relr_end = .; } . = ALIGN(SEGMENT_ALIGN); @@ -317,6 +317,10 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; +#ifdef CONFIG_RELOCATABLE + . += SZ_4K; /* stack for the early relocation code */ + early_init_stack = .; +#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Fri Nov 11 17:11:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6B1F4C433FE for ; Fri, 11 Nov 2022 17:17:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6Nl3S5TDd0Ulld8pkucbVvgbManzKsfrobxaoltUkuE=; b=F5DBPo869D8fy4 7pvuIV1gV80mV2RvqFnHXv6YqrVzYD8VDK7cLeMceEC1COVoBkLSb/5QmJj6ATgzOFQ111Mdd5P7l ZVp3AzXWIzfCfvIPsDX/erVu4lOz9+RyDEf0u1xhQVbRSHXYgMpu12ZK/G5Y+d625MAivRdMTQYl4 gXsJEH3DnjDq3FpL0RI6Z/us8rqHK+aPDobg1fWDFdcQSnn8nNnuryMZlNRfnGRxwcLzCkdZ25kUk BtPcAuY1yqXnpGnlKMhSN17tPaRWV7rcmcJ+T1nEy/GXw0NAnToS0UVYdyON3yG3kvC8r0nfdtmbU Aq4Rnchz4o6heMob0P0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXds-00H9oI-HY; Fri, 11 Nov 2022 17:16:36 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXa4-00H7dX-6m for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:42 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B5112618C9; Fri, 11 Nov 2022 17:12:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 493DBC43143; Fri, 11 Nov 2022 17:12:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186759; bh=2eyf0AIH+GQWjsvMONvdga33hzdAa7BsuLR5uQfzhHQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ff92znWUdwjGJA67aLvuc0sp98Yfd78oUldRjy4Pl3AwuvFvBDtA/rpEz0shn7tfM L1pMyqabHNIYB4fDhgEZkVDlO2Ip/YEWFpivJYGkJGOULaB1oYFl+QLsMmJzNpL8sQ dwZEvZVmyoanNd5S7ROT37u8q/Fa9u5z61/9sRszGUtnNy9/3Kf1ABPXW/dWISXl8r rZah/up98ZJ2RqdkMVCtMXp6X05ElmRdjGyqAzrtNpicnsnaL24JVDaeMlmi5GA73t BQ93HjPoITtse9NaKMfpaSAy10+CzlcQQy+61tZJqHAcNP69sbPB98LN2LFo0hNBQG 1QnoMMx01vdCA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 09/33] arm64: Turn kaslr_feature_override into a generic SW feature override Date: Fri, 11 Nov 2022 18:11:37 +0100 Message-Id: <20221111171201.2088501-10-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4011; i=ardb@kernel.org; h=from:subject; bh=WNLURScLXYyJdWXNxGhOyTkTvnJfPC+J8gCSCf8ggFc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI36cmSGybTSxkDQY2jwqNasRcjQjmdyjSDDNUi NiFbkyiJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CNwAKCRDDTyI5ktmPJMlkC/ sH6vempL5KyeXvJnxOrImpVVL27N0vvrrWyGdTxUqAafzoBL9G3+83G0D999BkwvV3Ewq8+gNMauzQ L/AyVzqxIkhJPcJnZJ4jywmWFS0XbxB1iditiYB/6zv9zScU9VHvG5AQDZUTUjeMJZ82GKG0kfRgnN 7KxzEUs9/J/P+swPqVkWXamvuoR4C7RvRmH2HxlE0Xlmlofmf/nkd9Xlhjb1eddjJ15u+0HHEDLb4M u40T8koOLklhmHROanaUl/kR0kAEUj+UhCaq3IwRNmbhMzAWMHDuiZC7VtKf22DIsedo1Q9GJZEiWv zIdDXSrSd+a4HlX8HfRFcxpyJWjQh7hSdEA6CHZMcQHdWUZGJqzmAGoTWUlUwgIcMoUwRGN/OsbYvj OJ4SixswGnQ1UHjWJ1mZchMCn/nnzNLls5Cm2+47sUoH2iqI1B5lu3+0Fa10hRCB52Ol1Pq7hjaiqt mPd7WMeAgcBsPJITeChYkBBLi4BLErLOr4Z1WwtFdn5fk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091240_390714_14367342 X-CRM114-Status: GOOD ( 17.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Marc Zyngier Disabling KASLR from the command line is implemented as a feature override. Repaint it slightly so that it can further be used as more generic infrastructure for SW override purposes. Signed-off-by: Marc Zyngier Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 4 ++++ arch/arm64/kernel/cpufeature.c | 2 ++ arch/arm64/kernel/idreg-override.c | 16 ++++++---------- arch/arm64/kernel/kaslr.c | 6 +++--- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index f73f11b5504254be..f44a7860636fd411 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -15,6 +15,8 @@ #define MAX_CPU_FEATURES 128 #define cpu_feature(x) KERNEL_HWCAP_ ## x +#define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 + #ifndef __ASSEMBLY__ #include @@ -914,6 +916,8 @@ extern struct arm64_ftr_override id_aa64smfr0_override; extern struct arm64_ftr_override id_aa64isar1_override; extern struct arm64_ftr_override id_aa64isar2_override; +extern struct arm64_ftr_override arm64_sw_feature_override; + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index fdbae2320b466d98..ebd8cabffb105e15 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -644,6 +644,8 @@ struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; struct arm64_ftr_override __ro_after_init id_aa64isar1_override; struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override arm64_sw_feature_override; + static const struct __ftr_reg_entry { u32 sys_id; struct arm64_ftr_reg *reg; diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 95133765ed29a0e4..4e8ef5e05db7a424 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -137,15 +137,11 @@ static const struct ftr_set_desc smfr0 __initconst = { }, }; -extern struct arm64_ftr_override kaslr_feature_override; - -static const struct ftr_set_desc kaslr __initconst = { - .name = "kaslr", -#ifdef CONFIG_RANDOMIZE_BASE - .override = &kaslr_feature_override, -#endif +static const struct ftr_set_desc sw_features __initconst = { + .name = "arm64_sw", + .override = &arm64_sw_feature_override, .fields = { - FIELD("disabled", 0, NULL), + FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), {} }, }; @@ -157,7 +153,7 @@ static const struct ftr_set_desc * const regs[] __initconst = { &isar1, &isar2, &smfr0, - &kaslr, + &sw_features, }; static const struct { @@ -174,7 +170,7 @@ static const struct { "id_aa64isar1.api=0 id_aa64isar1.apa=0 " "id_aa64isar2.gpa3=0 id_aa64isar2.apa3=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, - { "nokaslr", "kaslr.disabled=1" }, + { "nokaslr", "arm64_sw.nokaslr=1" }, }; static int __init find_field(const char *cmdline, diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index e7477f21a4c9d062..5d4ce7f5f157bb3f 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -23,8 +23,6 @@ u64 __ro_after_init module_alloc_base; u16 __initdata memstart_offset_seed; -struct arm64_ftr_override kaslr_feature_override __initdata; - static int __init kaslr_init(void) { u64 module_range; @@ -36,7 +34,9 @@ static int __init kaslr_init(void) */ module_alloc_base = (u64)_etext - MODULES_VSIZE; - if (kaslr_feature_override.val & kaslr_feature_override.mask & 0xf) { + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val & + arm64_sw_feature_override.mask, + ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) { pr_info("KASLR disabled on command line\n"); return 0; } From patchwork Fri Nov 11 17:11:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41533C433FE for ; Fri, 11 Nov 2022 17:18:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4kCPsBwQ0pYBeDKpXTAhWBI3GemYuGYvt+a6sY9yVOY=; b=tMAL+znU8R6Man s1fkTQZD2eHXpDbZY9FmoW2eltsA2WiszRZCt3UjfqOKsRTlvhvlOkKucdGFTGy2NOctrhfu6r0GL OA4qPPAapxufHkSB5InPTgBc/44aoYqDglvdHM/yPJLvNky9Gq2WWAs/05cJ5Y99yR6suI4xgW5Cm tlSveBoDMOWXtT3lvgfpsdtC5fvmdbAH3r9gIB/kXTSI6aegJRWgjK2q9BgY9MxDVIXsFlgpomNY2 v5LqxwFGkg/r5vDD3mOC75UGsqPBHb+4ztkfnhuBX0ZzIXnFWU8+SpXY3IojUP9h95l1OZOrJtup5 JN/kX6o8rqaHSU5TEeOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXeU-00HADf-VO; Fri, 11 Nov 2022 17:17:15 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXa6-00H7fB-Dv for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:43 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0E7256206C; Fri, 11 Nov 2022 17:12:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91C7FC433B5; Fri, 11 Nov 2022 17:12:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186761; bh=EOI1RlNeS2R2j+GU3HS6zPXW1KCWLwxVJVn7Af6Hd5A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HpF7oc+UtUJ6a+6uIOGnlhoCXAKrbedZxLZxbcCiuK0adz96oLOKfEDoH6MV3bFaI ewc3T+1RFgi35LDDJqkfGbeGFAW6qDcSXHjbWGPC0TXbaXAXaIv/5xYtTj/elvOuLt WfmAoZ3sw64IvMzCMVUKw5Ujm1GWlMsxR8qgZ8cb/DQQzCfekeC51F7gjVMkWD0cSg +fUXhhNTghM663GS4uS+mztxUbWabA6Pv6ZXraziEDnFxxHzhEjgwF1VbuaOZ8xUEq d8pKI6wkbwYslwe/CdY02Wm6C9vK1weRDUCmwb7jkeFydiYnr1yMHHttJEYkJlBL4g 0eTMasWOHGZJw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 10/33] arm64: idreg-override: Omit non-NULL checks for override pointer Date: Fri, 11 Nov 2022 18:11:38 +0100 Message-Id: <20221111171201.2088501-11-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1672; i=ardb@kernel.org; h=from:subject; bh=EOI1RlNeS2R2j+GU3HS6zPXW1KCWLwxVJVn7Af6Hd5A=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI5TxsBMdfCcEUbwz9KagMKCw2cvi6Imb1AkF38 OJHAYMKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26COQAKCRDDTyI5ktmPJA0FDA CqpBCbEBuO5eWenxkU2VeAfELtCDinrQjKU7oVSqCJPQhB4/GfxrklEj1Bz3L9kGTh+tPke49+CmOM uKikeyZrd5FOANnl8M5lC5DkXkLbZ80ohilXjA54AFUhqezVn5loWWDghlaSkdT3yc0COIcuujFNFe RD6aFitSuIZCAaDEFcfvNv9Qlcec6KUEfgClZTwpHZ3ns1dMT/IQ35EVM9Yam8AvZV7yh+XCZG/axM 90JX/hvFj41amK95aVanqZrfx+t9vZ7kIbYuICYtUT8gTOgFH6zNAK+zSEbx0RbKVSN7CBWyDnnv1e AtDUmE+53hXdsOQxcTrOoSaHPXOqoHCbTFKDWOgNAC79VZX928jpn1bkT78N8zwxOGVI26UdWjalkh qUxjOsJyHpRKOic6UQAn3mhGvosZpHtmZgKWkp7jnKp7VI2t546jt2O13YRGG7q0YGehk7D4AzEvbi J27b3uaq/dQ9RxtSBOF4oiaE7nDs9/+czOQ2PHzjXVYQw= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091242_568239_C67C3D17 X-CRM114-Status: GOOD ( 14.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that override pointers are always set, we can drop the various non-NULL checks that we have in the code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 4e8ef5e05db7a424..d7fc813ba5913e27 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -195,9 +195,6 @@ static void __init match_options(const char *cmdline) for (i = 0; i < ARRAY_SIZE(regs); i++) { int f; - if (!regs[i]->override) - continue; - for (f = 0; strlen(regs[i]->fields[f].name); f++) { u64 shift = regs[i]->fields[f].shift; u64 width = regs[i]->fields[f].width ?: 4; @@ -298,10 +295,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { - if (regs[i]->override) { - regs[i]->override->val = 0; - regs[i]->override->mask = 0; - } + regs[i]->override->val = 0; + regs[i]->override->mask = 0; } __boot_status = boot_status; @@ -309,9 +304,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) parse_cmdline(); for (i = 0; i < ARRAY_SIZE(regs); i++) { - if (regs[i]->override) - dcache_clean_inval_poc((unsigned long)regs[i]->override, - (unsigned long)regs[i]->override + - sizeof(*regs[i]->override)); + dcache_clean_inval_poc((unsigned long)regs[i]->override, + (unsigned long)regs[i]->override + + sizeof(*regs[i]->override)); } } From patchwork Fri Nov 11 17:11:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6655BC433FE for ; Fri, 11 Nov 2022 17:19:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Qsh3yM3S/BmBDIh5XP6z9aljcHiE81JWhCGGrNl+H5w=; b=dmA0Z+K2bvrroh T85nXg2744vxsVP5/6NI6JBRU1QG7LxLttSY2nlmOqe4FoYHHj7axqzbg/+odlQ9UeQgc5aXFLYmS B9NWSOToJWXYQ3wVFRu/SppV75q4hMwxbk+QALRkNrO226LfrVBF5/3sCKJI9HHs4ZLDVxWL7BdwC eSU692UFteGmBQPYkr9v5iqF8QpIqXRDSnwxl26YLTp99Y+VPG9/fnHP6jCujfaVBmWmmnRkZSbpn Zd94n+MpV/JlkQzoi0hDf8To9aAkM7DYGPdu7JDQnsUP2JzrOtyRNu4MJUvG4elKGWxLnok9ILFoh rafUqgDcZ8995qyyGvfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXf7-00HAdd-3k; Fri, 11 Nov 2022 17:17:53 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaA-00H7hN-Gx for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:48 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 122F9B8267D; Fri, 11 Nov 2022 17:12:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DBC14C433D6; Fri, 11 Nov 2022 17:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186763; bh=jAT8NmqnR+0AJP+whSFSIznCQ7PGmdov1nT3tBNiGDk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XhIKMPJsTvqjyE+OMVFUs6kVrPFeRw7H/hL2DSg//kVwzYrsmxqBGlKA7HWP76rpN rPF2scS+d8Pp3sbafRKGvnbYcnOABjltCv8VZoSoIC0d4yVsCNxAJZDBOxUJcw3NfF 2cCsyYg76cXz9VC6ySI6konuzGVqDFnKkWM4jvntnbfRHlP90qHanSIiqL4Vy1AcxH UqM919fC4lW+TYI5YKizD5zdMbfMUbkaVQj5QECTWlRFQj3cRW72c2rr+GgaT/j4bc 7I5bdi3ADxUNBNuG4fsUeAGuIFmQVmrnNGoIVWrog/HLkfTMyYO4bcj6563aWbFdn2 WAyrkKjIWSxGg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 11/33] arm64: idreg-override: Use relative references to override variables Date: Fri, 11 Nov 2022 18:11:39 +0100 Message-Id: <20221111171201.2088501-12-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7694; i=ardb@kernel.org; h=from:subject; bh=jAT8NmqnR+0AJP+whSFSIznCQ7PGmdov1nT3tBNiGDk=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI7WDvLn3hazBbQ1g/cXBJHJ6wGzwHJ/E6J07tV WvSuk++JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26COwAKCRDDTyI5ktmPJKiKC/ 9QGY3r17BzwVZhzTJ1Mm63Wu5YqdjtYO5kNNPR1GAbkxe1hUcyURhZ1+E4TOI3tbFCCbou4OZRzLqE 8I95gfkL9W1elIVfMZ5Uarfx7l+xc+sL3qxF5eOiM5q4zFc1r7/aI7vqx58tlDzyK4OixTHj5HI2vO qiz0alrR1sJTFJDhsl3T8EAnX7XM2Rgn+5ln1mlGTmNoTyJA/6aKRwrqbmjlN/dJngTUqYe5CkJbOq L5lBqDlSctHXkrENlKgpzL89XmEB2tWILhvKgjVAmoSgHrbwcUxa4SUTCbLs5eZoBFbAvmNTXF5+wO 9Hj4ZDOiyn0YTs4fG7FG8QBX33U+niKWPdOYMtttHYYPZgr6LviXRacyM7pLmoRprlzPyjqfx90R33 GTIEym1Am7fw+ZCiPrtHsrv/FXnWhn7W2Z56hNu5jKiXpDXHmn6HxfLmVoiJ25E+awdubleJD64uwH w/eI10b+HXji+lFTsrfXQd4Kzxf5JUL5RFsuId6fWDXME= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091246_913965_1B3CD54B X-CRM114-Status: GOOD ( 21.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To prepare the idreg-override for running in a context where statically initialized absolute symbol references are not permitted, use place relative relocations to refer to the 'override' global variables in each feature override descriptor set, and populate the regs[] array using relative references as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 144 +++++++++----------- 1 file changed, 63 insertions(+), 81 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index d7fc813ba5913e27..f8ae7f6d0d9b4fd0 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -6,6 +6,7 @@ * Author: Marc Zyngier */ +#include #include #include #include @@ -22,18 +23,29 @@ static u64 __boot_status __initdata; struct ftr_set_desc { - char name[FTR_DESC_NAME_LEN]; - struct arm64_ftr_override *override; + s32 override_offset; // must remain first + char name[FTR_DESC_NAME_LEN]; struct { - char name[FTR_DESC_FIELD_LEN]; - u8 shift; - u8 width; - bool (*filter)(u64 val); - } fields[]; + char name[FTR_DESC_FIELD_LEN]; + u8 shift; + u8 width; + bool (*filter)(u64 val); + } fields[]; }; +static_assert(offsetof(struct ftr_set_desc, override_offset) == 0); + #define FIELD(n, s, f) { .name = n, .shift = s, .width = 4, .filter = f } +#define DEFINE_OVERRIDE(__idx, __id, __name, __ovr, ...) \ + asmlinkage const struct ftr_set_desc __initconst __id = { \ + .name = __name, \ + .fields = { __VA_ARGS__ }, \ + }; \ + asm(".globl " #__ovr "; " \ + ".reloc " #__id ", R_AARCH64_PREL32, " #__ovr "; " \ + ".reloc regs + (4 * " #__idx "), R_AARCH64_PREL32, " #__id) + static bool __init mmfr1_vh_filter(u64 val) { /* @@ -46,14 +58,9 @@ static bool __init mmfr1_vh_filter(u64 val) val == 0); } -static const struct ftr_set_desc mmfr1 __initconst = { - .name = "id_aa64mmfr1", - .override = &id_aa64mmfr1_override, - .fields = { +DEFINE_OVERRIDE(0, mmfr1, "id_aa64mmfr1", id_aa64mmfr1_override, FIELD("vh", ID_AA64MMFR1_EL1_VH_SHIFT, mmfr1_vh_filter), - {} - }, -}; + {}); static bool __init pfr0_sve_filter(u64 val) { @@ -70,14 +77,9 @@ static bool __init pfr0_sve_filter(u64 val) return true; } -static const struct ftr_set_desc pfr0 __initconst = { - .name = "id_aa64pfr0", - .override = &id_aa64pfr0_override, - .fields = { +DEFINE_OVERRIDE(1, pfr0, "id_aa64pfr0", id_aa64pfr0_override, FIELD("sve", ID_AA64PFR0_EL1_SVE_SHIFT, pfr0_sve_filter), - {} - }, -}; + {}); static bool __init pfr1_sme_filter(u64 val) { @@ -94,67 +96,46 @@ static bool __init pfr1_sme_filter(u64 val) return true; } -static const struct ftr_set_desc pfr1 __initconst = { - .name = "id_aa64pfr1", - .override = &id_aa64pfr1_override, - .fields = { +DEFINE_OVERRIDE(2, pfr1, "id_aa64pfr1", id_aa64pfr1_override, FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ), FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL), FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter), - {} - }, -}; + {}); -static const struct ftr_set_desc isar1 __initconst = { - .name = "id_aa64isar1", - .override = &id_aa64isar1_override, - .fields = { +DEFINE_OVERRIDE(3, isar1, "id_aa64isar1", id_aa64isar1_override, FIELD("gpi", ID_AA64ISAR1_EL1_GPI_SHIFT, NULL), FIELD("gpa", ID_AA64ISAR1_EL1_GPA_SHIFT, NULL), FIELD("api", ID_AA64ISAR1_EL1_API_SHIFT, NULL), FIELD("apa", ID_AA64ISAR1_EL1_APA_SHIFT, NULL), - {} - }, -}; + {}); -static const struct ftr_set_desc isar2 __initconst = { - .name = "id_aa64isar2", - .override = &id_aa64isar2_override, - .fields = { +DEFINE_OVERRIDE(4, isar2, "id_aa64isar2", id_aa64isar2_override, FIELD("gpa3", ID_AA64ISAR2_EL1_GPA3_SHIFT, NULL), FIELD("apa3", ID_AA64ISAR2_EL1_APA3_SHIFT, NULL), - {} - }, -}; + {}); -static const struct ftr_set_desc smfr0 __initconst = { - .name = "id_aa64smfr0", - .override = &id_aa64smfr0_override, - .fields = { +DEFINE_OVERRIDE(5, smfr0, "id_aa64smfr0", id_aa64smfr0_override, /* FA64 is a one bit field... :-/ */ { "fa64", ID_AA64SMFR0_EL1_FA64_SHIFT, 1, }, - {} - }, -}; + {}); -static const struct ftr_set_desc sw_features __initconst = { - .name = "arm64_sw", - .override = &arm64_sw_feature_override, - .fields = { +DEFINE_OVERRIDE(6, sw_features, "arm64_sw", arm64_sw_feature_override, FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), - {} - }, -}; + {}); -static const struct ftr_set_desc * const regs[] __initconst = { - &mmfr1, - &pfr0, - &pfr1, - &isar1, - &isar2, - &smfr0, - &sw_features, -}; +/* + * regs[] is populated by R_AARCH64_PREL32 directives invisible to the compiler + * so it cannot be static or const, or the compiler might try to use constant + * propagation on the values. + */ +asmlinkage s32 regs[7] __initdata = { [0 ... ARRAY_SIZE(regs) - 1] = S32_MAX }; + +static struct arm64_ftr_override * __init reg_override(int i) +{ + const struct ftr_set_desc *reg = offset_to_ptr(®s[i]); + + return offset_to_ptr(®->override_offset); +} static const struct { char alias[FTR_ALIAS_NAME_LEN]; @@ -193,15 +174,16 @@ static void __init match_options(const char *cmdline) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { + const struct ftr_set_desc *reg = offset_to_ptr(®s[i]); int f; - for (f = 0; strlen(regs[i]->fields[f].name); f++) { - u64 shift = regs[i]->fields[f].shift; - u64 width = regs[i]->fields[f].width ?: 4; + for (f = 0; strlen(reg->fields[f].name); f++) { + u64 shift = reg->fields[f].shift; + u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); u64 v; - if (find_field(cmdline, regs[i], f, &v)) + if (find_field(cmdline, reg, f, &v)) continue; /* @@ -209,16 +191,16 @@ static void __init match_options(const char *cmdline) * it by setting the value to the all-ones while * clearing the mask... Yes, this is fragile. */ - if (regs[i]->fields[f].filter && - !regs[i]->fields[f].filter(v)) { - regs[i]->override->val |= mask; - regs[i]->override->mask &= ~mask; + if (reg->fields[f].filter && + !reg->fields[f].filter(v)) { + reg_override(i)->val |= mask; + reg_override(i)->mask &= ~mask; continue; } - regs[i]->override->val &= ~mask; - regs[i]->override->val |= (v << shift) & mask; - regs[i]->override->mask |= mask; + reg_override(i)->val &= ~mask; + reg_override(i)->val |= (v << shift) & mask; + reg_override(i)->mask |= mask; return; } @@ -295,8 +277,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { - regs[i]->override->val = 0; - regs[i]->override->mask = 0; + reg_override(i)->val = 0; + reg_override(i)->mask = 0; } __boot_status = boot_status; @@ -304,8 +286,8 @@ asmlinkage void __init init_feature_override(u64 boot_status) parse_cmdline(); for (i = 0; i < ARRAY_SIZE(regs); i++) { - dcache_clean_inval_poc((unsigned long)regs[i]->override, - (unsigned long)regs[i]->override + - sizeof(*regs[i]->override)); + dcache_clean_inval_poc((unsigned long)reg_override(i), + (unsigned long)reg_override(i) + + sizeof(struct arm64_ftr_override)); } } From patchwork Fri Nov 11 17:11:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9DD45C433FE for ; Fri, 11 Nov 2022 17:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OTVOmj4ngzQyla8dFKpq7VEjRmyLE28IEbk2tL3PqGY=; b=E/Wkf8yHaXb7x4 WUCz/6702gh9I+NNDFBa9xozK5MTk9wkODsF9aO6I33VypVZrBY7H3DJUsFPhp3rYfgBfRHIPHgUw GTx1+z7nAcQkStRTGp7X4tasYIOkXU4Do56QL5j6YytOAmN+PJbiRFqE1GT1hX1oVYVQ/7WGXuFGj PD0R/RSqZGCL7uYPKWT8gfgNB22uarI2Jp/F8BM27tsV3PI5tX4bkUjZwIqunEZupXC37d6aSAddF t2VcR73vtQ/eay4l2ybwh76+RICst10S9e0dZ/LiVyu6VMf4PrduUKmMVrfEplBy8TNFh/ftHb4Ho pZFlMVS3jD/zCbooJlKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXfp-00HB1d-8Y; Fri, 11 Nov 2022 17:18:38 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaB-00H7i6-63 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:49 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 98EA7618C9; Fri, 11 Nov 2022 17:12:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2EA52C43143; Fri, 11 Nov 2022 17:12:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186766; bh=gLi5zeKVhBRT8OeLpkGKtd7SxypDRIvJhPj5bROHmFQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W/u74QK8d5RuVF6qpDaf+5PHS5Q7NBKCSJI1BjAHCKWjlxNOYAq/p9Sq4O8e69E7I kEhkB3PkTqrX6nPDorC6UmK9GpatmaoV2xwlfBK9Xx/BNvW0HFjGJeluMmVlJAxRek +XCEBr5nz4eLwXfLLf/8KkZj4oPEkr1wfDq1akNbIGlLKOcFp6KGb1XCavUm7F5lzR Hr89xUsJEv8o/9T88whJ1SglzsujhkxvWttOaX0ouaPDxzQnE9e80zeC/lrHVpAGcV QYHqJH7pkZC6Rba0m04izUzPbfBI+gzK2OKXQtDVZci4swM9B9lDljn93Yjihl1Csw tqsvqlnZqyo5w== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 12/33] arm64: idreg-override: Use relative references to filter routines Date: Fri, 11 Nov 2022 18:11:40 +0100 Message-Id: <20221111171201.2088501-13-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6429; i=ardb@kernel.org; h=from:subject; bh=gLi5zeKVhBRT8OeLpkGKtd7SxypDRIvJhPj5bROHmFQ=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI8+j8VZM1CSX/rO62HZ0eW0hT6Ys3HmP9PAKwR LQwvQCaJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CPAAKCRDDTyI5ktmPJIjGC/ 48G3FrEJr2K8IAbmisVQ+/DEoABnZLbXN40qLPh9gS+rHSgiX75dWczg1WRUJvhFm5+uRQFiOSQsJj DdenhIIA9vhXeyBLLur7lYlxFjuRpiMXZoh579jjKPQZo4707QG4owEYiLOq4MnU+oTVjfR3jwL8Nk ApCTcCY5RZ2ZhWY6mUcWhdlMMQPj2IJGfzgaQSOG8FjXt+5PbhOqK4FA3QRicXw877tqFG3SGaJpaG nJP9r3s7EdChNdtCrYWowCvB8BAlDapMErI2VeJJquoxO++ovlZdSzNiaUWXH79mFZCs6acYmyCdTk kYMCWoTT3rpbMiNBzqJ9J5WZ58t9J6SAwhcN0TY0xNaL5LriTbr8VWIzmU9qV0+sXM53L0JTM8ma1J kIYb84bgVid5ONF7HmVztR3nr9KCmoys3RjbrSouSWNONm11jk7s0uDO1XEvDY4opxM9s5rT+1+Gmh YZzwrD0x63z70bO/sWk6AY7BaL7n+NrpNm3vTnGXwraKM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091247_388125_7A1CCD2E X-CRM114-Status: GOOD ( 19.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To avoid statically initialized pointer variables, which need runtime relocation and therefore prevent this code from being used before relocations have been processed, tweak the static declarations so that relative references are used instead. This means we will be doing the job of the compiler and calculate where exactly the relocation needs to point, so add some asserts to ensure we notice when we get it wrong. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 63 +++++++++++++------- 1 file changed, 41 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index f8ae7f6d0d9b4fd0..01eed0eaba7c1cdd 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -15,8 +15,8 @@ #include #include -#define FTR_DESC_NAME_LEN 20 -#define FTR_DESC_FIELD_LEN 10 +#define FTR_DESC_NAME_LEN 20 // must remain multiple of 4 +#define FTR_DESC_FIELD_LEN 10 // must remain multiple of 4 +/- 2 #define FTR_ALIAS_NAME_LEN 30 #define FTR_ALIAS_OPTION_LEN 116 @@ -26,16 +26,20 @@ struct ftr_set_desc { s32 override_offset; // must remain first char name[FTR_DESC_NAME_LEN]; struct { + s32 filter_offset; // must remain first char name[FTR_DESC_FIELD_LEN]; u8 shift; u8 width; - bool (*filter)(u64 val); } fields[]; }; static_assert(offsetof(struct ftr_set_desc, override_offset) == 0); +static_assert(offsetof(struct ftr_set_desc, fields[0].filter_offset) == + 4 + FTR_DESC_NAME_LEN); +static_assert(offsetof(struct ftr_set_desc, fields[1].filter_offset) == + 4 + FTR_DESC_NAME_LEN + 4 + FTR_DESC_FIELD_LEN + 2); -#define FIELD(n, s, f) { .name = n, .shift = s, .width = 4, .filter = f } +#define FIELD(n, s) { .name = n, .shift = s, .width = 4 } #define DEFINE_OVERRIDE(__idx, __id, __name, __ovr, ...) \ asmlinkage const struct ftr_set_desc __initconst __id = { \ @@ -46,7 +50,12 @@ static_assert(offsetof(struct ftr_set_desc, override_offset) == 0); ".reloc " #__id ", R_AARCH64_PREL32, " #__ovr "; " \ ".reloc regs + (4 * " #__idx "), R_AARCH64_PREL32, " #__id) -static bool __init mmfr1_vh_filter(u64 val) +#define DEFINE_OVERRIDE_FILTER(__id, __idx, __filter) \ + asm(".reloc " #__id " + 4 + " __stringify(FTR_DESC_NAME_LEN) \ + " + " #__idx " * (4 + " __stringify(FTR_DESC_FIELD_LEN) " + 2)," \ + "R_AARCH64_PREL32, " #__filter) + +asmlinkage bool __init mmfr1_vh_filter(u64 val) { /* * If we ever reach this point while running VHE, we're @@ -59,10 +68,11 @@ static bool __init mmfr1_vh_filter(u64 val) } DEFINE_OVERRIDE(0, mmfr1, "id_aa64mmfr1", id_aa64mmfr1_override, - FIELD("vh", ID_AA64MMFR1_EL1_VH_SHIFT, mmfr1_vh_filter), + FIELD("vh", ID_AA64MMFR1_EL1_VH_SHIFT), {}); +DEFINE_OVERRIDE_FILTER(mmfr1, 0, mmfr1_vh_filter); -static bool __init pfr0_sve_filter(u64 val) +asmlinkage bool __init pfr0_sve_filter(u64 val) { /* * Disabling SVE also means disabling all the features that @@ -78,10 +88,11 @@ static bool __init pfr0_sve_filter(u64 val) } DEFINE_OVERRIDE(1, pfr0, "id_aa64pfr0", id_aa64pfr0_override, - FIELD("sve", ID_AA64PFR0_EL1_SVE_SHIFT, pfr0_sve_filter), + FIELD("sve", ID_AA64PFR0_EL1_SVE_SHIFT), {}); +DEFINE_OVERRIDE_FILTER(pfr0, 0, pfr0_sve_filter); -static bool __init pfr1_sme_filter(u64 val) +asmlinkage bool __init pfr1_sme_filter(u64 val) { /* * Similarly to SVE, disabling SME also means disabling all @@ -97,30 +108,31 @@ static bool __init pfr1_sme_filter(u64 val) } DEFINE_OVERRIDE(2, pfr1, "id_aa64pfr1", id_aa64pfr1_override, - FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ), - FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL), - FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter), + FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT ), + FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT), + FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT), {}); +DEFINE_OVERRIDE_FILTER(pfr1, 2, pfr1_sme_filter); DEFINE_OVERRIDE(3, isar1, "id_aa64isar1", id_aa64isar1_override, - FIELD("gpi", ID_AA64ISAR1_EL1_GPI_SHIFT, NULL), - FIELD("gpa", ID_AA64ISAR1_EL1_GPA_SHIFT, NULL), - FIELD("api", ID_AA64ISAR1_EL1_API_SHIFT, NULL), - FIELD("apa", ID_AA64ISAR1_EL1_APA_SHIFT, NULL), + FIELD("gpi", ID_AA64ISAR1_EL1_GPI_SHIFT), + FIELD("gpa", ID_AA64ISAR1_EL1_GPA_SHIFT), + FIELD("api", ID_AA64ISAR1_EL1_API_SHIFT), + FIELD("apa", ID_AA64ISAR1_EL1_APA_SHIFT), {}); DEFINE_OVERRIDE(4, isar2, "id_aa64isar2", id_aa64isar2_override, - FIELD("gpa3", ID_AA64ISAR2_EL1_GPA3_SHIFT, NULL), - FIELD("apa3", ID_AA64ISAR2_EL1_APA3_SHIFT, NULL), + FIELD("gpa3", ID_AA64ISAR2_EL1_GPA3_SHIFT), + FIELD("apa3", ID_AA64ISAR2_EL1_APA3_SHIFT), {}); DEFINE_OVERRIDE(5, smfr0, "id_aa64smfr0", id_aa64smfr0_override, /* FA64 is a one bit field... :-/ */ - { "fa64", ID_AA64SMFR0_EL1_FA64_SHIFT, 1, }, + { 0, "fa64", ID_AA64SMFR0_EL1_FA64_SHIFT, 1, }, {}); DEFINE_OVERRIDE(6, sw_features, "arm64_sw", arm64_sw_feature_override, - FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR, NULL), + FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR), {}); /* @@ -169,6 +181,13 @@ static int __init find_field(const char *cmdline, return kstrtou64(cmdline + len, 0, v); } +static const void * __init get_filter(const struct ftr_set_desc *reg, int idx) +{ + const s32 *offset = ®->fields[idx].filter_offset; + + return *offset ? offset_to_ptr(offset) : NULL; +} + static void __init match_options(const char *cmdline) { int i; @@ -181,6 +200,7 @@ static void __init match_options(const char *cmdline) u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); + bool (*filter)(u64) = get_filter(reg, f); u64 v; if (find_field(cmdline, reg, f, &v)) @@ -191,8 +211,7 @@ static void __init match_options(const char *cmdline) * it by setting the value to the all-ones while * clearing the mask... Yes, this is fragile. */ - if (reg->fields[f].filter && - !reg->fields[f].filter(v)) { + if (filter && !filter(v)) { reg_override(i)->val |= mask; reg_override(i)->mask &= ~mask; continue; From patchwork Fri Nov 11 17:11:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E930DC4332F for ; Fri, 11 Nov 2022 17:20:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Lltd5r4zumHiK60aBx+8+1UDU6YQGc5H7o7NxhC0f6w=; b=apE/agDNa8qpil MXExBDP0EOvie6ZfSSaKyNS2dcNXOv4KNLTOvE91pQS8UyfUxjwJA49pNejhl0ecnNJAHWCGOjGE2 kuDmmVgjajc0d7DZ6A+eIwXo0Sk8nUxEU/ksF9m5P+UwVFPlood7O8OYNdboYx0FaxsfdvDAsqihw oB8HE9Kkz2EQBUoCoMEZLz+EmGhGRK79KzbKu/rnaal5DJfqIa255uBUPfIHAMErui76I2t9x4lUK Jdhy1lZk+vE0L+C/jovaTb8csR9I7jTCrUuhYivojUsmYeDMiHnOIrHJrqZp1iCC+34Q1QtcDI0Ut yVLonRJGF3UHGbKTQJXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXgj-00HBZt-JO; Fri, 11 Nov 2022 17:19:33 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaF-00H7kU-0a for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:52 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A7AC3B8266E; Fri, 11 Nov 2022 17:12:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77E4EC433C1; Fri, 11 Nov 2022 17:12:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186768; bh=tv21gCt/gFkvZkigToTDT5JLM8WfuFUkfUZ5nWhYLXo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dinqIf/nG+37+pF4CUHSIWL6u/7GOIdwZWZzOb3D0vKGYyHYmIPS1JkJXp1+f4KG6 gLmCS+m//gpHBIfyVeRsZZ0WM6AoLaGD1jS3/O13ywJYdxBMd7sX/GAh6TCB3e71XN 2fBU/+DpteOpfFJo+O4Qp93p0UKoh5s7gUpkH4GdPXIuoHpJCJlcieYGpgCj8WJfWQ JuYoh0PHpEkQ5qNtC1CG6ezGO5/5cklG76o3tYXBZi/X3Iq088HVrdeQ2jnrUPbcU6 ZwMDUE6amqWTmC8yEcuAQWfVsN8Doo7FXoTmV10W/9ojX9NeHorIUBOXyM5x9OV9+Q HOQfiZRFkKOEg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 13/33] arm64: idreg-override: Avoid parameq() and parameqn() Date: Fri, 11 Nov 2022 18:11:41 +0100 Message-Id: <20221111171201.2088501-14-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2747; i=ardb@kernel.org; h=from:subject; bh=tv21gCt/gFkvZkigToTDT5JLM8WfuFUkfUZ5nWhYLXo=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboI+dOahBhNM9UQBAnuMW8F7dmZyoCFjb8ojNQcX t/QIO6iJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CPgAKCRDDTyI5ktmPJOKSC/ 40vxbVwj7gUovb5P/U7O/wVn9PDXoXpYacRxJTpYrh4htF0iRjcTFvr/plEMGnrPpKy7ws581TpM9r bZa5dDXNyB0cJSDgpVLC8gO+i3BB9LKBkXZGnWaJPCXBcHhamMqdh4OM4Qvey1Guz8IfB5Zr9Q3c1F yn3zTBvJ8eeQ03nPE+EYrd24EGZai88zWVdfQsV7/BWWBfpbxglT0C24v2yaBuZ30xduftabvczZWz X49INSfpfkhzAVhrxK11L0oSpaNZKtBVf1OGq8UG/3kSsuLilOITbbXYjBFa7CSCDt/KPZmiRRQvO9 /F6umJ1jnQYWaCHPSw4TROzre2RDLkzHjBcQI6TbgYQ6GdDjlWsZBH8M+iDN7JlZH8Yzzkudo45jdt zkEntHU7vbupuwznuqfkoIi5+5IR116U+zjnt/VQXSA5fFBIpAg0SLX5qutpAtcJ0R741wVtdwMwoG 5Ks9YstPJrh109fHqEUejlukm5t7RD0Kl0/YZrZYkdLTk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091251_401100_F458EC9B X-CRM114-Status: GOOD ( 18.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The only way parameq() and parameqn() deviate from the ordinary string and memory routines is that they ignore the difference between dashes and underscores. Since we copy each command line argument into a buffer before passing it to parameq() and parameqn() numerous times, let's just convert all dashes to underscores just once, and update the alias array accordingly. This also helps reduce the dependency on kernel APIs that are no longer available once we move this code into the early mini C runtime. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 26 ++++++++++++-------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 01eed0eaba7c1cdd..b3288e827a6bbec3 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -153,8 +153,8 @@ static const struct { char alias[FTR_ALIAS_NAME_LEN]; char feature[FTR_ALIAS_OPTION_LEN]; } aliases[] __initconst = { - { "kvm-arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, - { "kvm-arm.mode=protected", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=protected", "id_aa64mmfr1.vh=0" }, { "arm64.nosve", "id_aa64pfr0.sve=0 id_aa64pfr1.sme=0" }, { "arm64.nosme", "id_aa64pfr1.sme=0" }, { "arm64.nobti", "id_aa64pfr1.bt=0" }, @@ -175,7 +175,7 @@ static int __init find_field(const char *cmdline, len = snprintf(opt, ARRAY_SIZE(opt), "%s.%s=", reg->name, reg->fields[f].name); - if (!parameqn(cmdline, opt, len)) + if (memcmp(cmdline, opt, len)) return -1; return kstrtou64(cmdline + len, 0, v); @@ -235,23 +235,29 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) cmdline = skip_spaces(cmdline); - for (len = 0; cmdline[len] && !isspace(cmdline[len]); len++); + /* terminate on "--" appearing on the command line by itself */ + if (cmdline[0] == '-' && cmdline[1] == '-' && isspace(cmdline[2])) + return; + + for (len = 0; cmdline[len] && !isspace(cmdline[len]); len++) { + if (len >= sizeof(buf) - 1) + break; + if (cmdline[len] == '-') + buf[len] = '_'; + else + buf[len] = cmdline[len]; + } if (!len) return; - len = min(len, ARRAY_SIZE(buf) - 1); - strncpy(buf, cmdline, len); buf[len] = 0; - if (strcmp(buf, "--") == 0) - return; - cmdline += len; match_options(buf); for (i = 0; parse_aliases && i < ARRAY_SIZE(aliases); i++) - if (parameq(buf, aliases[i].alias)) + if (!memcmp(buf, aliases[i].alias, len + 1)) __parse_cmdline(aliases[i].feature, false); } while (1); } From patchwork Fri Nov 11 17:11:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7E675C433FE for ; Fri, 11 Nov 2022 17:21:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5kgaRFuaVAIzm9iGb/I/GmkZHD+Rr4SgdJquB7wyJ+s=; b=DK/a0XAGZ9WzBl V7jOMvlq+RKPII1SDWgAbhZL3h81j/zoVxzNbuZViX5PscPzTqiFI0DyPbc0KRhHsLZQygFQfAIRF geFaxPwqpK6RttgUl1vmwiVacPKBd4E+VSfE+LuIhLttVsbZSv9hUi+B1w8OoEmMCxseZmvgOvpvH SZU/gVGjW2m0GdOu0hrrFuy0CSQbbtQMcfC3kNd3Fvq7pnDDayJUS9Rf3VHkF2BKi5ZL9V9F9EVA7 Xir6bIzf6l5h+0Zkk3Q1+fOQopW7H2hu95mI79iUTWFfjD29Z5Alatp4YL3syBMxdOl96vkqC3xqD OT54y264gXfJCWEnttDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXho-00HC9h-Dg; Fri, 11 Nov 2022 17:20:40 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaG-00H7kv-9Y for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:54 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id EFCF6B82680; Fri, 11 Nov 2022 17:12:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BF290C43145; Fri, 11 Nov 2022 17:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186770; bh=5HcnPAIJ4QPJNoy+UnmbfcBqA20mwt+bVYXNxusD5CA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ztj2e6AaqM43kHl5/vCfPeNXnaRbevz+Mv6O9XB3x7oKfXA2oWgurYS9wp+7s9TAZ uXbzvw5tEgkwDnvLlsF7nbQJuo9aP04E4wIp0vuBomkaHFnq16sMFbMhP2NuYcp0xj +r5X3gwAAnyFMmDh6Qtj5OvC85tdboYXWN4P09lPyZdIaEPy4am8KffXWAHLYzbBEN 6dzZajgFPqLhH868cl4HYHQ/Ga48qsytQf9YZGoQDx2BMMXwpfrgV5VO6ZQh0VbfpX TKFRcJ685dQl/Kg0Tfav3Q3QCTR07c+hKI2M3P+S/rkMVLGCiGU+U95egv/8M5yTiY k1F3OXO4Q01kg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 14/33] arm64: idreg-override: avoid strlen() to check for empty strings Date: Fri, 11 Nov 2022 18:11:42 +0100 Message-Id: <20221111171201.2088501-15-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=926; i=ardb@kernel.org; h=from:subject; bh=5HcnPAIJ4QPJNoy+UnmbfcBqA20mwt+bVYXNxusD5CA=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJAbXeCuRGUNf/99EalkD6+whsstqSdpjrLb/PB qnb9ZaKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CQAAKCRDDTyI5ktmPJPiiDA DEyKletnnLbMWfbjAebEc2KMs/21ybY4DtA4ArbygcvEJv/6wH+6VpXMtAlNNRFve4sA3OXSXVlkWG X1wgmP73HT1r6jSqGeq+rn2d7ED1G9JC+0eTZgf4F8NsvcSmQ5hlZqdSsiUoxSYTowi/3CRIqBseHR y+NQ3Kv60ANbBZyjsfbMK+XiujLpMlg4v9SJ2iOHyiQ1wkXQ8cb6XlsQBz8ex3ob131yxwwxzCo4hN 90GawOZbdHU33bvGpAs7lM77KuREoi7qhIGdlC0sLS6s3iwneBbKXuMd7lZ8rRv3EbPonrLGpRRI0o yw8BvjoSd5Rz0qyJc2MCZWU23W7PQ/nTlK1um+LCjxVveOMsRpuOx9fLhEK7Z2ucFMo/Cl5xCjBFEZ 9Rub8ZpAdJ05JM+XEqDrYDcCXD5sg8etRkCF1p96GnkNFCPkbbh1R4mc8vqhq1mDQcaR/XaEv/+8tt bGU98bjMh6WkfuCYtvRenF9j08o/TN2/sf8p291IVZMHI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091252_530494_C4A2D1C0 X-CRM114-Status: GOOD ( 15.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org strlen() is a costly way to decide whether a string is empty, as in that case, the first character will be NUL so we can check for that directly. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index b3288e827a6bbec3..97ec832d87d44f64 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -196,7 +196,7 @@ static void __init match_options(const char *cmdline) const struct ftr_set_desc *reg = offset_to_ptr(®s[i]); int f; - for (f = 0; strlen(reg->fields[f].name); f++) { + for (f = 0; reg->fields[f].name[0] != '\0'; f++) { u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; u64 mask = GENMASK_ULL(shift + width - 1, shift); From patchwork Fri Nov 11 17:11:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B4D2C4332F for ; Fri, 11 Nov 2022 17:23:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YB7eL06ZRqsMJAq81OkPRSkCN4NnQEVLWM24ZfTAqMM=; b=Yj11ON6fPcvQDl DQ+h1UXUW3jn0hDrleZR8MftpPjjCQAEDWDUybN2X9/H9u4Kgb/S7c/VkEmQEgaTTm8vb1Gt3H+ST Uiy07vn0ddcqJ0W+iRUP53K4acJRYzAU3iqHsVu+09U0aPFtHGK2vTL4nDeNxud14JD/WfbFl/RmI jcNoO3HDJ0OrRQ8NjBdbxENkGPP7JL6KTjTGhRoBvT0Z/HCvMrAh3e4R8T/GFgVz8RRk5ZkzXpY/m 5cY+H6zELjJUwHLSBTMClhU2PboN8FlJnt3mE98R+J6a5703X9cdH9rGbVnpPgTejJ8qbBEmEzYzY Aeg2hN/JdulDY0bam2gA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXj8-00HCmb-CP; Fri, 11 Nov 2022 17:22:03 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaH-00H7mV-Tq for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:55 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8257962068; Fri, 11 Nov 2022 17:12:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 14C0FC43146; Fri, 11 Nov 2022 17:12:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186772; bh=A2WyRGZ5o48qgcumyUSsq0JAEwTt/8p5p7vznI25Q3s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DWcpftWW5DB0E5zj/yeT3Zp7BzNCs9jFUq6QArkB9Wv+QjWwbpwLr3n2bwvCPiiOH ZmxkO/uMOocW+CBNQT8bWsl+SElU7kgqw4rhZKT2Yvp2gShkLfeNSgLK0BykgXjAZP PutHBaz3FO5JAQqwAL5Yc6Bb1as9F5ABtKfdE59S1KNriU+dorWMnJup+fEuTxvZ29 2eSnCk+jaGu+Hknm2h2pRS0KPDLIJ3TIdQ67UMuA16C2fyyoO1IWrUYy1fGAH90/WX CdH5T+5zA4UmYnwUCZ2ygipDoBTSAv5X9IGU9IFBxvjXcncLtFLU36rox5PO9rWGrR ZhTgrk/DFOR2w== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 15/33] arm64: idreg-override: Avoid sprintf() for simple string concatenation Date: Fri, 11 Nov 2022 18:11:43 +0100 Message-Id: <20221111171201.2088501-16-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2329; i=ardb@kernel.org; h=from:subject; bh=A2WyRGZ5o48qgcumyUSsq0JAEwTt/8p5p7vznI25Q3s=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJBs9NxBIY4U0OBxs89a4tMyVnsT5ycStjAz9mG uIfHnyuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CQQAKCRDDTyI5ktmPJFRZDA CO0nHCENwlSc22mvfRKldu23OV2e+HFsqDxPftQ0XTSDEKjHTX0aqs1Wpde+zy/t32eZkwK3zJFgvj jbIeW1OMN65pbYnsb7CqjBlGDpP6s0b8O2PhuZSFKJH34kg7R49Dp13TsHxzDC+7npFtJ2WiWMoV2O FCNRWL4Ls+xW3AqIdzZn4QKWij3ni5Ky8cHXeXHsuh8xdtGSJgVnob25Tag9Jf84LAVFwjBp6hGLqZ TPQMWJMoRUctmAZfDfy2cHQJx9QlDXLW2cTA7j025sbhZjfPiKDS+OweYxLktsjexvz0Gq7fLCTdUX 9ibIW2pttZpU8hu/sZni74YKVTDwwUNrmctGMCAXtP+Fjpd1/WaS83NQbNB0mDYMKo7qOO7XT+OS+7 NS1HdwDjqqf133mn0ooM5m1QRchyau7lQ2wwbnjNOwF/5uiKFpmFXadKbVCphYbJLjESFh07wfWZ8L EYG9YSGUZoGhLgXgIdunoH1B2/H8hsrmzwVYhdpyBjQfE= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091254_103415_650AC2A3 X-CRM114-Status: GOOD ( 17.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Instead of using sprintf() with the "%s.%s=" format, where the first string argument is always the same in the inner loop of match_options(), use simple memcpy() for string concatenation, and move the first copy to the outer loop. This removes the dependency on sprintf(), which will be difficult to fulfil when we move this code into the early mini C runtime. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index 97ec832d87d44f64..a5299aa1d1711adc 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -166,14 +166,15 @@ static const struct { { "nokaslr", "arm64_sw.nokaslr=1" }, }; -static int __init find_field(const char *cmdline, +static int __init find_field(const char *cmdline, char *opt, int len, const struct ftr_set_desc *reg, int f, u64 *v) { - char opt[FTR_DESC_NAME_LEN + FTR_DESC_FIELD_LEN + 2]; - int len; + int flen = strlen(reg->fields[f].name); - len = snprintf(opt, ARRAY_SIZE(opt), "%s.%s=", - reg->name, reg->fields[f].name); + // append '=' to obtain '.=' + memcpy(opt + len, reg->fields[f].name, flen); + len += flen; + opt[len++] = '='; if (memcmp(cmdline, opt, len)) return -1; @@ -190,12 +191,18 @@ static const void * __init get_filter(const struct ftr_set_desc *reg, int idx) static void __init match_options(const char *cmdline) { + char opt[FTR_DESC_NAME_LEN + FTR_DESC_FIELD_LEN + 2]; int i; for (i = 0; i < ARRAY_SIZE(regs); i++) { const struct ftr_set_desc *reg = offset_to_ptr(®s[i]); + int len = strlen(reg->name); int f; + // set opt[] to '.' + memcpy(opt, reg->name, len); + opt[len++] = '.'; + for (f = 0; reg->fields[f].name[0] != '\0'; f++) { u64 shift = reg->fields[f].shift; u64 width = reg->fields[f].width ?: 4; @@ -203,7 +210,7 @@ static void __init match_options(const char *cmdline) bool (*filter)(u64) = get_filter(reg, f); u64 v; - if (find_field(cmdline, reg, f, &v)) + if (find_field(cmdline, opt, len, reg, f, &v)) continue; /* From patchwork Fri Nov 11 17:11:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D8EAC4332F for ; Fri, 11 Nov 2022 17:23:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WRYZTZrEKvPlP5pY2UUd3oWWJ15nXi4fCYjSnAWw3ss=; b=kP5cZEADqogL/k PXVpDi5dSnhdLmv/m2jCyo4h/A8kdfBWgWPH7GFCm/5/Y3s/uAi1xkdvLYj7H/YmSe2Zf/ByMpr7p voFhsyqz+9td6h48O1722Hc8baXj3/xov3UsMua8NJnnfh6fb8bUdebFGxpz3mA9v/mp5egZUReMd PIEtzZpZpeEzrnEIIIwhU1U0aSPpPcjzIuv7yNVvW3Z8ST4bBBCdLwD9pgmk6ME7W4XZOtCaAk4rF 20PbjgzJsf3PjA+GLPLKOgdTLRZEkbp53O5J/ZLdhe46ZSPVGyn1+yNfsXBXHy7/wvZUddYX7YAYi fyEHMDE+uJwE2fKc3oDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXjs-00HDAm-9w; Fri, 11 Nov 2022 17:22:48 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaM-00H7oQ-29 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:12:59 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7B92EB8267B; Fri, 11 Nov 2022 17:12:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5CA38C433C1; Fri, 11 Nov 2022 17:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186775; bh=VCaBjaBSC69cfWOYCCxx7f+EfeKrpKSYJMoitiDgIDQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OgN6usmTCkJ6AqjsWDcHgS1AK9rAe0uJVmY40hjz0k3pRinIHdz0iNMM0Ced0vXN2 YIkEpFPMNVG74UPjruV1woaoElcLH7oUYLA8Hu07vPRCjTigqPrASPYMknRCSVDwp9 suXcKqf0JM8f7GOsIYWcnT3Zozkm0mHmfhg+g+vQPl/zICDXv3yuRqNnSVXfEV6UmP VBVldYCwaOKVOvSIErg8/36mPhklHTHVz4xJjUJhBm96YsmY+VXiZqJck4Gj4h1L6m b7qLXw7EwKnnvtW2BEIeQhNYQJF2W3Ws5H0Z7Uns4AZlKjjYrOzs7yuUT8ZinbKFca 6ej9AthUs4K6Q== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 16/33] arm64: idreg_override: Avoid kstrtou64() to parse a single hex digit Date: Fri, 11 Nov 2022 18:11:44 +0100 Message-Id: <20221111171201.2088501-17-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1909; i=ardb@kernel.org; h=from:subject; bh=VCaBjaBSC69cfWOYCCxx7f+EfeKrpKSYJMoitiDgIDQ=; b=owEB7AET/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJDkEBpVOUEwT7POU7JhiS2ERXH8fhMDAgvE2tx mBAKaZiJAbIEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CQwAKCRDDTyI5ktmPJBgPC/ ivYlw4KiQxxoTqTdTmmbHReZbDF6B/3T/Hb/heZ+ovJA5e6njAiqvnUwnVnzHhR1gR2Teh4CB/0FN5 M9dDeXWQNWsGEmK97fnjS58xm2tT6e7yyWICwPk0GRLag6zGWn3+k7tKMmRSixCWMnnaxTGxsWIOdf 8FEwclJGJiZ/ELNcqBzTsbuUcp19ydtFF2GtH4ZpM5c+C5GC6M8Ap4E6N5H8y70j3UBfA30c2XGuBD DF8tKQSuj7YusKXNhG8D9a6MXY7UlY517+koPuJAAfZtr5PcuxV03biGVJ3rCYgTH9M7QtS41NKy/7 tAhD8JuCsVJOdMNy8RLT1/WK6/TtvY+qsVJsAEJ7XnJkzonP30ibz0KIkyBfRAZJSt7k8+VlPT7DBi 0ydeE591t8spR7D0V/p4d40YOjdSfKY6v+r/Kn0p2NxqTZPfx2va01iMvMDuKDal6Kt//m2XKJ1Opr KiJihvWc0opSx/8EHP9S0yUCB30+3Yi9zv1PGPg5G7+A== X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091258_460633_350027C6 X-CRM114-Status: GOOD ( 17.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org All ID register value overrides are =0 with the exception of the nokaslr pseudo feature which uses =1. In order to remove the dependency on kstrtou64(), which is part of the core kernel and no longer usable once we move idreg-override into the early mini C runtime, let's just parse a single hex digit (with optional leading 0x) and set the output value accordingly. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/idreg-override.c | 27 +++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c index a5299aa1d1711adc..7e3eb48f5c83d539 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/idreg-override.c @@ -170,6 +170,7 @@ static int __init find_field(const char *cmdline, char *opt, int len, const struct ftr_set_desc *reg, int f, u64 *v) { int flen = strlen(reg->fields[f].name); + const char *p; // append '=' to obtain '.=' memcpy(opt + len, reg->fields[f].name, flen); @@ -179,7 +180,31 @@ static int __init find_field(const char *cmdline, char *opt, int len, if (memcmp(cmdline, opt, len)) return -1; - return kstrtou64(cmdline + len, 0, v); + p = cmdline + len; + + // skip "0x" if it comes next + if (p[0] == '0' && (p[1] == 'x' || p[1] == 'X')) + p += 2; + + // check whether the RHS is a single non-whitespace character + if (*p == '\0' || (p[1] && !isspace(p[1]))) + return -1; + + // only accept a single hex character as the value + switch (*p) { + case '0' ... '9': + *v = *p - '0'; + break; + case 'a' ... 'f': + *v = *p - 'a' + 10; + break; + case 'A' ... 'F': + *v = *p - 'A' + 10; + break; + default: + return -1; + } + return 0; } static const void * __init get_filter(const struct ftr_set_desc *reg, int idx) From patchwork Fri Nov 11 17:11:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6438DC433FE for ; Fri, 11 Nov 2022 17:25:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IhBmW5wIRMCoj1X3Qn00+m0kMQV2Am1dpi3VUIsDB8I=; b=tonEgVYINFZktc gZgwL9b3rlo5nIvkY69R+AP0oSGIOneD6sABnz+jZTNNPH1WUdt+Zn5WnfrF4hODVEZ+Xn0sUzbHW 2oPSTI5CIa/MWEpmjJ1MaKIPHeySBqE8pfGZ68mYJ/srC7lutqHXCd5S6BsPQ1Nj6YAOVhFiCYGoj eBvAt9AOzZLYDhXAHtBT4daO2udoiHML5UcvuEBD/77wNHGLkm+9VqdexrIp7OUiCUdpeS+EmJWnv 7meWNKAWFbLBERKAn7nU9xiYaoBGeG7sxH5VHjIXEtSot/JjnY4JzB5byDvcPLRS6XtDPPmS0jK8r bB4GT6VbsCpe7p9wiIWg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXks-00HDfD-31; Fri, 11 Nov 2022 17:23:50 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaO-00H7q4-BZ for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:02 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D7A50B82681; Fri, 11 Nov 2022 17:12:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A4EAAC43143; Fri, 11 Nov 2022 17:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186777; bh=XcCalZgDu8W95+sS2ba5T6FJRehgpS6pDpztbl7asjk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J2OAGUM0oATDTDEa8yCEkMODhj+XAlTX7PL3vW2QU2nE9RwZXMm8PFIE6h1cSoShx Jmyza6lPX0Rp7CH6WU2byQTS2/T42QyKE3i0xDJZesopc9veOigi0xiy6nIO4IfXrZ FALsFWVFN8K9GMZiyrRjUSIO0Q/0b+7f2BqRXIqXpOPgxDxRk/aAY3Byqu4Dwc16ds 5fld2Ga3vdKGBCXiN05WsFPJEsPqIM3HS2Bb59IoyFhg1X8Y/lF9MHgYhJ8FGHbUpH gMBdF8W8w3LcGEG5W1etRtY7pTyDtxzKPcxT/F4gUMRqF8iS1xEiJlvRlZBhj+TKgf qxiNnuDp9tqIg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 17/33] arm64: idreg-override: Move to early mini C runtime Date: Fri, 11 Nov 2022 18:11:45 +0100 Message-Id: <20221111171201.2088501-18-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6084; i=ardb@kernel.org; h=from:subject; bh=XcCalZgDu8W95+sS2ba5T6FJRehgpS6pDpztbl7asjk=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJFWD4sbYsFJ+cz2RpVyDwFbJ3WDantu4xZXnCF 5ggeV7uJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CRQAKCRDDTyI5ktmPJJhIC/ wNnBABmYHGxvJ9BqRzmL4Cvc5x7aWjegs+7MhOK4YZHaOSB0QLN6ZLSIgnlbhNLAlHBytCNPg5MVSZ +2pSy7yoXgGX8InKAN/L2Dai/vr0c0TstUyMKtfRSDCPomuRbyoJEp0J0Pm1KpVfuq/DMtiMK7I9Jh El9qWQQ7hJf2bGajiJpaywUhQmVIgSPIAskhXi1+hoE4Z5wn2FYjybh3Q+n4DzrviHd4vjoRg8DDaz R0uUtWXAk0D8/U3BX1JL02ca+zbziyXPl5xHWdX1mtQTCQhjUw83qZXijTy8Fhr4pDYfdQei4EwPlM wMbGMzOonCvbhlfTM4eS9iKFPmXd+xcMklAT+g6Zj3/NGG/S4ylcHUC6lP2qiZyN1RU8Dp6+gBH+rj KNyFRFHMi6bEPGlmABhySbeHuXoUeUtxN4yWT+Q9D2qBF7+g7OdDGLHl9jQQsVgGyRdJmaNPlm2MwG HIel2uLmC12fRbshtrju3omCJtJfHzkltyOv6Bixgp0jg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091300_755564_02D3DB13 X-CRM114-Status: GOOD ( 16.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will want to parse the ID register overrides even earlier, so that we can take them into account before creating the kernel mapping. So migrate the code and make it work in the context of the early C runtime. We will move the invocation to an earlier stage in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/Makefile | 4 +--- arch/arm64/kernel/head.S | 5 ++-- arch/arm64/kernel/image-vars.h | 10 ++++++++ arch/arm64/kernel/pi/Makefile | 5 ++-- arch/arm64/kernel/{ => pi}/idreg-override.c | 24 +++++++++++--------- 5 files changed, 29 insertions(+), 19 deletions(-) diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index a8717865fee5c296..2b003834c320c20e 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -33,8 +33,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idreg-override.o idle.o \ - patching.o + syscall.o proton-pack.o idle.o patching.o pi/ targets += efi-entry.o @@ -65,7 +64,6 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RELOCATABLE) += pi/ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 998a3e066b2fdf0a..786b7bd79a4026e9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -458,10 +458,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status - bl init_feature_override // Parse cpu feature overrides + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 8151412653de209c..6ff6efbc1ce98ba6 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -41,6 +41,16 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); +PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); +PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); +PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); +PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); +PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi__ctype = _ctype); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index e046c10606cb822e..47d3ffcff3ac9a03 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,6 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c similarity index 95% rename from arch/arm64/kernel/idreg-override.c rename to arch/arm64/kernel/pi/idreg-override.c index 7e3eb48f5c83d539..86d994424779bc0d 100644 --- a/arch/arm64/kernel/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -294,16 +294,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(void) +static __init const u8 *get_bootargs_cmdline(const void *fdt) { const u8 *prop; - void *fdt; int node; - fdt = get_early_fdt_ptr(); - if (!fdt) - return NULL; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; @@ -315,9 +310,9 @@ static __init const u8 *get_bootargs_cmdline(void) return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(void) +static __init void parse_cmdline(const void *fdt) { - const u8 *prop = get_bootargs_cmdline(); + const u8 *prop = get_bootargs_cmdline(fdt); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) __parse_cmdline(CONFIG_CMDLINE, true); @@ -327,9 +322,9 @@ static __init void parse_cmdline(void) } /* Keep checkers quiet */ -void init_feature_override(u64 boot_status); +void init_feature_override(u64 boot_status, const void *fdt); -asmlinkage void __init init_feature_override(u64 boot_status) +asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) { int i; @@ -340,7 +335,7 @@ asmlinkage void __init init_feature_override(u64 boot_status) __boot_status = boot_status; - parse_cmdline(); + parse_cmdline(fdt); for (i = 0; i < ARRAY_SIZE(regs); i++) { dcache_clean_inval_poc((unsigned long)reg_override(i), @@ -348,3 +343,10 @@ asmlinkage void __init init_feature_override(u64 boot_status) sizeof(struct arm64_ftr_override)); } } + +char * __init skip_spaces(const char *str) +{ + while (isspace(*str)) + ++str; + return (char *)str; +} From patchwork Fri Nov 11 17:11:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 57896C4332F for ; Fri, 11 Nov 2022 17:25:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gup/5xhMfZOd82IdjbWu3ck8OTwVG0rpzNNb9IisH0A=; b=0waLQqgEu3F7jD aCUWlptyLng8wHMYWVvICFpg1rgpfLc0PnOwZNn1lgJ9KWlkrNHxuYOMMeAMeXM19NSgjdu8ak6HQ IyDQNozbx8aPuvjdewZOvoDEj5VxAFL06via9oRKXBxYoQC5MZQGywftUrb9qBKdvG6LJTidYg746 uNDD9iD+1Pbz6RdVN11nzlb1XSwkXvDi9nok/l3CjokbJNRvfY3+rBbKG4p7I3v4AUAP9R+UCERNp dQBbjxJP5NGXDq2FtrK9kNGEWeW+XzU+N6ED6a6bRQ2tU9bw6RM76QyK9XC3U2px6al55cHg97x74 9XK79ayAwoA6YHSdgDrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXlf-00HE5k-Ev; Fri, 11 Nov 2022 17:24:40 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaO-00H7qu-SE for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:03 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 66A316206C; Fri, 11 Nov 2022 17:13:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EC875C433C1; Fri, 11 Nov 2022 17:12:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186779; bh=OKEi9S6PU3zfBkd/5gRT4Fws/KLATAvA+n3Ale7bVWE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XUUXGIsrY7FqL225vyB0D6D0xVeaQVLoLAuvhIyMaLQh49xgzAVaFNhYjlPjpk5oq rLsVQxI45S1T0fG5uDvMP50pGk1bZBe6dG21/Y84vIci8C+03WFr+Yk/rY4DuOxjSv t870kVi/m38DthA+RDE7yCC+rQcI4lJuHlpVBG+/WDPdD6fCkzqlbI4eIvkzAMpi6o QABJCZU6xB/h2d0U3Oq+W1ZeVIp88OAZ+S4kZumWeZEqQSKWmdxz1gBWfFr+6y/7RV eXkYANoDMQKNKbDaxrOKF4WAhzE0X73awofHor7WobaK1DO59XCEOTbA8lIiOQXi6x hD0bCeF0kI9tg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 18/33] arm64: kernel: Remove early fdt remap code Date: Fri, 11 Nov 2022 18:11:46 +0100 Message-Id: <20221111171201.2088501-19-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1349; i=ardb@kernel.org; h=from:subject; bh=OKEi9S6PU3zfBkd/5gRT4Fws/KLATAvA+n3Ale7bVWE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJGhIw9AO7pZv07ebBNQkowjElPHpb6Pu7XiD/G nFn9QLuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CRgAKCRDDTyI5ktmPJDS1C/ 9aNTx0M1a6EHM1Ma5ajuISunOZxYOaV4JRp1rb+p+P6Zaxg7Dgx9jdZYJm8aroESKVXHyWN5ihBJ98 SCwU12NzCgfmJBDBufjhRd7q9tMXPfApagHt5mTbnbMlEDZUSwn1hrp5wdjMc5nWNxObWM7KV7YVDT vwazexS1yTVWx9bhd139LbpKWeB6xUH4Qxbd1KU5gdsNg0Tb7eI9zw4wlmipod1ml/GQcg1r3OfFEJ mtFitCTWskzbilnJjMWvZRrHz6J44/4QOCkbztOJmsa8ZBmwjNgEG2oRMM4MJMOCILxicVnRvTKAUh 0ivN37ze6ikStwWCuWCQZj7EwHY3mkiKltQrj5s0QJeSf0AH9M/BXmyv3FB0u6Zgn6efYtAKnzHLID ghfz6LDCC470gpg8HgrFlGv+hrDZ/p+bO9J9d55wtzQNHt9GZkG+CXHNEzFg7QoZ3cwF4wJOIug0my F7GoBzO/kye4wQmcTYpU2U3Rx4ZAFd8nJuHXySTJDZdZ8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091301_006512_F3FC571F X-CRM114-Status: GOOD ( 11.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The early FDT remap code is no longer used so let's drop it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/setup.c | 15 --------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/setup.h b/arch/arm64/include/asm/setup.h index f4af547ef54caa70..acc5e00bf3b0fafb 100644 --- a/arch/arm64/include/asm/setup.h +++ b/arch/arm64/include/asm/setup.h @@ -7,9 +7,6 @@ #include -void *get_early_fdt_ptr(void); -void early_fdt_map(u64 dt_phys); - /* * These two variables are used in the head.S file. */ diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 12cfe9d0d3fac10d..37e0ba95afc3e045 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -165,21 +165,6 @@ static void __init smp_build_mpidr_hash(void) pr_warn("Large number of MPIDR hash buckets detected\n"); } -static void *early_fdt_ptr __initdata; - -void __init *get_early_fdt_ptr(void) -{ - return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ - int fdt_size; - - early_fixmap_init(); - early_fdt_ptr = fixmap_remap_fdt(dt_phys, &fdt_size, PAGE_KERNEL); -} - static void __init setup_machine_fdt(phys_addr_t dt_phys) { int size; From patchwork Fri Nov 11 17:11:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A580C4332F for ; Fri, 11 Nov 2022 17:26:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cP0N586VK5xIHik6oEEBZ/l3qIIPiO2/EDBN+IweqSw=; b=kgbyLNCj515pFT 17cXXLpWYJI7uuUkz6yysqa5lAKzgg+J97G+xWehDk5sOWrGkl0R/ozOpcRsRCPPKD67ED1/sMOjL ivAQUXf/JbdNIHBVPkjnzMjeHKKrehP0jAJuKn3nuPaTx/yivet+/BtCuDvgQGYEBaMiFKu+mw7Yg PRS4wX1GsMsRKkc3YS7f+/DBGOpfqWfO2Qqg7+HTDAzFKqDlbWsWTpxUw00rba1iers68Zz32aTfp yZP4SAAw3O1KhGsp5h/XRQ2twCQLPAKQ+HkBQeHGo+3H0e5De8ooVLsWXHjzbzT3ICreQ7oFAvuay 1t5Xe8St25Jj0TdHwBHQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXmR-00HEU9-4C; Fri, 11 Nov 2022 17:25:27 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaR-00H7sL-7M for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:05 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B199262068; Fri, 11 Nov 2022 17:13:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F09DC433D6; Fri, 11 Nov 2022 17:13:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186782; bh=47ONy3JhJC4HIAvQ6skMJjbwiDZp25Gx2Jf027x3lRU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XDUilcY6LwqxCAXAv9GPNEdxiOlbmSzMSM+1I/ThVNai4uYjoTaZ8l90yziVWjvPK S8miw0YThFwnxxs63BvWo2wXGedzP4EOOmkHVSdxxHS4heoJH2b2F0PWCzR4v3efN3 7YO402Xxrhps+aucTTwPrcFZtelgiKrfwCZRp3V9j+gkNq71iH95e5JSPeGNOVolj+ YqKvhIG7Aq/1UpShAoe5oftK0OnXFTp0NkLWSTKQe4md5Hfvk9XzzFoSU6eHS/tVZX 2ynyJva6fXmOJGTaxQ1s3vw/Dd8Als81bAiy/B+rMHeJOOS9OJNalPKG4v2eti4cW8 Bqb9n8Y8MPueQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 19/33] arm64: head: Clear BSS and the kernel page tables in one go Date: Fri, 11 Nov 2022 18:11:47 +0100 Message-Id: <20221111171201.2088501-20-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2739; i=ardb@kernel.org; h=from:subject; bh=47ONy3JhJC4HIAvQ6skMJjbwiDZp25Gx2Jf027x3lRU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJIg8xKEi1l187QYJI3mHvpUWHw8wbNqdDl7HXs vzkZ75KJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CSAAKCRDDTyI5ktmPJK5IC/ 4owhyNvqdGEKmANvkvY6KRORJRmPqN6Hw2E7QUvCYEKcYdemUCmBO3Q5EtQKVqj1VXU5FzDhDWTtJE J6fhXnbMVjxLQ8oMt6es9vz9J23M/RZuUReoOxVN/Yr1o77lbZXXwikvxxAOesbKuRZ3u9lfgwvo4p zIT+niqtIsND2y3/AzfqmKwxwn4BLon6N2zQi9tW2+hy0R3BT5avFX3Qi0vSUMistMqjhNWWOtNdOL Po4A2gAm1kLNrZgSFDtXv75z6kXotsZ1SHBPYP02po+hbaMoJgRZDvm1ORb9glGydHa6KC+1FNJYWP UDzXvXyv8J9UJ8KOu9aPrRvD4T6x5DsjHrt5OUIJUxODI23yk+fpWg/pm/lFvFRhxEgL/e9AGWfsQ4 dhC3d7FE360u9t3c3oI8GKz+GD4TWNFoTk3sApvn9VySF6C64MCK9Q3hlRB6rlWItNhwiuui447PCz kMuRhj2cINOHIJmMSgCG1wnZoTlWOv95JLosxgXNA9FXU= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091303_463995_AA320DB3 X-CRM114-Status: GOOD ( 14.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping. Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 33 +++++++------------- 1 file changed, 11 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 786b7bd79a4026e9..0e7aaa65ea174efc 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -126,17 +126,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b dcache_inval_poc // tail call SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory. @@ -335,9 +324,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov x5, SWAPPER_RW_MMUFLAGS @@ -437,14 +426,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value @@ -708,6 +689,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1 @@ -722,7 +712,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir From patchwork Fri Nov 11 17:11:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8D6B1C433FE for ; Fri, 11 Nov 2022 17:27:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mB9rurDSkfnIb5ZmLbTAbyZIUNROBKBgPqXgBfIWCaY=; b=lAJIk6l5iaiQZ1 q70De08bQ4b9cYxO0XubNolNQuPZ9QEgqUhM46U7HE97Ow4uBwIJs4zmxPCXmsOjVRO96oRDgcGQ5 8xiSwPaeYNRk5VUDF5O0SZNVGj1gMFkYwyVaAOuU0WmWzfMODZtXi4w+HHpQwFpYF6tWuiCUuz8Uz hldnizC/2YJni9CJ/kMzosCceS20jlrxcJy7X6QaMm5xAbVzG8PhiMp9JwYlZ9NuNYiZ1R4TOI/wA WoMEyRDUmrAtr1y4wf1qea0u8SMEcNQPSyd/rrTHmUeyOZtuu86MmrCvYGEiJpj5xNgrBeGcEcl6n vuwLB2LZiaYkHKgRLIVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXnG-00HEux-VC; Fri, 11 Nov 2022 17:26:19 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaT-00H7tp-EL for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:06 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 02F3162066; Fri, 11 Nov 2022 17:13:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 866D3C43143; Fri, 11 Nov 2022 17:13:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186784; bh=Bz7DmxM0QyWxMPOZV5SEwQziNrpV/7poJwrAfXfKW4k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=THaOnaEMFhlLxQtp033zb+rvGgfTbTdJI8W+fQco/ow/oV3qOyv6f3vHRm4ZPMtYz YVTKubmiE2qwI1TrVe87WyIL1jIx6nqikaaT19tfoSqRttG85Zv0pabpvxMxZFbihq ThDH79C29a4GaehZ7znTru1GBjpxnhgDzdTtt2gtyoOxPeKHPNEbtr71483V9xwzoM OVoraXwIJ4CVPcmMm9n4gq6JHOIk0Kl8tAgD0faf7JzE9RbmyMoNGtxuSoOzI0STtm vSKBWvk/IlJs4vIwKh7GVYnrB8NMxE0jsAUz1xdC3iPl7zvnNlYzFdnoMZ1He96saj RkKA1PjaDz6GQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 20/33] arm64: Move feature overrides into the BSS section Date: Fri, 11 Nov 2022 18:11:48 +0100 Message-Id: <20221111171201.2088501-21-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1664; i=ardb@kernel.org; h=from:subject; bh=Bz7DmxM0QyWxMPOZV5SEwQziNrpV/7poJwrAfXfKW4k=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJK5gpkGhmIOcURr17g+CBraJNrGoW+j/BW4eeT MctXkQ6JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CSgAKCRDDTyI5ktmPJNcIC/ 4mbOU5Ha+njblnGHJGZIhL/BM6zBh5bomY1wwiVvjycu6br5EcaIgjcF2w5HtDHvb0sWPzjPp1kgfR PHm/nIdJDYFfbxJGpX1ZJVEYtGnVmX2/OICQFfuTRu4UA5U67RjnmBuEm137RI8iU1k2NyB+vrVWes urrdHk/COZs+3rB8KylxzRwEggTGdPzYO/nQSvpuDcMWZEBWiH42VWc/llDLY040J5MpicGqho0Mis CJjMDfJIcK3id+cvDtyNv6nODaXiB8sL+6T9B0wO8joQIRVmhgst7hTPIUsDl8tfih79Z2qLDpfOBO F9m3WZse0uJcuWhkkw8C1nQDreLWRf9me6b8Qj1Wlb8JSbsy2dRLO9bO1nxA0QNQta7AcdM5H2MXFN MJP29tOm+kIuGpA/eQ7+IieDJY37qmWMe15vt2m6/HU5/hLn4UGUciD4LjLxWoQVFjuDIfd5VIsJ6a uAC0BYel0YGYOTQrdMyQc/925lTK5Xwjz8kUwevsfNMBs= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091305_589487_6EEEFE36 X-CRM114-Status: GOOD ( 12.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to allow the CPU feature override detection code to run even earlier, move the feature override global variables into BSS, which is the only part of the static kernel image that is mapped read-write in the initial ID map. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index ebd8cabffb105e15..08ab04dc9393652a 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -636,13 +636,13 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) -struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64zfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64isar1_override; -struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64pfr0_override; +struct arm64_ftr_override id_aa64pfr1_override; +struct arm64_ftr_override id_aa64zfr0_override; +struct arm64_ftr_override id_aa64smfr0_override; +struct arm64_ftr_override id_aa64isar1_override; +struct arm64_ftr_override id_aa64isar2_override; struct arm64_ftr_override arm64_sw_feature_override; From patchwork Fri Nov 11 17:11:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35436C433FE for ; Fri, 11 Nov 2022 17:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=piFqmlrdBPzkcayx3KyAntKc+RaCJkM6u6JDMPvH4KI=; b=ywOmwXHhOYbtXj qyFtvKx7VD1XjNdE97gX7dJgNHpnr/KILyrFBo5Ba3+v4lTaxUckl9dzHoTY4gzxTQ2tCRhPRvKhp WD3OLDAoQrHNfBhzTWcWDHyhghKjM9tDcPYWNiGMe3yg+VkJAc+yzYYd9QWVsFsTJjRJNESGlCaBP v93agOfTsLY9BQUvSFZN/zuUsftFsFYF7MfhtC9bF4ubAHNbZzPc7tV2aUyndFqVydHICT9vqx3tO +LYZjxT4rBY7loYNJuQHv+lOJ0dIJZvMKGP8mfxO7mCOYPvr+6FogO6prIiZVcOULnukU6zOJMiRT aiJMiiwEvMXmo3UeFThA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXoB-00HFPq-EL; Fri, 11 Nov 2022 17:27:16 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaV-00H7vG-Mv for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:09 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4BC4562068; Fri, 11 Nov 2022 17:13:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CDB9AC4347C; Fri, 11 Nov 2022 17:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186786; bh=yeHmNwCi2DjXiIunUjsDLpeU+2aibBavxEszHoOh7S4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ajJdHNRRSoqQE4wlCnC8pAdyNSh2Ca1l1fH7eAybGFecHYq6N4b4GFRe0loOP7+th vU52TFzp6GPLr1zyJdJFS9LYqHHp67JtKiQkrABttPCQWlizOA+agTxXMqnE0axZfx vjcJ4eV2hRHhmggph4En18vqxJKUM+DpN+pzpzdLCwkTBJBSXhIUvCavq/wUhfbplK zgZ/risbYC+jbFDtTeQGLP9I/StGjLmQvbRCpR10MZLw4a2eBKx5LWvJfpblgP91xU GaNFdsYA80OYc3bBVTf+1FPpB9EkgDhKLG/oXMPsULYKARmA4eS0FDuxuz9+2tT0Fs DUux8TyWhe11g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 21/33] arm64: head: Run feature override detection before mapping the kernel Date: Fri, 11 Nov 2022 18:11:49 +0100 Message-Id: <20221111171201.2088501-22-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2887; i=ardb@kernel.org; h=from:subject; bh=yeHmNwCi2DjXiIunUjsDLpeU+2aibBavxEszHoOh7S4=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJMWTQ9o198WCsxSUOlG3E8/hL1g0EiQ6Ba7/AC Xf7aLiOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CTAAKCRDDTyI5ktmPJKNZDA CTmcY44I9GBzYmpyDiRZdnmn0aPAPP/dyuvv44Y4hEDtX/ABpmt9pxU84d7PvrDMx+NgsRWJInxoKo S0p5Wpgmm8Xc72geCVR5oCiydmby+lSDFsTXc3no8L6QJoTBxCnRJ2pyFYcyXmWp6LvBmtNb6NTqqd Of1YNycrUOtKGXbGf/6e2un646jl7x6DxlOzayahRyIm0zRyss6g2tteIjCulnuIsxw4ZO1jCQXbyE Ox3C8VtkDIGNdTuciprDznL+GQVVrLt5N1TR2/tsfKHe2aRRz9/Dwv1uJ0X9h2HG8OD+Sd8VrwNFci jDI0RMiGI2uZGwc8TdipO0SekEAyvAMBhCMUdBpzh21D+ptOg0qDPzh2wRHCrD/6uTLlxWqh+taSeA KNmF92n4ME7NPlAzgChbabqBAZpBVM0BMQbj3WRwB5c5Z3r+ejBQqLYwPsG44eacLqx2pMN+2XhB8Q 3z0glfu82tlLHquz4SBfXOE/m22HiRC0qiuh3FBXRc6N4= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091307_874150_0A2C1FB3 X-CRM114-Status: GOOD ( 14.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To permit the feature overrides to be taken into account before the KASLR init code runs and the kernel mapping is created, move the detection code to an earlier stage in the boot. In a subsequent patch, this will be taken advantage of by merging the preliminary and permanent mappings of the kernel text and data into a single one that gets created and relocated before start_kernel() is called. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 13 +++++++------ arch/arm64/kernel/pi/idreg-override.c | 2 +- arch/arm64/kernel/vmlinux.lds.S | 4 +--- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 0e7aaa65ea174efc..9ea7f4e355ef5849 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -439,9 +439,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif @@ -698,12 +695,16 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __pi_memset dsb ishst // Make zero page visible to PTW -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 adrp x1, early_init_stack mov sp, x1 mov x29, xzr + mov x0, x20 // pass the full boot status + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides + +#ifdef CONFIG_RELOCATABLE + adrp x23, KERNEL_START + and x23, x23, MIN_KIMG_ALIGN - 1 #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 86d994424779bc0d..c21d1e9f43a11ba7 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -20,7 +20,7 @@ #define FTR_ALIAS_NAME_LEN 30 #define FTR_ALIAS_OPTION_LEN 116 -static u64 __boot_status __initdata; +static u64 __boot_status; struct ftr_set_desc { s32 override_offset; // must remain first diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index bebb88daf4c52039..3f86a0db2952600c 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -317,10 +317,8 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; -#ifdef CONFIG_RELOCATABLE - . += SZ_4K; /* stack for the early relocation code */ + . += SZ_4K; /* stack for the early C runtime */ early_init_stack = .; -#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin); From patchwork Fri Nov 11 17:11:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5661C433FE for ; Fri, 11 Nov 2022 17:29:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HoJD/+UFSv7jSMA6dUhw0rQVI2QOaAPRw/9QmpVXqiw=; b=N/962YP3+40l3y 6Q85UfhhMiCds2+nSNy9iFtLb/eRBwA3/pV+DR7BsSixxLWnB4356k0tbmXTQGDQq0ihAoF/DWx38 gYKUntv7K1EImNpbIXAaugu6HjcIEFU5gZ/4oxcGxyH54jC/MDutoNlPiTegvEeDRl7/bNZg7jpqs 7cgaTt4apCa8sNwN2fE+PY6i0KZlhgmg7neYgnNlJ6VbgsdSTZ45WFSMIm5QO2DDhfiOJ6knkLj5E bjdAlgaZlZagBNWUbvrF/VvYv8u3shG49vA7o52J7WTNoO3kmD5r5dY+ar2l2DJKtWdNhgGDYlIsm kNmFRrA0sNI9WnEiBgbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXou-00HFos-AC; Fri, 11 Nov 2022 17:28:01 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaZ-00H7wv-Jc for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:14 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4E663B82680; Fri, 11 Nov 2022 17:13:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 226FFC433D6; Fri, 11 Nov 2022 17:13:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186789; bh=79UT6hrkKbxqKInsN7QsQMwompw/Ew3GtJHjIs1jLXU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=robjUWvcvm2msA2fwxvQGQuv8ztBuVy0B4Fh03OD81lhhFl64GtSFvkpV+u8zQua1 4R2JrM66Rnmg8M0WMeJ7Cwd17REfEo1xaq6T7Ga41jVc7+R5B6MjPdBiDT0gC5dW5K P3gMBmPhIwO876M63jAB+iVt/RWRddskpBXDlAtzzdkaK72FKe1mOV/CFZ+GL3zGUJ 4jhAFq4bhtoWbSF6a3tnzoyZk1BxZ6Dp7erGnQbMBtdIbZid8mmfX+VC9XSWypwd6z xiB+T7eHoaWgIQiynawTAyzZ1xl6vQsomihuFc1W4/W8MyXTa4HzQMqxLjYa56A3eY uA+BUUBWqGfqQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 22/33] arm64: head: move dynamic shadow call stack patching into early C runtime Date: Fri, 11 Nov 2022 18:11:50 +0100 Message-Id: <20221111171201.2088501-23-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6196; i=ardb@kernel.org; h=from:subject; bh=79UT6hrkKbxqKInsN7QsQMwompw/Ew3GtJHjIs1jLXU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJN9O5RJSizh1MbM4rjnjacaPyD2X6qu3TAZnBu pqUNAZCJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CTQAKCRDDTyI5ktmPJNT4DA CCp5fSQ7DvQq1OdZetq/pegK+l3TJm25nV4Ddmfl9qSu1q8DvvCcpzMyrrOuXsl8DTobP4Ie82o2r6 RWbPY9LyzAeM5c8S7Ypg94DIB/fg6VdDRJvugTqPOyFXEBc9ePq/wnMRQJwyIJIHjIZ3dIlX8HNTjL FSfskBEvF6z7WeERcQZQPGiMSFiRyVrMugwxyO/GGrPoAZ4qPZYKaXfsKaa76dwYDW2vqrWhwJJuW9 x1UwMvY7D3Ts+VXAsdpjfw9ltSbvwCxovAn9zTq77GthbDa/q/OWb9zC+NKzQ8cwoxD++RvGFS6J4o C7Kap/Lv8D7x2JrpavDzT5pjYYsNFuf1PDVdqhqAw0d6o3QHS0qlvKCHYy8qMb/iVRRQnaZvy7WSpf bYMcyckLYp3JfN0ev/5dS8LICfhJXuiKhk3RVBXpGEMiG7LRhwZb8MFWLA4KE2Ieh0dbixKXoDKil6 fn4/fXx0F4QUlQAE7GkRTVT6mOlbWkKfhZzmjDriyIBUI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091311_988697_4B92749F X-CRM114-Status: GOOD ( 18.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping. So move this code to an earlier stage of the boot, right after applying the relocations. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 2 +- arch/arm64/kernel/Makefile | 2 -- arch/arm64/kernel/head.S | 8 ++++--- arch/arm64/kernel/module.c | 2 +- arch/arm64/kernel/pi/Makefile | 10 ++++---- arch/arm64/kernel/{ => pi}/patch-scs.c | 24 ++++++++++---------- 6 files changed, 25 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index ff7da1268a52ab79..d20dcb2a1a5284ce 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -71,7 +71,7 @@ static inline void dynamic_scs_init(void) static inline void dynamic_scs_init(void) {} #endif -int scs_patch(const u8 eh_frame[], int size); +int __pi_scs_patch(const u8 eh_frame[], int size); #endif /* __ASSEMBLY __ */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 2b003834c320c20e..35354365f5f39f3f 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -79,8 +79,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 9ea7f4e355ef5849..5f1476c0f3a33d75 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -438,9 +438,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible @@ -720,6 +717,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c index fa7b3228944b325e..ea505022737d531d 100644 --- a/arch/arm64/kernel/module.c +++ b/arch/arm64/kernel/module.c @@ -519,7 +519,7 @@ int module_finalize(const Elf_Ehdr *hdr, if (scs_is_dynamic()) { s = find_section(hdr, sechdrs, ".init.eh_frame"); if (s) - scs_patch((void *)s->sh_addr, s->sh_size); + __pi_scs_patch((void *)s->sh_addr, s->sh_size); } return module_init_ftrace_plt(hdr, sechdrs, me); diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 47d3ffcff3ac9a03..293a04a5dc3ef516 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) diff --git a/arch/arm64/kernel/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c similarity index 91% rename from arch/arm64/kernel/patch-scs.c rename to arch/arm64/kernel/pi/patch-scs.c index 1b3da02d5b741bc3..d15833df10d3d4c6 100644 --- a/arch/arm64/kernel/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -4,14 +4,11 @@ * Author: Ard Biesheuvel */ -#include #include #include #include -#include #include -#include #include // @@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc) */ return; } - dcache_clean_pou(loc, loc + sizeof(u32)); + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_CLEAN_CACHE)) + asm("dc civac, %0" :: "r"(loc)); + else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); } /* @@ -128,9 +129,9 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, - bool fde_has_augmentation_data, - int code_alignment_factor) +static int scs_handle_fde_frame(const struct eh_frame *frame, + bool fde_has_augmentation_data, + int code_alignment_factor) { int size = frame->size - offsetof(struct eh_frame, opcodes) + 4; u64 loc = (u64)offset_to_ptr(&frame->initial_loc); @@ -196,14 +197,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, break; default: - pr_err("unhandled opcode: %02x in FDE frame %lx\n", opcode[-1], (uintptr_t)frame); return -ENOEXEC; } } return 0; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch(const u8 eh_frame[], int size) { const u8 *p = eh_frame; @@ -246,12 +246,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size) return 0; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) { if (!should_patch_pac_into_scs()) return; - WARN_ON(scs_patch(__eh_frame_start, __eh_frame_end - __eh_frame_start)); - icache_inval_all_pou(); + scs_patch(start, end - start); + asm("ic ialluis"); isb(); } From patchwork Fri Nov 11 17:11:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24F60C433FE for ; Fri, 11 Nov 2022 17:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AoiUZ5R1nyfspTPoGzxXkJ1KUcE6CzOiSxL8a/QIWdo=; b=iDynUc6lX2oQpB xTDzXMlBDjBbhXfsM7KKAQzdTbBYZYQ0ZJSoOq0PmbeDnSCgdAtSUbZTfzxeRR3t2V1kEO60uh2DO fLszctco4wtALXs8Do07w8kSb+5/mKbyZWNeIfhP44Vi1RmtPpTBgCzzPHJHLV/n6oRtzqomGUGTj 40x4EyzNeG2VS2ZZCIRCAphuOqoHL0fVIwG1vZ7yA0XgVcgsLRAM/cb0bINpV/vQhEQ0Tk8YpPtF/ IAnWoVro9lFcd1XkKy/mlhnuI/g/4mGcRvyDjBB9EmmRtNAoZjGey0v98rj6xHQnd4V4bKO6yLixN QetOw8/Y7u4rbWZpu5cA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXpe-00HGCE-6A; Fri, 11 Nov 2022 17:28:47 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaa-00H7xc-CU for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:15 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D79C66206E; Fri, 11 Nov 2022 17:13:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A211C433C1; Fri, 11 Nov 2022 17:13:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186791; bh=UwSljH0yp9SP2lFi4DpK22SZSHyqwcwbbyDeK8R76ak=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GSNhnEquh52jFrYFjwCphP6Xq6aGfTY/C6FPvcM5XtX6j74PRF6MkF89xWeKK0waf c6Emzqye1AsgZbaJFmrwGxl4lCaT9fmkcqgYwnbn6R9A93jdWutHS9nGZZ9euLhgwH YRXMMMCvb7plfjhq0NQJ4FtXLdTy89ImN+qcwT90jN6yrfJY3+h/y3TGd+qoMD1kSi hooPR+is3ipG2MM0khqVj3dl31xKxHwi1BuZLW0nO9h/YvOSzdMKWhYyynI6TotU+i BntWwpeG33G7uii1KlDDPYaurq5b9Ne9kKKfJHfw+8eA8vZhIRBX/HOtXtRD1BeVsz mFxPterRa9ZRg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 23/33] arm64: kaslr: Use feature override instead of parsing the cmdline again Date: Fri, 11 Nov 2022 18:11:51 +0100 Message-Id: <20221111171201.2088501-24-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2146; i=ardb@kernel.org; h=from:subject; bh=UwSljH0yp9SP2lFi4DpK22SZSHyqwcwbbyDeK8R76ak=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJPgthlMhrmuyg2e+OWD/9rgP5uAAAl42jx7mLo RDeCXROJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CTwAKCRDDTyI5ktmPJLayC/ 48FIP4j2hq90coaOxh9HIr+6tGzrCfwWrRph6Og1vCHBz8595jd0UhiPoK9NcBJ6JWuUjKJLUgdaHK uKs3MKJ50UHMiQHo/hN6NteUkwwmLO+44Csxjs5pmwQRlq/Cmi4FJi/qhKmQHba4QXF6tXZc9oCYC2 w4B7H4GxPytO/zyUnxHUa6YAMahunGB33cR54E4TOEG/4Z3h8HOEZb+jfA8coUTJdhphcpwD2+97y1 zzRAw6C9nZGiQfkJtjtR9D7Gxg58MIuhbqadgFvrXa21NCjmG3sl5jKSwmvQui1SZYTmsjvLObCKnR RVYLNZZmaaOYMopGy8gXcr23yjc6KZ7mD0L1cgyhC6fIjqDO864aC51cR9vosbGAiAoH20P448SyKX ZavoLiqh2VPDcCZ6Q7iB01TbEkZujw1NjpXwKpVxoFDEZpGVmcPJ3wVCoCPaCFi+szxMmG0uDNZRGf wcfoAm29tay7W6rBC3u8081KMooFLUK3kYXnTNlnZh1Dg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091312_590008_A9852E81 X-CRM114-Status: GOOD ( 14.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The early kaslr code open codes the detection of 'nokaslr' on the kernel command line, which is no longer necessary now that the feature detection code, which also looks for the same string, executes before this code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/pi/kaslr_early.c | 54 +------------------- 1 file changed, 2 insertions(+), 52 deletions(-) diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 86ae0273c95016c6..934e95fbd4278d0b 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -15,57 +15,6 @@ #include #include -/* taken from lib/string.c */ -static char *__init __strstr(const char *s1, const char *s2) -{ - size_t l1, l2; - - l2 = strlen(s2); - if (!l2) - return (char *)s1; - l1 = strlen(s1); - while (l1 >= l2) { - l1--; - if (!memcmp(s1, s2, l2)) - return (char *)s1; - s1++; - } - return NULL; -} -static bool __init cmdline_contains_nokaslr(const u8 *cmdline) -{ - const u8 *str; - - str = __strstr(cmdline, "nokaslr"); - return str == cmdline || (str > cmdline && *(str - 1) == ' '); -} - -static bool __init is_kaslr_disabled_cmdline(void *fdt) -{ - if (!IS_ENABLED(CONFIG_CMDLINE_FORCE)) { - int node; - const u8 *prop; - - node = fdt_path_offset(fdt, "/chosen"); - if (node < 0) - goto out; - - prop = fdt_getprop(fdt, node, "bootargs", NULL); - if (!prop) - goto out; - - if (cmdline_contains_nokaslr(prop)) - return true; - - if (IS_ENABLED(CONFIG_CMDLINE_EXTEND)) - goto out; - - return false; - } -out: - return cmdline_contains_nokaslr(CONFIG_CMDLINE); -} - static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -91,7 +40,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) { u64 seed; - if (is_kaslr_disabled_cmdline(fdt)) + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) return 0; seed = get_kaslr_seed(fdt); From patchwork Fri Nov 11 17:11:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35FA9C4332F for ; Fri, 11 Nov 2022 17:30:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=elqNKG87J1QNs/JFrUFcIlOYJLQzbKT9JjPHgRHhlTA=; b=ThO8jI4EaKsKzq BcAN/jZ65KImdmvAo5evYDMmmngeg/OIguOjXsn5MSgSlIUVaHbcXdWNTZONhzXlk7jv9MwZcxCrN Uik3nE19A/6APrZCS9NGGtb/zqxAweGoA2Vns+7vM/mLa6g2KVPgoo8pl7WlV2dlMINIG4Hm+Am10 v3ctLQP3UMqO1kN5MayoT9JLIJ4muNBroEexYy9Q07JogLMnOrP1amzXbXkNUfhPrwKK41UmLag0i HI6ljmEEmJg7Ad7AcvF+9+ekm5sLYjtCT19NhrUpyJId8SaDHqOQNTZBGYI6P0sL2t7rWpfmh1W0x pf+W1shOgOE9TieeaNww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXqK-00HGXo-B6; Fri, 11 Nov 2022 17:29:28 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXae-00H7zF-87 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:18 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D5C2DB8267A; Fri, 11 Nov 2022 17:13:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B1748C433D6; Fri, 11 Nov 2022 17:13:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186793; bh=2L8qq3K6u+GRcQSYfqhjGnpNpR+PijC/5NMw6UtzX7E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bsPODVAi6HtynEkFOdx6tFecrOq5h1XJZkShmdAdQwu1ZaxIwOUOgtciSw6y7kdG1 lI1ZwIAfxZ7Vwfd+JawbvrDpevbjlUVQJsgwOEmN37r4K8iQr04gp8cYdmOfwWku6j c6Kvo0lcElavm9DueL9hPeLTz9J8SPFgH1CE1GZ0Dp8RjyzxOfc2RvgM0Mlu1fsB8f 3RHcl/utp0nEllQxz/0P/MqEUSAknv5ScYzbxAy8PK8hoqzkFBATWyOZlNOMu9kcPB 57PIR/YlSQKB7/k/1j7SlPIsM0SOoAZks3TPCfks7ff6nUNvajkU9H3jWcINd2Emyx 0V2LfC/uHC1aQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 24/33] arm64: idreg-override: Create a pseudo feature for rodata=off Date: Fri, 11 Nov 2022 18:11:52 +0100 Message-Id: <20221111171201.2088501-25-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1642; i=ardb@kernel.org; h=from:subject; bh=2L8qq3K6u+GRcQSYfqhjGnpNpR+PijC/5NMw6UtzX7E=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJR9BznuYr1gP5ZGeSxJ1m1etUnyN+MXK0zCmiJ cNGUNhmJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CUQAKCRDDTyI5ktmPJHbEC/ wIfUMsg5/cxXsuWsiB6GAgyghFDs6xXCUscA8aOW8x3jtYnzTYNq6furkWuortWtxE+ucxJSXGFwti 7T51YFMItH7xi2D6n8awEFBGOANJ/rgciY3Z2oqqbXaSEAs7ssJSCxjis/85frZJW+o0cZywVuAkAG u+XXCegn1OOcDYlkVTp6OKT3/a1NzZJ1LYODgzkHz+DHANLiJQjU3pZ72i1TQ5FHSKrLZzu1HC1MFx ioYFeXwF9UOiQs152ja8ajWWSQKkhLCqnjffBBA6Evkn6TinUeyR7I7FS5QydLPV3KwpHB00A6NrnV pJz+xCYmAJmRJW6wzHRT/qNoiyMAdxaAdwvSG/4w2RtUlPVjb3OOlXxjmyAMyJDLm3DikV8uRENFCB +0KahcRHAluRZcskouIrGevzqdrnCsGVfmlXx35lC4wAOcQG3UOf8weNdc2TywmLIF7QzYKoPfljV5 9nRWhLiK7HA6sMIzpoxCcDecOP94jRBzeqxEQPjL0IcTA= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091316_591641_38E4FF72 X-CRM114-Status: GOOD ( 13.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add rodata=off to the set of kernel command line options that is parsed early using the CPU feature override detection code, so we can easily refer to it when creating the kernel mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index f44a7860636fd411..b8c7a2d13bbe44e2 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -16,6 +16,7 @@ #define cpu_feature(x) KERNEL_HWCAP_ ## x #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 +#define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 4 #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index c21d1e9f43a11ba7..e524d36d8fc1526f 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -133,6 +133,7 @@ DEFINE_OVERRIDE(5, smfr0, "id_aa64smfr0", id_aa64smfr0_override, DEFINE_OVERRIDE(6, sw_features, "arm64_sw", arm64_sw_feature_override, FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR), + FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF), {}); /* @@ -164,6 +165,7 @@ static const struct { "id_aa64isar2.gpa3=0 id_aa64isar2.apa3=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, + { "rodata=off", "arm64_sw.rodataoff=1" }, }; static int __init find_field(const char *cmdline, char *opt, int len, From patchwork Fri Nov 11 17:11:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E975FC4332F for ; Fri, 11 Nov 2022 17:31:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sSojMrNuHCqA9vo3elYfcq7VtgN55VlMF7S4IR4u0n4=; b=G7l2jlErNRenZJ sd8mQvPTt5VBgTKM0Iaq/R0bovygIEWKz8wD4uqLbZ28EZv6qhbe+fnnJY57ojOvRm4w3nNo/V8ca 4WKlh+wU+neFfSK9bSFW15mDGF8OTwW81ybg0HylQkazulCKjYtXFHdezHJ750vv9/3H+tV6hiXLZ HbE+78XmbhmXBM5gvqhougz6iKDvpuDBtwT1D11cf6sPOUkrwLDD/ldQ7FdwMJUdOaJjsv7TOy4xU AFfKiP4kQHCr1AAnBbODZiOumOoVQTmfSw1NuPuTAB34QOHOJSk4j18/KhXZLGufycsA1PjtFkyb9 Dx5aGWWCuIVm4xc0CPaQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXqy-00HGu4-HH; Fri, 11 Nov 2022 17:30:09 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaf-00H7zp-7R for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:18 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 753AC6206D; Fri, 11 Nov 2022 17:13:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05B70C43143; Fri, 11 Nov 2022 17:13:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186795; bh=1v3DWjp9u6JUJ8R6/MzaiaJY023C2XcQtQfQsZ68rbk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z5mCV13tJis9u4QL4RE2BbQ7E2n1RRnyb6VcjXkkxhN5eCU01H1xoVUBXDiuGyG95 2b/PnyKFaoX6OUecLHr+FCrqn6v1/VM/VMsVk2yEK9W3SYxToNlCx7w0fN8FoCtIIK /tJ55fPUv+pNnE3uFA+Bmu8PmLdT5447AbThLLCFEFFJhvBltflyNzZzJtCzkC5RKf xmHRmg0Q831NMeQ5ClDOk0A0tyiDkHppCjdaW4QgSM6F8NYKSC+2IBJ6YdBFK7wSo/ boxzvDUI6HIyuVcfcrmZaTck1ygp6uaYhXcuUQ+OjUKiNVfc+Do10ndHOrl+uh+HZc ihNUQOk434iiQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 25/33] arm64: head: allocate more pages for the kernel mapping Date: Fri, 11 Nov 2022 18:11:53 +0100 Message-Id: <20221111171201.2088501-26-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1825; i=ardb@kernel.org; h=from:subject; bh=1v3DWjp9u6JUJ8R6/MzaiaJY023C2XcQtQfQsZ68rbk=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJStHfyR8xmljYehRL4CmLUPlwJlw2nbF4jbjQb etKOSiiJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CUgAKCRDDTyI5ktmPJA2mC/ 45YmtZgKZUJNsPg9NVRBRFfqA0EOSPz63gKxA2sBny8GoH87eQTamUdke9S4uLpXG1GXQsOkapNa1H rAbJty072ti3NmKY1uiLY949EO29XxraS5LQddquu2nMxTxRdFub9/EgEN+1ebhFPW48HxSBlavVpE Hz1RsemrZkJ1Lq9BdJXtiDQbntJPYndDiqRi6ANQ1dFCOROOwnjbDb/lshrXBV45/JBZbRET2ghKxk ziY9uTb+JSM2Gp/wObvUZCiJ60gZmQmZNB3mlBTlLIbOz1mp2fn+tIvjtmj/X6QAJ2AFqjtxOVZw90 QZ1KRaTrdWljgiaJ5dSGgk2f2DvjGQa4iO+pi1trrdZe9RuBKa0Sta1SvV4s1mPDxlUKwe2Ivwaavx la+uxzQSZm+YiTuclB53CLxEu+FmCfOWnL1w5/yF7FwgWb1TfjD9wYsdMzEQfTJBpAI0u4jW7Ulr4C XbV1Y7dW9pF8441pOMlOhDwaArT0Rhf8XJ9NmSGDysups= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091317_408213_73D1B991 X-CRM114-Status: GOOD ( 14.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for switching to an early kernel mapping routine that maps each segment according to its precise boundaries, and with the correct attributes, let's allocate some extra pages for page tables for the 4k page size configuration. This is necessary because the start and end of each segment may not be aligned to the block size, and so we'll need an extra page table at each segment boundary. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 32d14f481f0c3f37..ed0db7fc0022d34e 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -85,7 +85,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR)) +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS < 48 @@ -106,6 +106,15 @@ #define SWAPPER_TABLE_SHIFT PMD_SHIFT #endif +/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ +#define KERNEL_SEGMENT_COUNT 5 + +#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN +#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) +#else +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#endif + /* * Initial memory map attributes. */ From patchwork Fri Nov 11 17:11:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040646 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A1DAAC433FE for ; Fri, 11 Nov 2022 17:32:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ejKy35skR77JSEQ2TCt7nkX18oXs/I7uohLNpDcuCS4=; b=DHzLX2CD9ZbdNX 7qlFKG3rPcM7FAQMGpvGEvgOCB1fVFgUzo2MwZr70Ka8X034ZrDIv4ZmvmWfkavrS+xcJCDBLE9ni tZzu0dwMSoCiYjDs9Ciel8DGK1SLk9j3nb9IxC3qr+RV/ZEVrlcWYr2H3ac6WiFKnRw91mipgw9Nj iUyCTCa55p4THMxEzkRY5LocRQ3Pw++NYpONYS/Z8rMIlernC3LXrY8PsY8bbxVaPmxhVEUH8jzVr U+ca+O0KpLii6aX+q1lsCF2tnpO7nrQqMzj8XFUV9pUQw2ilsjxOik8JnJArp6uwub8phuf3Om3+R ZKjqWeRBeH4O5rKB/5bw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXrm-00HHLl-36; Fri, 11 Nov 2022 17:30:58 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXah-00H81v-96 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:21 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BA2176205C; Fri, 11 Nov 2022 17:13:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4E71BC433D6; Fri, 11 Nov 2022 17:13:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186798; bh=t0leZ0gGl6DxYzu8y9a9G4oLsChLGyL399/KcQIFWsw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Faz+3LPqOTnV+Qdjr2QAy7APFNrwQiWNLSCpLYB3tfKmVtuXXd7+wHUS6tzQYInXk fB7OeXy4Jr73lwzAtrHLqOp4cgdylZ5DtAZ3VVViZvQpJVXKaK/yEEQBYRl1Itwmb7 08mCqAv2+veAuUPQgTFijbQ7fz1q0WUxNMQGICisgAcg2pXGed2K9g143nTCM1d4SD 8aHPtZMXumVzOPvC/j+lT+bTTDF7woKx+vRWSCuF1aJXTU2NRu15dkjBhJSbe2L8i+ wRtbuvG3vdhY5u4hTWvcjPX0OHpaqlYOupPaHHDvzyyNmUYArCyHYBezXmWM+0RjcG BF9cyws4JGj7Q== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 26/33] arm64: head: move memstart_offset_seed handling to C code Date: Fri, 11 Nov 2022 18:11:54 +0100 Message-Id: <20221111171201.2088501-27-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3986; i=ardb@kernel.org; h=from:subject; bh=t0leZ0gGl6DxYzu8y9a9G4oLsChLGyL399/KcQIFWsw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJU7SEjZB7GJlOpE99yYBCB+gDbPNoB00vK4hRp WrgRociJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CVAAKCRDDTyI5ktmPJPjWC/ 43d/dPYLBUCbD7rS/nMbvZmxDrK/r6bIe5jJ15NYks47O70V3jboIrGoCMEuwB87nkF2xhfEcNfvyW eJKr/HTHDJjHEf3AIc9yQeejU86bEzme4uMOi06Cn1GWwtaiagL3hgYRi89zdUG6C6vismE+KZ+EOt IK5yJyxfcpAgv2RaHNQKNPimgg3kS9qInKsNp2nC6M8o4+oJWqRXNKJo14LetLLYxpgrxo3mkApsVS 7yU3XS9nshOEbze9rOR/tjrUoUe6IxgZctRMc+0nEg5lN8BydEk1NccFT2in4HyupoQp6fBMcoPett 1JOFCrlKDF/UfEp+SDkTBAyl7CwRYW83kw7+5Ub94fvoMb6+yB8VuR1Ez6gwSqPzcdADCXC8HzaUPk 1Gt4kySqTssHE3GG1yWdf8Qwjb0eiHrjb6WO4gPXBcqMQsLR9wXDVPNulCJn/sphWaJN73P0839/Yr 7O9BjXiG2AtGtfQVrNHLbu9iEirvlaLoXy7XGeJ9Hkzfo= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091319_457180_D3B75DC2 X-CRM114-Status: GOOD ( 15.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that we can set BSS variables from the early code running from the ID map, we can set memstart_offset_seed directly from the C code that derives the value instead of passing it back and forth between C and asm code. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 7 ------- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/kaslr.c | 2 +- arch/arm64/kernel/pi/kaslr_early.c | 6 +++++- 4 files changed, 8 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 5f1476c0f3a33d75..4b88ca8766133fd3 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 __primary_switch() physical misalignment/KASLR offset - * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -431,11 +430,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) str x25, [x8] // ... observes the correct value dc civac, x8 // Make visible to booting secondaries #endif - -#ifdef CONFIG_RANDOMIZE_BASE - adrp x5, memstart_offset_seed // Save KASLR linear map seed - strh w24, [x5, :lo12:memstart_offset_seed] -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif @@ -705,7 +699,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init - and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 orr x23, x23, x0 // record kernel offset #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 6ff6efbc1ce98ba6..6c6dd100a9cbadf8 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -41,6 +41,8 @@ PROVIDE(__pi___memcpy = __pi_memcpy); PROVIDE(__pi___memmove = __pi_memmove); PROVIDE(__pi___memset = __pi_memset); +PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); + PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 5d4ce7f5f157bb3f..37a9deed2aec9297 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -21,7 +21,7 @@ #include u64 __ro_after_init module_alloc_base; -u16 __initdata memstart_offset_seed; +u16 memstart_offset_seed; static int __init kaslr_init(void) { diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index 934e95fbd4278d0b..c46bccd593f2ff6b 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -15,6 +15,8 @@ #include #include +extern u16 memstart_offset_seed; + static u64 __init get_kaslr_seed(void *fdt) { static char const chosen_str[] __initconst = "chosen"; @@ -51,6 +53,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) return 0; } + memstart_offset_seed = seed & U16_MAX; + /* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the @@ -58,5 +62,5 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) * the lower and upper quarters to avoid colliding with other * allocations. */ - return BIT(VA_BITS_MIN - 3) + (seed & GENMASK(VA_BITS_MIN - 3, 0)); + return BIT(VA_BITS_MIN - 3) + (seed & GENMASK(VA_BITS_MIN - 3, 16)); } From patchwork Fri Nov 11 17:11:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 93E28C4332F for ; Fri, 11 Nov 2022 17:32:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eFmdN72UfJ+YtnbZZrp2peTbh/GpAISDSVbZ759S340=; b=L4MR4ChiEXV9H5 qf7WmSxmF1IKZH9rM/FIZ0r/bv8W/xENBpV5SeF6cvctnl2mvBGgSpVqeutXCOw2Vfa1Yb3HPUDf9 g1/5gupneHEGefpWexiO27xnn2uewyx81d9cH1MiyEGRWDJB+95E06SC8K70jBu1b1HCEbK57VDSk yQhZVRYm+JapG5RyiW7gtTL5lJsblPPfs9EyVcnD0cMiwrEIC7GlK+hBDCNyngY86BYOwdYfPTz2B DWIMnXDLsgs1GNqPTXi1W8ugDOasGwoV8T+pu48qgb1iAcrJVXgHnZrTloW62hXcrpXnUDs1P7/X7 3w1fG+nq+4o9xsnoBrLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXsV-00HHjT-E2; Fri, 11 Nov 2022 17:31:44 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaj-00H83V-NO for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:24 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3F72562066; Fri, 11 Nov 2022 17:13:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 96934C433C1; Fri, 11 Nov 2022 17:13:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186800; bh=JY4qR993JdNSkwgpLu2ZjQC9wWlYDuaVS48Om0vTb48=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Asl7N+h68AAJ7nm6YWmg4OnJqOkvpMe8Kzk7xHJNad1y5SomOEiBK748sn3gPPKX/ 4KMuq4Vtk80vGd1UKG2Z+LH9ejgsX2zw3vuLLIDyzTqn9zoewfRjPrW2JuOPe/Q+Ig 32YzkCj+2KugvQGcaqiEhVE1sOxMogolC0A6ekkP9n8ZUS51wo+TnFW3vA3rLvnYBe xATZyv6I9BZLRB3/j8Mwimosx6YhcbCxu6A3fP0K6cFUBEtLBGFjhLo+3SD3TK2qhM YZn5eZqj8pEjW3I5XGoXD0QRUaxhn43NHbbheSSO/24ykvG8+4m8LjH1nnoUNzXAvW hboNot/qSN/lw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 27/33] arm64: head: Move early kernel mapping routines into C code Date: Fri, 11 Nov 2022 18:11:55 +0100 Message-Id: <20221111171201.2088501-28-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=23184; i=ardb@kernel.org; h=from:subject; bh=JY4qR993JdNSkwgpLu2ZjQC9wWlYDuaVS48Om0vTb48=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJWPuUO4oF8EjNUxXZ2IXVOQmpEokD47ffEvWGG 5GS5XpOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CVgAKCRDDTyI5ktmPJO3uC/ 9w8rIkcsmNiRcWDjEuRKrk0gs0lnZqi23ulz+o9x9Ichh+1s/sEWi4u3B/du6csW52YGBWa31hQFwo F5IjCaY/xIbxFpokvF+Ll5FaHKtVvdAFNnRGQhkRQnfd1VvklYLo5GIJ8UV3wKdbafact3J1w6l9Nb lEe+Q4ceyyHW0FKYIf7olL5yxkTPYHQvetXL1PzsLucD2NG/exRRRIvKHdtFlXxh8adbX6ZEKDuMay b+3xoWTyngYeipxUOgB0J4TRbTLXTf6bkIvhGfnJgVqQAm4lU3CxPZc2/zLVoTU3HTqewVNgBL1nEQ uyhPuAZvlz49QBwodh8B3vXWlrUp/t7OTDgIQ6BkGCtCoXcBbi2rzfeR7HQyFVrX/OLGN9Y3+gqT2H s0MxZHQ5g+WkPqFSu6jtuTMVwGMt+jlTylyTYKUUzr/37aefwBjHQ73Z5EZxvypry9UBzHw7JEPv89 v55BPzNyuwqxCBVio5CaTHzdkOVJm/oj3gnLNWFajYAXg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091321_910251_DC3DAE99 X-CRM114-Status: GOOD ( 31.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The asm version of the kernel mapping code works fine for creating a coarse grained identity map, but for mapping the kernel down to its exact boundaries with the right attributes, it is not suitable. This is why we create a preliminary RWX kernel mapping first, and then rebuild it from scratch later on. So let's reimplement this in C, in a way that will make it unnecessary to create the kernel page tables yet another time in paging_init(). Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/scs.h | 32 +-- arch/arm64/kernel/head.S | 52 +--- arch/arm64/kernel/image-vars.h | 16 ++ arch/arm64/kernel/pi/Makefile | 2 +- arch/arm64/kernel/pi/idreg-override.c | 24 +- arch/arm64/kernel/pi/kaslr_early.c | 12 +- arch/arm64/kernel/pi/map_kernel.c | 277 ++++++++++++++++++++ arch/arm64/kernel/pi/patch-scs.c | 16 +- arch/arm64/kernel/pi/pi.h | 12 + arch/arm64/kernel/pi/relocate.c | 2 + arch/arm64/kernel/setup.c | 7 - arch/arm64/kernel/vmlinux.lds.S | 6 +- arch/arm64/mm/proc.S | 1 + 13 files changed, 338 insertions(+), 121 deletions(-) diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h index d20dcb2a1a5284ce..74744f4d6820f5e1 100644 --- a/arch/arm64/include/asm/scs.h +++ b/arch/arm64/include/asm/scs.h @@ -32,37 +32,11 @@ #include #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS -static inline bool should_patch_pac_into_scs(void) -{ - u64 reg; - - /* - * We only enable the shadow call stack dynamically if we are running - * on a system that does not implement PAC or BTI. PAC and SCS provide - * roughly the same level of protection, and BTI relies on the PACIASP - * instructions serving as landing pads, preventing us from patching - * those instructions into something else. - */ - reg = read_sysreg_s(SYS_ID_AA64ISAR1_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR1_EL1, APA, reg) | - SYS_FIELD_GET(ID_AA64ISAR1_EL1, API, reg)) - return false; - - reg = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); - if (SYS_FIELD_GET(ID_AA64ISAR2_EL1, APA3, reg)) - return false; - - if (IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) { - reg = read_sysreg_s(SYS_ID_AA64PFR1_EL1); - if (reg & (0xf << ID_AA64PFR1_EL1_BT_SHIFT)) - return false; - } - return true; -} - static inline void dynamic_scs_init(void) { - if (should_patch_pac_into_scs()) { + extern bool __pi_dynamic_scs_is_enabled; + + if (__pi_dynamic_scs_is_enabled) { pr_info("Enabling dynamic shadow call stack\n"); static_branch_enable(&dynamic_scs_enabled); } diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 4b88ca8766133fd3..6e730a0be1e8196d 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -80,7 +80,6 @@ * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob - * x23 __primary_switch() physical misalignment/KASLR offset * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */ @@ -356,24 +355,6 @@ SYM_FUNC_START_LOCAL(create_idmap) ret x28 SYM_FUNC_END(create_idmap) -SYM_FUNC_START_LOCAL(create_kernel_mapping) - adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) -#ifdef CONFIG_RELOCATABLE - add x5, x5, x23 // add KASLR displacement -#endif - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - mov x7, SWAPPER_RW_MMUFLAGS - - map_memory x0, x1, x5, x6, x7, x3, (VA_BITS - PGDIR_SHIFT), x10, x11, x12, x13, x14 - - dsb ishst // sync with page table walker - ret -SYM_FUNC_END(create_kernel_mapping) - /* * Initialize CPU registers with task-specific and cpu-specific context. * @@ -678,44 +659,13 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x2, init_idmap_pg_dir bl __enable_mmu - // Clear BSS - adrp x0, __bss_start - mov x1, xzr - adrp x2, init_pg_end - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - adrp x1, early_init_stack mov sp, x1 mov x29, xzr mov x0, x20 // pass the full boot status mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides - -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 -#ifdef CONFIG_RANDOMIZE_BASE - mov x0, x22 - bl __pi_kaslr_early_init - bic x0, x0, #SZ_2M - 1 - orr x23, x23, x0 // record kernel offset -#endif -#endif - bl create_kernel_mapping + bl __pi_early_map_kernel // Map and relocate the kernel - adrp x1, init_pg_dir - load_ttbr1 x1, x1, x2 -#ifdef CONFIG_RELOCATABLE - mov x0, x23 - bl __pi_relocate_kernel -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - ldr x0, =__eh_frame_start - ldr x1, =__eh_frame_end - bl __pi_scs_patch_vmlinux -#endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START) br x8 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 6c6dd100a9cbadf8..88f864f28f03630c 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -51,8 +51,24 @@ PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); PROVIDE(__pi_id_aa64zfr0_override = id_aa64zfr0_override); PROVIDE(__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE(__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); PROVIDE(__pi__ctype = _ctype); +PROVIDE(__pi_init_pg_dir = init_pg_dir); +PROVIDE(__pi_init_pg_end = init_pg_end); +PROVIDE(__pi__end = _end); + +PROVIDE(__pi__text = _text); +PROVIDE(__pi__stext = _stext); +PROVIDE(__pi__etext = _etext); +PROVIDE(__pi___start_rodata = __start_rodata); +PROVIDE(__pi___inittext_begin = __inittext_begin); +PROVIDE(__pi___inittext_end = __inittext_end); +PROVIDE(__pi___initdata_begin = __initdata_begin); +PROVIDE(__pi___initdata_end = __initdata_end); +PROVIDE(__pi__data = _data); +PROVIDE(__pi___bss_start = __bss_start); + #ifdef CONFIG_KVM /* diff --git a/arch/arm64/kernel/pi/Makefile b/arch/arm64/kernel/pi/Makefile index 293a04a5dc3ef516..ac27f1cac6b89684 100644 --- a/arch/arm64/kernel/pi/Makefile +++ b/arch/arm64/kernel/pi/Makefile @@ -38,7 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o \ +obj-y := idreg-override.pi.o map_kernel.pi.o \ lib-fdt.pi.o lib-fdt_ro.pi.o obj-$(CONFIG_RELOCATABLE) += relocate.pi.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index e524d36d8fc1526f..d0ce3dc4e07aaf4d 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -15,6 +15,8 @@ #include #include +#include "pi.h" + #define FTR_DESC_NAME_LEN 20 // must remain multiple of 4 #define FTR_DESC_FIELD_LEN 10 // must remain multiple of 4 +/- 2 #define FTR_ALIAS_NAME_LEN 30 @@ -296,37 +298,35 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) } while (1); } -static __init const u8 *get_bootargs_cmdline(const void *fdt) +static __init const u8 *get_bootargs_cmdline(const void *fdt, int node) { + static char const bootargs[] __initconst = "bootargs"; const u8 *prop; - int node; - node = fdt_path_offset(fdt, "/chosen"); if (node < 0) return NULL; - prop = fdt_getprop(fdt, node, "bootargs", NULL); + prop = fdt_getprop(fdt, node, bootargs, NULL); if (!prop) return NULL; return strlen(prop) ? prop : NULL; } -static __init void parse_cmdline(const void *fdt) +static __init void parse_cmdline(const void *fdt, int chosen) { - const u8 *prop = get_bootargs_cmdline(fdt); + static char const cmdline[] __initconst = CONFIG_CMDLINE; + const u8 *prop = get_bootargs_cmdline(fdt, chosen); if (IS_ENABLED(CONFIG_CMDLINE_FORCE) || !prop) - __parse_cmdline(CONFIG_CMDLINE, true); + __parse_cmdline(cmdline, true); if (!IS_ENABLED(CONFIG_CMDLINE_FORCE) && prop) __parse_cmdline(prop, true); } -/* Keep checkers quiet */ -void init_feature_override(u64 boot_status, const void *fdt); - -asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) +void __init init_feature_override(u64 boot_status, const void *fdt, + int chosen) { int i; @@ -337,7 +337,7 @@ asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) __boot_status = boot_status; - parse_cmdline(fdt); + parse_cmdline(fdt, chosen); for (i = 0; i < ARRAY_SIZE(regs); i++) { dcache_clean_inval_poc((unsigned long)reg_override(i), diff --git a/arch/arm64/kernel/pi/kaslr_early.c b/arch/arm64/kernel/pi/kaslr_early.c index c46bccd593f2ff6b..965515f7f1809808 100644 --- a/arch/arm64/kernel/pi/kaslr_early.c +++ b/arch/arm64/kernel/pi/kaslr_early.c @@ -15,17 +15,17 @@ #include #include +#include "pi.h" + extern u16 memstart_offset_seed; -static u64 __init get_kaslr_seed(void *fdt) +static u64 __init get_kaslr_seed(void *fdt, int node) { - static char const chosen_str[] __initconst = "chosen"; static char const seed_str[] __initconst = "kaslr-seed"; - int node, len; fdt64_t *prop; u64 ret; + int len; - node = fdt_path_offset(fdt, chosen_str); if (node < 0) return 0; @@ -38,7 +38,7 @@ static u64 __init get_kaslr_seed(void *fdt) return ret; } -asmlinkage u64 __init kaslr_early_init(void *fdt) +u64 __init kaslr_early_init(void *fdt, int chosen) { u64 seed; @@ -46,7 +46,7 @@ asmlinkage u64 __init kaslr_early_init(void *fdt) ARM64_SW_FEATURE_OVERRIDE_NOKASLR)) return 0; - seed = get_kaslr_seed(fdt); + seed = get_kaslr_seed(fdt, chosen); if (!seed) { if (!__early_cpu_has_rndr() || !__arm64_rndr((unsigned long *)&seed)) diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c new file mode 100644 index 0000000000000000..c5c6eebef684f81d --- /dev/null +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -0,0 +1,277 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2022 Google LLC +// Author: Ard Biesheuvel + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include "pi.h" + +extern const u8 __eh_frame_start[], __eh_frame_end[]; + +extern void idmap_cpu_replace_ttbr1(void *pgdir); + +static void __init map_range(pgd_t **pgd, u64 start, u64 end, u64 pa, + pgprot_t prot, int level, pte_t *tbl, + bool may_use_cont) +{ + u64 cmask = (level == 3) ? CONT_PTE_SIZE - 1 : U64_MAX; + u64 protval = pgprot_val(prot) & ~PTE_TYPE_MASK; + int lshift = (3 - level) * (PAGE_SHIFT - 3); + u64 lmask = (PAGE_SIZE << lshift) - 1; + + /* Advance tbl to the entry that covers start */ + tbl += (start >> (lshift + PAGE_SHIFT)) % BIT(PAGE_SHIFT - 3); + + /* + * Set the right block/page bits for this level unless we are + * clearing the mapping + */ + if (protval) + protval |= (level < 3) ? PMD_TYPE_SECT : PTE_TYPE_PAGE; + + while (start < end) { + u64 next = min((start | lmask) + 1, end); + + if (level < 3 && (start | next | pa) & lmask) { + /* + * This chunk needs a finer grained mapping. Put down + * a table mapping if necessary and recurse. + */ + if (pte_none(*tbl)) { + *tbl = __pte(__phys_to_pte_val((u64)*pgd) | + PMD_TYPE_TABLE | PMD_TABLE_UXN); + *pgd += PTRS_PER_PTE; + } + map_range(pgd, start, next, pa, prot, level + 1, + (pte_t *)__pte_to_phys(*tbl), may_use_cont); + } else { + /* + * Start a contiguous range if start and pa are + * suitably aligned + */ + if (((start | pa) & cmask) == 0 && may_use_cont) + protval |= PTE_CONT; + + /* + * Clear the contiguous attribute if the remaining + * range does not cover a contiguous block + */ + if ((end & ~cmask) <= start) + protval &= ~PTE_CONT; + + /* Put down a block or page mapping */ + *tbl = __pte(__phys_to_pte_val(pa) | protval); + } + pa += next - start; + start = next; + tbl++; + } +} + +static void __init map_segment(pgd_t **pgd, u64 va_offset, void *start, + void *end, pgprot_t prot, bool may_use_cont) +{ + map_range(pgd, ((u64)start + va_offset) & ~PAGE_OFFSET, + ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, + prot, 4 - CONFIG_PGTABLE_LEVELS, (pte_t *)init_pg_dir, + may_use_cont); +} + +static void __init unmap_segment(u64 va_offset, void *start, void *end) +{ + map_range(NULL, ((u64)start + va_offset) & ~PAGE_OFFSET, + ((u64)end + va_offset) & ~PAGE_OFFSET, (u64)start, + __pgprot(0), 4 - CONFIG_PGTABLE_LEVELS, (pte_t *)init_pg_dir, + false); +} + +static bool __init arm64_early_this_cpu_has_bti(void) +{ + u64 pfr1; + + if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) + return false; + + pfr1 = read_sysreg(ID_AA64PFR1_EL1); + pfr1 &= ~id_aa64pfr1_override.mask; + pfr1 |= id_aa64pfr1_override.val; + + return cpuid_feature_extract_unsigned_field(pfr1, + ID_AA64PFR1_EL1_BT_SHIFT); +} + +static bool __init arm64_early_this_cpu_has_e0pd(void) +{ + u64 mmfr2; + + if (!IS_ENABLED(CONFIG_ARM64_E0PD)) + return false; + + mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + return cpuid_feature_extract_unsigned_field(mmfr2, + ID_AA64MMFR2_EL1_E0PD_SHIFT); +} + +static bool __init arm64_early_this_cpu_has_pac(void) +{ + u64 isar1, isar2; + u8 feat; + + if (!IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) + return false; + + isar1 = read_sysreg(ID_AA64ISAR1_EL1); + isar1 &= ~id_aa64isar1_override.mask; + isar1 |= id_aa64isar1_override.val; + feat = cpuid_feature_extract_unsigned_field(isar1, + ID_AA64ISAR1_EL1_APA_SHIFT); + if (feat) + return true; + + feat = cpuid_feature_extract_unsigned_field(isar1, + ID_AA64ISAR1_EL1_API_SHIFT); + if (feat) + return true; + + isar2 = read_sysreg_s(SYS_ID_AA64ISAR2_EL1); + isar2 &= ~id_aa64isar2_override.mask; + isar2 |= id_aa64isar2_override.val; + feat = cpuid_feature_extract_unsigned_field(isar2, + ID_AA64ISAR2_EL1_APA3_SHIFT); + return feat; +} + +static void __init map_kernel(u64 kaslr_offset, u64 va_offset) +{ + bool enable_scs = IS_ENABLED(CONFIG_UNWIND_PATCH_PAC_INTO_SCS); + bool twopass = IS_ENABLED(CONFIG_RELOCATABLE); + pgd_t *pgdp = (void *)init_pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + pgprot_t prot; + + /* + * External debuggers may need to write directly to the text mapping to + * install SW breakpoints. Allow this (only) when explicitly requested + * with rodata=off. + */ + if (cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF)) + text_prot = PAGE_KERNEL_EXEC; + + /* + * We only enable the shadow call stack dynamically if we are running + * on a system that does not implement PAC or BTI. PAC and SCS provide + * roughly the same level of protection, and BTI relies on the PACIASP + * instructions serving as landing pads, preventing us from patching + * those instructions into something else. + */ + if (arm64_early_this_cpu_has_pac()) + enable_scs = false; + + if (arm64_early_this_cpu_has_bti()) { + enable_scs = false; + + /* + * If we have a CPU that supports BTI and a kernel built for + * BTI then mark the kernel executable text as guarded pages + * now so we don't have to rewrite the page tables later. + */ + text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + } + + /* Map all code read-write on the first pass if needed */ + twopass |= enable_scs; + prot = twopass ? data_prot : text_prot; + + map_segment(&pgdp, va_offset, _stext, _etext, prot, !twopass); + map_segment(&pgdp, va_offset, __start_rodata, __inittext_begin, data_prot, false); + map_segment(&pgdp, va_offset, __inittext_begin, __inittext_end, prot, false); + map_segment(&pgdp, va_offset, __initdata_begin, __initdata_end, data_prot, false); + map_segment(&pgdp, va_offset, _data, _end, data_prot, true); + dsb(ishst); + + idmap_cpu_replace_ttbr1(init_pg_dir); + + if (twopass) { + if (IS_ENABLED(CONFIG_RELOCATABLE)) + relocate_kernel(kaslr_offset); + + if (enable_scs) { + scs_patch(__eh_frame_start + va_offset, + __eh_frame_end - __eh_frame_start); + asm("ic ialluis"); + + dynamic_scs_is_enabled = true; + } + + /* + * Unmap the text region before remapping it, to avoid + * potential TLB conflicts when creating the contiguous + * descriptors. + */ + unmap_segment(va_offset, _stext, _etext); + dsb(ishst); + isb(); + __tlbi(vmalle1); + isb(); + + /* + * Remap these segments with different permissions + * No new page table allocations should be needed + */ + map_segment(NULL, va_offset, _stext, _etext, text_prot, true); + map_segment(NULL, va_offset, __inittext_begin, __inittext_end, + text_prot, false); + dsb(ishst); + } +} + +asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) +{ + static char const chosen_str[] __initconst = "/chosen"; + int chosen = fdt_path_offset(fdt, chosen_str); + u64 va_base, pa_base = (u64)&_text; + u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; + + /* Clear BSS and the initial page tables */ + memset(__bss_start, 0, (u64)init_pg_end - (u64)__bss_start); + + /* Parse the command line for CPU feature overrides */ + init_feature_override(boot_status, fdt, chosen); + + /* + * The virtual KASLR displacement modulo 2MiB is decided by the + * physical placement of the image, as otherwise, we might not be able + * to create the early kernel mapping using 2 MiB block descriptors. So + * take the low bits of the KASLR offset from the physical address, and + * fill in the high bits from the seed. + */ + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { + u64 kaslr_seed = kaslr_early_init(fdt, chosen); + + kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); + + /* + * Assume that any CPU that does not implement E0PD needs KPTI + * to ensure that KASLR randomized addresses will not leak. + * This means we need to use non-global mappings for the kernel + * text and data. + */ + if (kaslr_seed && !arm64_early_this_cpu_has_e0pd()) + arm64_use_ng_mappings = true; + } + + va_base = KIMAGE_VADDR + kaslr_offset; + map_kernel(kaslr_offset, va_base - pa_base); +} diff --git a/arch/arm64/kernel/pi/patch-scs.c b/arch/arm64/kernel/pi/patch-scs.c index d15833df10d3d4c6..bfeeaffc1e233083 100644 --- a/arch/arm64/kernel/pi/patch-scs.c +++ b/arch/arm64/kernel/pi/patch-scs.c @@ -11,6 +11,10 @@ #include +#include "pi.h" + +bool dynamic_scs_is_enabled; + // // This minimal DWARF CFI parser is partially based on the code in // arch/arc/kernel/unwind.c, and on the document below: @@ -46,8 +50,6 @@ #define DW_CFA_GNU_negative_offset_extended 0x2f #define DW_CFA_hi_user 0x3f -extern const u8 __eh_frame_start[], __eh_frame_end[]; - enum { PACIASP = 0xd503233f, AUTIASP = 0xd50323bf, @@ -245,13 +247,3 @@ int scs_patch(const u8 eh_frame[], int size) } return 0; } - -asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) -{ - if (!should_patch_pac_into_scs()) - return; - - scs_patch(start, end - start); - asm("ic ialluis"); - isb(); -} diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h new file mode 100644 index 0000000000000000..c0b00199e3dc80e0 --- /dev/null +++ b/arch/arm64/kernel/pi/pi.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2022 Google LLC +// Author: Ard Biesheuvel + +#include + +extern bool dynamic_scs_is_enabled; + +void init_feature_override(u64 boot_status, const void *fdt, int chosen); +u64 kaslr_early_init(void *fdt, int chosen); +void relocate_kernel(u64 offset); +int scs_patch(const u8 eh_frame[], int size); diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c index c35cb918fa2a004a..06e3cc6cdd6a68ca 100644 --- a/arch/arm64/kernel/pi/relocate.c +++ b/arch/arm64/kernel/pi/relocate.c @@ -6,6 +6,8 @@ #include #include +#include "pi.h" + extern const Elf64_Rela rela_start[], rela_end[]; extern const u64 relr_start[], relr_end[]; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 37e0ba95afc3e045..149e7c3ee1321363 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -280,13 +280,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) *cmdline_p = boot_command_line; - /* - * If know now we are going to need KPTI then use non-global - * mappings from the start, avoiding the cost of rewriting - * everything later. - */ - arm64_use_ng_mappings = kaslr_requires_kpti(); - early_fixmap_init(); early_ioremap_init(); diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 3f86a0db2952600c..33c3e59233f6fa6e 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -124,9 +124,9 @@ jiffies = jiffies_64; #ifdef CONFIG_UNWIND_TABLES #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ - __eh_frame_start = .; \ + __pi___eh_frame_start = .; \ *(.eh_frame) \ - __eh_frame_end = .; \ + __pi___eh_frame_end = .; \ } #else #define UNWIND_DATA_SECTIONS @@ -313,7 +313,7 @@ SECTIONS BSS_SECTION(SBSS_ALIGN, 0, 0) - . = ALIGN(PAGE_SIZE); + . = ALIGN(SEGMENT_ALIGN); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index b9ecbbae1e1abca1..f0db2c05e797aeed 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -201,6 +201,7 @@ SYM_TYPED_FUNC_START(idmap_cpu_replace_ttbr1) ret SYM_FUNC_END(idmap_cpu_replace_ttbr1) +SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 From patchwork Fri Nov 11 17:11:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6AF22C433FE for ; Fri, 11 Nov 2022 17:33:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NvUyei26wyYKhECAN2LhbS5UFaZFlzMYzQJqPzLbh+k=; b=knHhL6g71y23Yu ztjHCKXJJw0xdJLGU//XfDYGr8GU+VZYtuiS2etCVGQc8+2aaJQMNJ8WrtXQWfn9uTioMkAdGCFIZ v2/CLoNjr4EZYV4vfAqe0A6yefFv9n1aAdA3uSUlG8+a6wpO6ZKAkOUsG+TECKQKjTWXR+wt5LWPw gjLjyCvr3r2Pxn5lcNJDvWIfVr4k6DJFNfBEyoTrM7xYQcSNaPmktjJNkKAglvKnpe7XPwrM2FKDV Bb9wXW286IOsKe0XrqTMxat7Zyb4hvg8/ff4sad3oVUie05upIDlafG9Em9LQtdRSo1ECLtMUBPDk jNBV0WLuxxCOkRKAFphA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXtF-00HI9H-Lm; Fri, 11 Nov 2022 17:32:30 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXam-00H84T-2r for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:25 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 87FCB6205C; Fri, 11 Nov 2022 17:13:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 14B68C43144; Fri, 11 Nov 2022 17:13:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186802; bh=nVh7PpTz2KdcicpZhASftr31fDniIS6jDeYMIEAwpMk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N2rHcuwj/kGNwdBMPMFBkTD8haqs3ceZTnZ0icotU3NYixQ2KmAmV3rCjW1FAhfjo /OOrVugzbV8x7ZjbffvAFPRk9VlGRllASlTfqZiY/GWCJZ/aJn14Uce+zLArtX/lzb DROFCt0hEMWJxBwqTWuAXcO1kyJEyKWfCMztICDf6/M5IVAMA+xB4YrZNGpzuJGcvl AV94iV2rrdDPJ7lVaDAXM2tqcWIabGiOIYEj14SizQUYkSakZl20Zk7emJw1bEN4xA UN+JMqevYcysG3yW9JyWwgQB57CZfXD87/1JDppv4O9u5YWOsut8P1hLgLY5Vb1b7b xJ/FJYUpLMQ8g== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 28/33] arm64: mm: avoid fixmap for early swapper_pg_dir updates Date: Fri, 11 Nov 2022 18:11:56 +0100 Message-Id: <20221111171201.2088501-29-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2146; i=ardb@kernel.org; h=from:subject; bh=nVh7PpTz2KdcicpZhASftr31fDniIS6jDeYMIEAwpMk=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJXpLz7olDX386Tge6wUaBufeoF3qkK9fAcEK6G tBxD6OKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CVwAKCRDDTyI5ktmPJNWIDA C0rGbikX4v7Gn/0O/6pGa94oPvMj99XINVo1AxyJWuYKQAJLPAf7EfBLZNJp6e0ztFpdodSTMx+GNa 30h0XeMrlb1DWmako6n/ta71dAOhwe8x6FSQq4jBLfBkO9gNqiVYNKznJyjEEo8ORa+5qQ+CBy01CI 5jzLAQBWZBM2HUHP9SWfKMPIfEzjw8HeS7gFA4Hmb0W7WrY7Z0N4hA90wfxgHBMho7L9vsubfUrQJR /76xjwbgv6YRe5p/5IR+9xFXoVDqEMkCDftnU1xbqKpygrAha8Ahf7iu8nCPl98UvJW5sezybN84zR jFp1cS/DvrqPcK+9qQ+kPF1+rjtwZn0w2NU43kszU8TcVxzIBKImeWRfvGdR4u7H9HfFDUD1Xvqz7c yRE7gMXuWUQmqO61biUx6c8n9+OJG4H/InzOLaTGkxBBlqoc4tFTGttynDPZRzaw8ECjJm3r5gPeur hcv01A9EMvtOynb0ak1POSNrLxxGJ+HcsBhXTBWWaTGns= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091324_275138_809C8C52 X-CRM114-Status: GOOD ( 15.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case. As the same reasoning applies to early KASAN init, make the function noinstr as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 0c35e1f195678695..68e66b979fc3ac5d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -58,6 +58,8 @@ EXPORT_SYMBOL(kimage_voffset); u32 __boot_cpu_mode[] = { BOOT_CPU_MODE_EL2, BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw __ro_after_init = true; + /* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off. @@ -78,10 +80,21 @@ static pud_t bm_pud[PTRS_PER_PUD] __page_aligned_bss __maybe_unused; static DEFINE_SPINLOCK(swapper_pgdir_lock); static DEFINE_MUTEX(fixmap_lock); -void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; + /* + * Don't bother with the fixmap if swapper_pg_dir is still mapped + * writable in the kernel mapping. + */ + if (rodata_is_rw) { + WRITE_ONCE(*pgdp, pgd); + dsb(ishst); + isb(); + return; + } + spin_lock(&swapper_pgdir_lock); fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp)); WRITE_ONCE(*fixmap_pgdp, pgd); @@ -615,6 +628,7 @@ void mark_rodata_ro(void) * to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long)__init_begin - (unsigned long)__start_rodata; + WRITE_ONCE(rodata_is_rw, false); update_mapping_prot(__pa_symbol(__start_rodata), (unsigned long)__start_rodata, section_size, PAGE_KERNEL_RO); From patchwork Fri Nov 11 17:11:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 137F2C4332F for ; Fri, 11 Nov 2022 17:34:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PKyM0M/pHwMvasmXPC9ierOdahzUZk4sQTmeSKFBSuQ=; b=pwWKCYFSZz/btj 8ihEISqL8lMvCgyIbKfWYGCJXb2BcYiiY7hTFSB8QRWNz/6zIjcmX6zckrnnYGGzTCgYcStw8q78O MCAV29nzYHldWzErVoaXwerH/DY0zQIm2gjZVQ6MV0stRQGpCCOP+OxHGXCtm8AGHheJILi3c4WvE NAtwzfDIo5C2m2m5mDxBxGoE6c/CO9LPOJ7oA9P8QPk2CtTD0PA/K/15kB3tj4AdGV4n8Y+JfBLGY 1Z/uyH0btyb60z3/0QadJCeCLABll0LDaKAnbnH4rqbcLsTRG/MUEt9Pr0Fm/RRQRed29X138g8Nd wgRj1CWgDDWyliE1PlGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXuJ-00HIjX-Dk; Fri, 11 Nov 2022 17:33:36 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXap-00H86d-SE for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:31 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 85126B8267D; Fri, 11 Nov 2022 17:13:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EA95C43143; Fri, 11 Nov 2022 17:13:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186805; bh=eX6HHnO4OidM0af8Sa/TY+Ph+xpausTa0xC2JQB+wUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IImeGTnp0DCFl/ztr8wjdZsJHUuFklaB43OxfCvDrAMgXzzMP6zJAfrXMQhHtma/6 C+nkg9NmA00ysVuyaT3pVNt0LptyvfgPWu++vKPatm9h/ZnbCsKFuZ4qWdaDMJ5tz9 O9whJ2rjrwgd6d1Pi3igCw+RWFpqv/fuXiectWeU50cJKH3Lx6gFuufGm3PCGf/Oip 5AgOod1OGcK14G1edm44fko+FwAOmIRux/O/FbaVIhrdqM9YYucLCsd0rjXJTiHDTV nzwRMXxqtB7xOOgJHaEG/K3RA1PhrmH3hFx/PBweK7nQzW4+2P/7Fy9H+aG+Kof8y7 lAeYRsqG1atxA== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 29/33] arm64: mm: omit redundant remap of kernel image Date: Fri, 11 Nov 2022 18:11:57 +0100 Message-Id: <20221111171201.2088501-30-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9744; i=ardb@kernel.org; h=from:subject; bh=eX6HHnO4OidM0af8Sa/TY+Ph+xpausTa0xC2JQB+wUM=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJZ2rWfrBUsu+MyzKp1S9k+qFBTk6vJybiKWK+h jhQCtpSJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CWQAKCRDDTyI5ktmPJEdVDA ChFDysIE2JoShHmugWpC9PN2caod/p7lpsX3/kfrKfFM4WYjS/UmMDe4ia/NUpbtr9RB1OWknRJYr4 8w7b4VrxfnPHKuQirbIdRFMvSYXqQS81BJPiY+O+HPAFDNnx1SxmhNLtZ3j+xk8XBl6iBz3Y2O6UK7 9px5Qu11TptIVoTixxR6IBClG1334hWu6tQuiPwND+4O5TchVdLdIGTVBNSkyN0KUj8ZZ4BOKpXwMi H7NJlQzy8ct5BmqyZBXsbFn4Qm9NBz6In+yR5zArodHWfPPZhhai1yYUBPELiu0wT5dwoSJa8yNOh+ Ql93GGPEzZs37i5R4iIK9OFhs1CWbeDurjMywzLDo/RAxGgINca/dpNONw3licdZuRNeqCT0pBY9vq 8yn0zMzemaEf2nmX7Zqii+W93izdxkXjhx3gtaNLqSemxSry01EEMgSmx1JNDhyPv95j7qnWKE4TVY 6TNHdLi5Qi6YrFddh4sjV1ssRvUkHUpKMmSCU8jF9Xbxk= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091328_295260_2315F390 X-CRM114-Status: GOOD ( 26.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 2 +- arch/arm64/kernel/pi/map_kernel.c | 9 +- arch/arm64/mm/kasan_init.c | 15 --- arch/arm64/mm/mmu.c | 110 +++----------------- 6 files changed, 22 insertions(+), 118 deletions(-) diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index 12d5f47f7dbec628..ab52688ac4bd43b6 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -36,12 +36,10 @@ void kasan_init(void); #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) -void kasan_copy_shadow(pgd_t *pgdir); asmlinkage void kasan_early_init(void); #else static inline void kasan_init(void) { } -static inline void kasan_copy_shadow(pgd_t *pgdir) { } #endif #endif diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 48f8466a4be92ac3..a93d495d6e8c94a2 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -73,7 +73,7 @@ extern void mark_linear_text_alias_ro(void); extern bool kaslr_requires_kpti(void); #define INIT_MM_CONTEXT(name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir, #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 88f864f28f03630c..5bd878f414d85366 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -56,7 +56,7 @@ PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); -PROVIDE(__pi__end = _end); +PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); PROVIDE(__pi__text = _text); PROVIDE(__pi__stext = _stext); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index c5c6eebef684f81d..4b604b104460c3ef 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -198,7 +198,8 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset) map_segment(&pgdp, va_offset, __start_rodata, __inittext_begin, data_prot, false); map_segment(&pgdp, va_offset, __inittext_begin, __inittext_end, prot, false); map_segment(&pgdp, va_offset, __initdata_begin, __initdata_end, data_prot, false); - map_segment(&pgdp, va_offset, _data, _end, data_prot, true); + map_segment(&pgdp, va_offset, _data, init_pg_dir, data_prot, true); + /* omit [init_pg_dir, _end] - it doesn't need a kernel mapping */ dsb(ishst); idmap_cpu_replace_ttbr1(init_pg_dir); @@ -233,8 +234,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset) map_segment(NULL, va_offset, _stext, _etext, text_prot, true); map_segment(NULL, va_offset, __inittext_begin, __inittext_end, text_prot, false); - dsb(ishst); } + + /* Copy the root page table to its final location */ + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + dsb(ishst); + idmap_cpu_replace_ttbr1(swapper_pg_dir); } asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index e969e68de005fd2a..df98f496539f0e39 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -184,21 +184,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end, kasan_pgd_populate(start & PAGE_MASK, PAGE_ALIGN(end), node, false); } -/* - * Copy the current shadow region into a new pgdir. - */ -void __init kasan_copy_shadow(pgd_t *pgdir) -{ - pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp = pgd_offset_k(KASAN_SHADOW_START); - pgdp_end = pgd_offset_k(KASAN_SHADOW_END); - pgdp_new = pgd_offset_pgd(pgdir, KASAN_SHADOW_START); - do { - set_pgd(pgdp_new, READ_ONCE(*pgdp)); - } while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} - static void __init clear_pgds(unsigned long start, unsigned long end) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 68e66b979fc3ac5d..6942255056aed5ae 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -635,9 +635,9 @@ void mark_rodata_ro(void) debug_checkwx(); } -static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, - pgprot_t prot, struct vm_struct *vma, - int flags, unsigned long vm_flags) +static void __init declare_vma(struct vm_struct *vma, + void *va_start, void *va_end, + unsigned long vm_flags) { phys_addr_t pa_start = __pa_symbol(va_start); unsigned long size = va_end - va_start; @@ -645,9 +645,6 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); - if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -692,87 +689,17 @@ core_initcall(map_entry_trampoline); #endif /* - * Open coded check for BTI, only for use to determine configuration - * for early mappings for before the cpufeature code has run. + * Declare the VMA areas for the kernel */ -static bool arm64_early_this_cpu_has_bti(void) +static void __init declare_kernel_vmas(void) { - u64 pfr1; - - if (!IS_ENABLED(CONFIG_ARM64_BTI_KERNEL)) - return false; - - pfr1 = __read_sysreg_by_encoding(SYS_ID_AA64PFR1_EL1); - return cpuid_feature_extract_unsigned_field(pfr1, - ID_AA64PFR1_EL1_BT_SHIFT); -} - -/* - * Create fine-grained mappings for the kernel. - */ -static void __init map_kernel(pgd_t *pgdp) -{ - static struct vm_struct vmlinux_text, vmlinux_rodata, vmlinux_inittext, - vmlinux_initdata, vmlinux_data; - - /* - * External debuggers may need to write directly to the text - * mapping to install SW breakpoints. Allow this (only) when - * explicitly requested with rodata=off. - */ - pgprot_t text_prot = rodata_enabled ? PAGE_KERNEL_ROX : PAGE_KERNEL_EXEC; - - /* - * If we have a CPU that supports BTI and a kernel built for - * BTI then mark the kernel executable text as guarded pages - * now so we don't have to rewrite the page tables later. - */ - if (arm64_early_this_cpu_has_bti()) - text_prot = __pgprot_modify(text_prot, PTE_GP, PTE_GP); + static struct vm_struct vmlinux_seg[KERNEL_SEGMENT_COUNT]; - /* - * Only rodata will be remapped with different permissions later on, - * all other segments are allowed to use contiguous mappings. - */ - map_kernel_segment(pgdp, _stext, _etext, text_prot, &vmlinux_text, 0, - VM_NO_GUARD); - map_kernel_segment(pgdp, __start_rodata, __inittext_begin, PAGE_KERNEL, - &vmlinux_rodata, NO_CONT_MAPPINGS, VM_NO_GUARD); - map_kernel_segment(pgdp, __inittext_begin, __inittext_end, text_prot, - &vmlinux_inittext, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, __initdata_begin, __initdata_end, PAGE_KERNEL, - &vmlinux_initdata, 0, VM_NO_GUARD); - map_kernel_segment(pgdp, _data, _end, PAGE_KERNEL, &vmlinux_data, 0, 0); - - if (!READ_ONCE(pgd_val(*pgd_offset_pgd(pgdp, FIXADDR_START)))) { - /* - * The fixmap falls in a separate pgd to the kernel, and doesn't - * live in the carveout for the swapper_pg_dir. We can simply - * re-use the existing dir for the fixmap. - */ - set_pgd(pgd_offset_pgd(pgdp, FIXADDR_START), - READ_ONCE(*pgd_offset_k(FIXADDR_START))); - } else if (CONFIG_PGTABLE_LEVELS > 3) { - pgd_t *bm_pgdp; - p4d_t *bm_p4dp; - pud_t *bm_pudp; - /* - * The fixmap shares its top level pgd entry with the kernel - * mapping. This can really only occur when we are running - * with 16k/4 levels, so we can simply reuse the pud level - * entry instead. - */ - BUG_ON(!IS_ENABLED(CONFIG_ARM64_16K_PAGES)); - bm_pgdp = pgd_offset_pgd(pgdp, FIXADDR_START); - bm_p4dp = p4d_offset(bm_pgdp, FIXADDR_START); - bm_pudp = pud_set_fixmap_offset(bm_p4dp, FIXADDR_START); - pud_populate(&init_mm, bm_pudp, lm_alias(bm_pmd)); - pud_clear_fixmap(); - } else { - BUG(); - } - - kasan_copy_shadow(pgdp); + declare_vma(&vmlinux_seg[0], _stext, _etext, VM_NO_GUARD); + declare_vma(&vmlinux_seg[1], __start_rodata, __inittext_begin, VM_NO_GUARD); + declare_vma(&vmlinux_seg[2], __inittext_begin, __inittext_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[3], __initdata_begin, __initdata_end, VM_NO_GUARD); + declare_vma(&vmlinux_seg[4], _data, _end, 0); } static void __init create_idmap(void) @@ -807,25 +734,14 @@ static void __init create_idmap(void) void __init paging_init(void) { - pgd_t *pgdp = pgd_set_fixmap(__pa_symbol(swapper_pg_dir)); - extern pgd_t init_idmap_pg_dir[]; - idmap_t0sz = 63UL - __fls(__pa_symbol(_end) | GENMASK(VA_BITS_MIN - 1, 0)); - map_kernel(pgdp); - map_mem(pgdp); - - pgd_clear_fixmap(); - - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), init_idmap_pg_dir); - init_mm.pgd = swapper_pg_dir; - - memblock_phys_free(__pa_symbol(init_pg_dir), - __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir)); + map_mem(swapper_pg_dir); memblock_allow_resize(); create_idmap(); + declare_kernel_vmas(); } /* From patchwork Fri Nov 11 17:11:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0A643C4332F for ; Fri, 11 Nov 2022 17:36:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ht87nAj579IATkGWpITlSG275pZZvgN42ioGrCcB7KY=; b=bBt51sM/YChFxg 0U3Qi1i3pK6YZINFuAI32LtgpR77S3/rT7DzT6gF594wnlMAUUoiczsDQkPk6L4SyAw2EdbrY+zAz N+Aqy/kPsmLp7V7uttGKwfrkrN0AJpmgdY2KrUti733a6ttrpk3SgO+CJoFxjcIpRwUYSU+mGNlTm fnyLqe7qzIxdYilRVam2XyzpwaHB80qV8y0LAjWk+RwKT5uOBL+Ydivf8Z2cUltptbihDfJo9ZirJ 1ECkDpNyBgdI/AVz9SbIOliobSqCSm1MjOSM9Er0xqNzOlCdvNmhUZFPKO7nYRwxPAh6ezpxvnSyk JZ6zxXY1IhIACFD2juDw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXwO-00HJst-Lj; Fri, 11 Nov 2022 17:35:44 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXaq-00H87l-Jq for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:33 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 215F96206C; Fri, 11 Nov 2022 17:13:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7B8AC433D7; Fri, 11 Nov 2022 17:13:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186807; bh=Blw0gWbIE9SQyI0x/IqXVoZWk0/G+DQ7lMnPJtMO5CU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tbo160W/a0FkhIi2ol65NsvnXqtrCG8NK+sf+sM2NWx2ODEjwQV7CulYfxT91I8FZ 0pCns2mUY4hUJtdRkx9OuHKhY5+fZTRX+gTZCmaLI+LUwX7jykPXKen1OBSC9y+TUz tIjldDKuWd5sZhJX6iJWV63E7uuRiwqwOoTQzsz5LLD1jW0AsD/vPGTdm+RX0r5VhF ZQiMwk5TUk78dlzJ4A3/ek6lPFT7kJKinnSqmsj1fEhB0zHzumZze3PMDWq6P3v7zA 8D4Pq6ZkFtrceDsXToVStAw7zCjg8ls+LpXAOe/86TRaNLCkgEm7pmRB6GzHWHCU5W Z8JtOh9SWrcvw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 30/33] arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()" Date: Fri, 11 Nov 2022 18:11:58 +0100 Message-Id: <20221111171201.2088501-31-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3745; i=ardb@kernel.org; h=from:subject; bh=Blw0gWbIE9SQyI0x/IqXVoZWk0/G+DQ7lMnPJtMO5CU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJbrVxEICZi45aN/PsF2NeeRgT9Nej7GSyBdWSi kyjsiXqJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CWwAKCRDDTyI5ktmPJP5PDA CvD7JYCGlCtDgrzHJtIh45Q2d7srN7b5ZGB00wROioNlfiLHpisXl26ee1KORht7rtpRd7TTo+iz5i D8xsB++WVMpE3NnO5pzCvLieTPdt4x/X7KqjPfgL33mFohby6Tp9REUMNxHL3aGi6nHIXNBwHHyy71 RobWdsA1n0wPgNd3AMBoYkODsR2hw1eVRQ6pqWkV/M2CEGxIzNt3Mp90Za9Lg0thtBYwz4GwVuXYvD WG5uzA9p97MIqvl8f2mRHGAg5dsz7AfSzShPURqM4iiT+JRl3dqre4v72xPYAZpDGp02ZYYLMdhCEn apglLwUDFIJzld4MV3uK5mAyN/vsm58dPy+UToEDqPC3l5LGJ4/EL4S+Ct40XlY8KDpsrB+tlHB7US Z4o3fgT9zZOj0sDAXniLiF7cLlNWPaSKfPFGV+eEr29PyZtbIeCN3xI0HPchPdO8vY86WsuWZJKyK+ mWHD8KW8uFbHDy/7YyQ1kHLej1R+9JZaW5NGCcb0yvLpI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091328_794770_31B2D5AC X-CRM114-Status: GOOD ( 15.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This reverts commit 1682c45b920643c, which is no longer needed now that we create the permanent kernel mapping directly during early boot. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 13 ++++--------- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/kernel/suspend.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- 4 files changed, 8 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index d3f8b5df0c1fe315..3c80c34f14e152d9 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -105,18 +105,13 @@ static inline void cpu_uninstall_idmap(void) cpu_switch_mm(mm->pgd, mm); } -static inline void __cpu_install_idmap(pgd_t *idmap) +static inline void cpu_install_idmap(void) { cpu_set_reserved_ttbr0(); local_flush_tlb_all(); cpu_set_idmap_tcr_t0sz(); - cpu_switch_mm(lm_alias(idmap), &init_mm); -} - -static inline void cpu_install_idmap(void) -{ - __cpu_install_idmap(idmap_pg_dir); + cpu_switch_mm(lm_alias(idmap_pg_dir), &init_mm); } /* @@ -147,7 +142,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) +static inline void cpu_replace_ttbr1(pgd_t *pgdp) { typedef void (ttbr_replace_func)(phys_addr_t); extern ttbr_replace_func idmap_cpu_replace_ttbr1; @@ -170,7 +165,7 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap) replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - __cpu_install_idmap(idmap); + cpu_install_idmap(); replace_phys(ttbr1); cpu_uninstall_idmap(); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 08ab04dc9393652a..eca9df123a8b354b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -3348,7 +3348,7 @@ subsys_initcall_sync(init_32bit_el0_mask); static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap) { - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } /* diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index 8b02d310838f9240..033cd080af680c2e 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -54,7 +54,7 @@ void notrace __cpu_suspend_exit(void) /* Restore CnP bit in TTBR1_EL1 */ if (system_supports_cnp()) - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); /* * PSTATE was not saved over suspend/resume, re-enable any detected diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index df98f496539f0e39..7e32f21fb8e1e227 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -221,7 +221,7 @@ static void __init kasan_init_shadow(void) */ memcpy(tmp_pg_dir, swapper_pg_dir, sizeof(tmp_pg_dir)); dsb(ishst); - cpu_replace_ttbr1(lm_alias(tmp_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(tmp_pg_dir)); clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -265,7 +265,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO)); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); - cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir); + cpu_replace_ttbr1(lm_alias(swapper_pg_dir)); } static void __init kasan_init_depth(void) From patchwork Fri Nov 11 17:11:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040650 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2759AC433FE for ; Fri, 11 Nov 2022 17:36:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vq+VJYmokM/LVWybEfGeWPtVvxg6zqAquWsOidle9zs=; b=VIt8Gzyw7V7bq2 5uS/D5niJuwYokDf4+LkIsYkpHt6bddSkx22jCmmHjAmbCjNY2iu5VJ6II6EqpcnmWz0a6xTPCy0P Vwx2DD0fcXdmUbGdJs8w/vrMsTomsHFyNFPplilJbMpUkT93dOfKTJ2VbKWYWuUCvE7mhXkh5wg5d 4x2uNwQO7LloPvvn6mdhTXb8qleBw3T6TggJCGrEQkYnP0WtBTVEEaREbpE9XnKD0k2lVcJi2CdVK VP3NY9a6sY18+KWSTnthgZd+Xb5Ev9vu6a3NAjURGaDYlnH11u8RjzVcjsSke7CDyCTpFNHol4gD4 Nii4SwwYBo6NxXYbGWjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXvg-00HJT1-1D; Fri, 11 Nov 2022 17:35:00 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXas-00H88u-Tb for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:34 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 67A686205A; Fri, 11 Nov 2022 17:13:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F12EBC43145; Fri, 11 Nov 2022 17:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186809; bh=tgmlFnpiF9RkwBN3vxx8H4aJOPNe1ta4QCFRQlJGuzE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QNkxN5DDp0wuCtf7Lk6fafJHNbhYMtqwTWkD+bx0RXVERMYdaGpYlDqbg+fzk3A1O 8YxEryhAtwA8ueAvgW19iCEJeXQFHOt8dF/Von9uL0iCUkMvho5y+VWOxs2OtHccyl ZQovWBamim1iGnX4KE4TbFD3yxtvZKjgtF16x1mMVlxP/tbbUNiL07dhfYEcndeN2+ WWs9i1rRipv8xH3+zCMkXRkmA4jPHJf+RM+NlLtDmYwBrR7FIhb1vUSKoYhktPzYSO QZoajESSQCEhSMwAe5KRFZObiDiHtCiapFV1FHKP/Jf6QbaqMJ5avj5efaYs8KG7Ww XWieGBtbBlzsw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 31/33] arm64: mmu: Retire SWAPPER_BLOCK_xxx and related constants Date: Fri, 11 Nov 2022 18:11:59 +0100 Message-Id: <20221111171201.2088501-32-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8726; i=ardb@kernel.org; h=from:subject; bh=tgmlFnpiF9RkwBN3vxx8H4aJOPNe1ta4QCFRQlJGuzE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJchiyWNJHejgMy5FKdUplSDIf3CTF4B1QP8xyX CVYidBWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CXAAKCRDDTyI5ktmPJCmFC/ 93qMvN/RgcKls3bgdPJ6CG4yR7czP+p7Tg/sYeM2YgKw0w7SVinOomsWapOe55gwQl1ZOoSQKrYdmX zLMk1AYKZP0MLX6M9/hWUeZ65wgZCMzYzZkLnNnKHeoDkizpeQSGzbOCiOb3bPyf+V8L5ztp7IKNY0 JXZ/b/UnbTnWZF1x2gYeEic//z8PUSbPFRJ0UMT6coLK7cUeP1R2lNK0KyE1F/s7bK3Uv4/j5ExFYw euot/fqlFSSDGTPZ8j8IVxrbbQToKOmoYggpdBzA0T25rRn5OykfR7ZPKs57qtRuaTG5JFH1XuEm1O fLoZjjooK0MdBUox9r2pRYhkDYDwo0xmrQoMs4WFArPfNCbs6w9Ysk7q9OFdXBiAcAQo32bfXesexD 9kKcnhZ2/JdaU5WI5Vc31v52lDQHtXmNe0O0ldANwBnydc4Yfbn2/1xYIDqiT2KXdZOm6wKfuZHsJF P4d/wxDAYI1V7k4BHCfyJIUOHJcK0B6fow3nZ7np4Gwoo= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091331_088464_96811981 X-CRM114-Status: GOOD ( 21.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Initially, the kernel mapping as well as the ID map used block mappings on 4k pagesize configurations, but this hasn't been the case for a long time. Currently, only the initial ID map uses the larger granularity, to simplify the early mapping code, which is implemented in assembler. The permanent ID map as well as the kernel mapping (which is now created only once) always map the kernel down to pages. This means the SWAPPER_BLOCK_xxx and related constants are no longer named appropriately, so let's rename them to INIT_IDMAP_BLOCK_xxx instead. Get rid of a stale comment while at it. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 60 +++++++------------- arch/arm64/kernel/head.S | 36 ++++++------ arch/arm64/mm/proc.S | 2 +- 3 files changed, 40 insertions(+), 58 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index ed0db7fc0022d34e..4278cd088347fefd 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -19,28 +19,13 @@ * 64K (section size = 512M). */ #ifdef CONFIG_ARM64_4K_PAGES -#define ARM64_KERNEL_USES_PMD_MAPS 1 +#define INIT_IDMAP_USES_PMD_MAPS 1 +#define INIT_IDMAP_TABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) #else -#define ARM64_KERNEL_USES_PMD_MAPS 0 +#define INIT_IDMAP_USES_PMD_MAPS 0 +#define INIT_IDMAP_TABLE_LEVELS (CONFIG_PGTABLE_LEVELS) #endif -/* - * The idmap and swapper page tables need some space reserved in the kernel - * image. Both require pgd, pud (4 levels only) and pmd tables to (section) - * map the kernel. With the 64K page configuration, swapper and idmap need to - * map to pte level. The swapper also maps the FDT (see __create_page_tables - * for more information). Note that the number of ID map translation levels - * could be increased on the fly if system RAM is out of reach for the default - * VA range, so pages required to map highest possible PA are reserved in all - * cases. - */ -#if ARM64_KERNEL_USES_PMD_MAPS -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) -#else -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) -#endif - - /* * If KASLR is enabled, then an offset K is added to the kernel address * space. The bottom 21 bits of this offset are zero to guarantee 2MB @@ -69,14 +54,14 @@ #define EARLY_PGDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) -#if SWAPPER_PGTABLE_LEVELS > 3 +#if INIT_IDMAP_TABLE_LEVELS > 3 #define EARLY_PUDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) #else #define EARLY_PUDS(vstart, vend, add) (0) #endif -#if SWAPPER_PGTABLE_LEVELS > 2 -#define EARLY_PMDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, SWAPPER_TABLE_SHIFT, add)) +#if INIT_IDMAP_TABLE_LEVELS > 2 +#define EARLY_PMDS(vstart, vend, add) (EARLY_ENTRIES(vstart, vend, INIT_IDMAP_TABLE_SHIFT, add)) #else #define EARLY_PMDS(vstart, vend, add) (0) #endif @@ -93,23 +78,23 @@ #else #define INIT_IDMAP_DIR_SIZE (INIT_IDMAP_DIR_PAGES * PAGE_SIZE) #endif -#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) +#define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + INIT_IDMAP_BLOCK_SIZE, 1) /* Initial memory map size */ -#if ARM64_KERNEL_USES_PMD_MAPS -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_BLOCK_SIZE PMD_SIZE -#define SWAPPER_TABLE_SHIFT PUD_SHIFT +#if INIT_IDMAP_USES_PMD_MAPS +#define INIT_IDMAP_BLOCK_SHIFT PMD_SHIFT +#define INIT_IDMAP_BLOCK_SIZE PMD_SIZE +#define INIT_IDMAP_TABLE_SHIFT PUD_SHIFT #else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_BLOCK_SIZE PAGE_SIZE -#define SWAPPER_TABLE_SHIFT PMD_SHIFT +#define INIT_IDMAP_BLOCK_SHIFT PAGE_SHIFT +#define INIT_IDMAP_BLOCK_SIZE PAGE_SIZE +#define INIT_IDMAP_TABLE_SHIFT PMD_SHIFT #endif /* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5 -#if SWAPPER_BLOCK_SIZE > SEGMENT_ALIGN +#if INIT_IDMAP_BLOCK_SIZE > SEGMENT_ALIGN #define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) #else #define EARLY_SEGMENT_EXTRA_PAGES 0 @@ -118,15 +103,12 @@ /* * Initial memory map attributes. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) - -#if ARM64_KERNEL_USES_PMD_MAPS -#define SWAPPER_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PMD_SECT_RDONLY) +#if INIT_IDMAP_USES_PMD_MAPS +#define INIT_IDMAP_RW_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) +#define INIT_IDMAP_RX_MMUFLAGS (INIT_IDMAP_RW_MMUFLAGS | PMD_SECT_RDONLY) #else -#define SWAPPER_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) -#define SWAPPER_RX_MMUFLAGS (SWAPPER_RW_MMUFLAGS | PTE_RDONLY) +#define INIT_IDMAP_RW_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define INIT_IDMAP_RX_MMUFLAGS (INIT_IDMAP_RW_MMUFLAGS | PTE_RDONLY) #endif /* diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 6e730a0be1e8196d..3bc96ef82f0f74e4 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -214,23 +214,23 @@ SYM_CODE_END(preserve_boot_args) populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp mov \tbl, \sv -#if SWAPPER_PGTABLE_LEVELS > 3 +#if INIT_IDMAP_TABLE_LEVELS > 3 compute_indices \vstart, \vend, #PUD_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp mov \tbl, \sv #endif -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count +#if INIT_IDMAP_TABLE_LEVELS > 2 + compute_indices \vstart, \vend, #INIT_IDMAP_TABLE_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count mov \sv, \rtbl populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp mov \tbl, \sv #endif - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count - bic \rtbl, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \rtbl, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp + compute_indices \vstart, \vend, #INIT_IDMAP_BLOCK_SHIFT, #(PAGE_SHIFT - 3), \istart, \iend, \count + bic \rtbl, \phys, #INIT_IDMAP_BLOCK_SIZE - 1 + populate_entries \tbl, \rtbl, \istart, \iend, \flags, #INIT_IDMAP_BLOCK_SIZE, \tmp .endm /* @@ -317,8 +317,8 @@ SYM_FUNC_START_LOCAL(create_idmap) #endif adrp x0, init_idmap_pg_dir adrp x3, _text - adrp x6, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - mov x7, SWAPPER_RX_MMUFLAGS + adrp x6, _end + MAX_FDT_SIZE + INIT_IDMAP_BLOCK_SIZE + mov x7, INIT_IDMAP_RX_MMUFLAGS map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT @@ -326,20 +326,20 @@ SYM_FUNC_START_LOCAL(create_idmap) adrp x1, _text adrp x2, __bss_start adrp x3, _end - bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 - mov x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT + bic x4, x2, #INIT_IDMAP_BLOCK_SIZE - 1 + mov x5, INIT_IDMAP_RW_MMUFLAGS + mov x6, #INIT_IDMAP_BLOCK_SHIFT bl remap_region /* Remap the FDT after the kernel image */ adrp x1, _text - adrp x22, _end + SWAPPER_BLOCK_SIZE - bic x2, x22, #SWAPPER_BLOCK_SIZE - 1 - bfi x22, x21, #0, #SWAPPER_BLOCK_SHIFT // remapped FDT address - add x3, x2, #MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE - bic x4, x21, #SWAPPER_BLOCK_SIZE - 1 - mov x5, SWAPPER_RW_MMUFLAGS - mov x6, #SWAPPER_BLOCK_SHIFT + adrp x22, _end + INIT_IDMAP_BLOCK_SIZE + bic x2, x22, #INIT_IDMAP_BLOCK_SIZE - 1 + bfi x22, x21, #0, #INIT_IDMAP_BLOCK_SHIFT // remapped FDT address + add x3, x2, #MAX_FDT_SIZE + INIT_IDMAP_BLOCK_SIZE + bic x4, x21, #INIT_IDMAP_BLOCK_SIZE - 1 + mov x5, INIT_IDMAP_RW_MMUFLAGS + mov x6, #INIT_IDMAP_BLOCK_SHIFT bl remap_region /* diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index f0db2c05e797aeed..b596a39394ba5363 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -206,7 +206,7 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) .pushsection ".idmap.text", "awx" From patchwork Fri Nov 11 17:12:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040652 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F3C98C4332F for ; Fri, 11 Nov 2022 17:37:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xCM4DicxGMg/osuGMPO0oBu6Y5PZPgMLoLN3IMNjTSI=; b=TgPrOQL+cv+Ti9 aOQp/oC3wnxZTf626MaGGp9qZmUUtDeSxILwvGrkMGGmiVpy/YSkb7xI+0jfMcOq7kgm1WYrVNNTy P6/ENhYszSIFH4f3/6xhA4XTt5OllNjnuig+PjlC0s/uEPLrVWL0F7JOLwFUefRqITKTrm9wMlnoY pBljfylySG+gDE8vUDXVofdL8IjJLR9amjXclhRc9FTx+7B5GD6r0czTcXMVc1bOaN+ViuzNbNPiS jx5hM+Y8exxcleckjo9SlH3TS72Cvnz+SH4MnYKVw2Ypr8k8LPOCAG4Smp/lJjQy22NnmHa3l4xZ0 D2hQcmA5/5J0pnX/rYag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXx6-00HKF4-2l; Fri, 11 Nov 2022 17:36:29 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXay-00H8Bg-6g for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:38 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9BE84B8268D; Fri, 11 Nov 2022 17:13:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 45BB0C433D7; Fri, 11 Nov 2022 17:13:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186812; bh=iteubLXOq/uhgUjHZqJzLGd7RrPuAONw2ofLvqdLHvw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m7t5EaJ2Ti03kd+xKf/2dN6qoj2ysy4LOcL0+vpXDHX4eJsrtT0XJ1esz5+du5VMg VgV17dCu1rO3s1udSB2mN7VDM6vyn5PamoVOqCpmedqliQv7yJF8iioqz1OKfDP5ZC Up8nfOU+kf3IInWG0GUsjnYsI0JNDRmNzic7pOmOL1y3TMERGEYqnGV9HDl38Xnt0y bcPVH8JuLftTcRsJDkJYMWVTfJBgFz1u8IV+W+anLVpulxgXFfqZGJdGxlyvlxK9al Fvc9I48bHW6rjxg46Wzd2zS6VhcsnPD7R7FvSgnJIut8dC6d/56PnFE4PQZiQ35W9L AWS5R7FuwRu7A== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 32/33] mm: add arch hook to validate mmap() prot flags Date: Fri, 11 Nov 2022 18:12:00 +0100 Message-Id: <20221111171201.2088501-33-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1745; i=ardb@kernel.org; h=from:subject; bh=iteubLXOq/uhgUjHZqJzLGd7RrPuAONw2ofLvqdLHvw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJeL2wALVDSOkDm1Hadkog2GbaPBv1NUcyu4fbo m+2EFhWJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CXgAKCRDDTyI5ktmPJKopDA CHYLmyjH1GHqFKRnQcJYEWGdlqI0nYFuNUv61dy1E1bRGjycQPS3U7tcHr3BHcTJFo+BgGOp663LkZ AKkMR4WJOIYs9x2xYNXzr47YKVvPNP4wos1edyAPQwQLpXmAGwzDee+lKLqaGXbwo3vxcyxKnPr1Lz ssmmohxMhrsY6KIEKY1KhZvYiIRrZ+69v8a/N9OHApjgNJfMeM5FXg3G2mlJv8C1ds1C/PdwPLetKE BKvloyNl6yvdHLDV5ComzqTAy+yJYa9X91lC11SXtwfKW4V58gjCSSanggpmwHgQGLSysaXK8DaPTp jrIqdo87xnPYt3tqd7o/5eY0ayHlvL9JewRzVyteKLTrPBO1W9gZZQXsaxF2ONPbhhDDd2qAY26n2l JMLq/Bo6blK1Hp6QUc+TktsKCcuumciK3CtdmuJtP8iEAmcaUzcCzg9wKkFTgNkhAEYE3szcHpa2Bu SaaE69oLhxOaAUFR2JVY69RlDOMPBDyvCZS1dNI8S1Ehw= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091336_583376_A435B923 X-CRM114-Status: GOOD ( 14.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add a hook to permit architectures to perform validation on the prot flags passed to mmap(), like arch_validate_prot() does for mprotect(). This will be used by arm64 to reject PROT_WRITE+PROT_EXEC mappings on configurations that run with WXN enabled. Reviewed-by: Kees Cook Signed-off-by: Ard Biesheuvel --- include/linux/mman.h | 15 +++++++++++++++ mm/mmap.c | 3 +++ 2 files changed, 18 insertions(+) diff --git a/include/linux/mman.h b/include/linux/mman.h index 58b3abd457a38df4..53ac72310ce0935d 100644 --- a/include/linux/mman.h +++ b/include/linux/mman.h @@ -120,6 +120,21 @@ static inline bool arch_validate_flags(unsigned long flags) #define arch_validate_flags arch_validate_flags #endif +#ifndef arch_validate_mmap_prot +/* + * This is called from mmap(), which ignores unknown prot bits so the default + * is to accept anything. + * + * Returns true if the prot flags are valid + */ +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return true; +} +#define arch_validate_mmap_prot arch_validate_mmap_prot +#endif + /* * Optimisation macro. It is equivalent to: * (x & bit1) ? bit2 : 0 diff --git a/mm/mmap.c b/mm/mmap.c index 2def55555e05f103..cb82740b7527680b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1262,6 +1262,9 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (!(file && path_noexec(&file->f_path))) prot |= PROT_EXEC; + if (!arch_validate_mmap_prot(prot, addr)) + return -EACCES; + /* force arch specific MAP_FIXED handling in get_unmapped_area */ if (flags & MAP_FIXED_NOREPLACE) flags |= MAP_FIXED; From patchwork Fri Nov 11 17:12:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13040653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BEF30C433FE for ; Fri, 11 Nov 2022 17:38:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=D0fa0BHH+Y8GIxnHQuSd6G7ScsjYjA3BTEoJp4Z+xvk=; b=IjDdz8fZb+Uia/ j05Z3vpp+VMI0ZvLUYPuyKqNyVLBXPJJfIdBrjwLlMC4CjywVVe71cw519pZZp2szF9sopc7FklBo viPovRQQVqxjZSQvmWgoWoA/cI/ugV152UE/jyzdoLUE1BDE59MxqhzKQZDV47gNWSttohbZvDRMm m0C0o5HzJSf3x8cYMA6bDGjhO3Nb+hYcQ7NJjdJl3DPFCnRs0Gda4zV8RRMRRFUMcxP8gtXn5nv7L eU7B/utVEINLwPImtRW01U2qnRcYLGyrKHPVET4ZgokRzDcGQH3suVUo7Z2Tv/vQ2wZjxQoSQH4p8 YYzXeLmtOYbDwsKXLE5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXxt-00HKh0-Ig; Fri, 11 Nov 2022 17:37:18 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otXay-00H8Bq-H9 for linux-arm-kernel@lists.infradead.org; Fri, 11 Nov 2022 17:13:40 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C7A7EB8268F; Fri, 11 Nov 2022 17:13:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 944EFC43143; Fri, 11 Nov 2022 17:13:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668186814; bh=ebArC0XiYBSD8p5TdHRqhvm10oMGJtMytnGQbc+7Tps=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YA6nHjaWiczlJgcYpvjGX0k6hIsWnFPo5nJEprpjjM1JyTDm1SkmNTihPkq390gfx T5GASpTr6g+1cXHu/kisUY3q+fNWrXCwhHFnP/HR6M8cYabPJxcuqZlmuzTeMuiFE3 yHurQq06UWy9yr8YinfIpLAjr2MAWtiD8pR4UsqtLZlcFK9BUjifFq0O/0K2wP0JCK DqoStgzSN8yiUH77jbHklRIUlUS83Nk465XFgfs2jskU+HG5qBB227EZbcZpRhwzos 7k74+wbmsYXD/xHixLzg3lAgq1w2aTE1sFfnSOK2tA+PdpTbli2eoo0GHppywIiKn5 95MOJIpyNujBw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual Subject: [PATCH v7 33/33] arm64: mm: add support for WXN memory translation attribute Date: Fri, 11 Nov 2022 18:12:01 +0100 Message-Id: <20221111171201.2088501-34-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221111171201.2088501-1-ardb@kernel.org> References: <20221111171201.2088501-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10074; i=ardb@kernel.org; h=from:subject; bh=ebArC0XiYBSD8p5TdHRqhvm10oMGJtMytnGQbc+7Tps=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjboJgsPlElKLjwcg6Jv7uc0Ysg4Uj0yN2mD3MScIu F0thc9eJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY26CYAAKCRDDTyI5ktmPJGG2DA COiitnNwR7xaxrgtPvzyURLFvEoVhWbY21H3Nypgf8aQpsIfmG+WLDxrss+5tPmk7vlK/svmDpMRRp hC2ZpKqrg6a3jqmZX/vPbsVIkg3pW3vRcdo/qRpvyuEk66ogv4UrJfLMvHePU1xcJ9Knr014PREW3y XnfNFLCDsosSm8A0fZnH67orTj3zAOGaEkmKt7u3DYv+LyA4YnHHL0GfwMjTxumTCdezEzgBpp/oKU iS95KiibMS2w23BNzCLdCT0WEIEVqDwD8ohUdIVFp44FD1FEEkAGlLjCfS5pUlpdIrYOFnkhUhGGZF DWT2oLEcJyLW4xA3xJ+dUVpzDDasBOrt9v85jzjhHxg+GETnOIV+mBNiDw2n8stSfib10q0QuyYfIA PmpLUDs8goaGjxgnrDxSt5I4ouH2kFawHZ5g8N1hs/54rkvLoMOvQRsAfs3bb60lXFpzBt8PW9u6lu pGxA5aB6xuZYF1isUPMUbk4u6NususAzvV04XKBD5eMP0= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221111_091336_948288_46D98DBB X-CRM114-Status: GOOD ( 31.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The AArch64 virtual memory system supports a global WXN control, which can be enabled to make all writable mappings implicitly no-exec. This is a useful hardening feature, as it prevents mistakes in managing page table permissions from being exploited to attack the system. When enabled at EL1, the restrictions apply to both EL1 and EL0. EL1 is completely under our control, and has been cleaned up to allow WXN to be enabled from boot onwards. EL0 is not under our control, but given that widely deployed security features such as selinux or PaX already limit the ability of user space to create mappings that are writable and executable at the same time, the impact of enabling this for EL0 is expected to be limited. (For this reason, common user space libraries that have a legitimate need for manipulating executable code already carry fallbacks such as [0].) If enabled at compile time, the feature can still be disabled at boot if needed, by passing arm64.nowxn on the kernel command line. [0] https://github.com/libffi/libffi/blob/master/src/closures.c#L440 Signed-off-by: Ard Biesheuvel Reviewed-by: Kees Cook --- arch/arm64/Kconfig | 11 ++++++ arch/arm64/include/asm/cpufeature.h | 10 ++++++ arch/arm64/include/asm/mman.h | 36 ++++++++++++++++++++ arch/arm64/include/asm/mmu_context.h | 30 +++++++++++++++- arch/arm64/kernel/pi/idreg-override.c | 4 ++- arch/arm64/kernel/pi/map_kernel.c | 24 +++++++++++++ arch/arm64/mm/proc.S | 6 ++++ 7 files changed, 119 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 170832f31eff4567..79ec4bc05694acec 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1509,6 +1509,17 @@ config RODATA_FULL_DEFAULT_ENABLED This requires the linear region to be mapped down to pages, which may adversely affect performance in some cases. +config ARM64_WXN + bool "Enable WXN attribute so all writable mappings are non-exec" + help + Set the WXN bit in the SCTLR system register so that all writable + mappings are treated as if the PXN/UXN bit is set as well. + If this is set to Y, it can still be disabled at runtime by + passing 'arm64.nowxn' on the kernel command line. + + This should only be set if no software needs to be supported that + relies on being able to execute from writable mappings. + config ARM64_SW_TTBR0_PAN bool "Emulate Privileged Access Never using TTBR0_EL1 switching" help diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index b8c7a2d13bbe44e2..4b5c639a5a0a7fab 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -17,6 +17,7 @@ #define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0 #define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 4 +#define ARM64_SW_FEATURE_OVERRIDE_NOWXN 8 #ifndef __ASSEMBLY__ @@ -919,6 +920,15 @@ extern struct arm64_ftr_override id_aa64isar2_override; extern struct arm64_ftr_override arm64_sw_feature_override; +static inline bool arm64_wxn_enabled(void) +{ + if (!IS_ENABLED(CONFIG_ARM64_WXN) || + cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOWXN)) + return false; + return true; +} + u32 get_kvm_ipa_limit(void); void dump_cpu_features(void); diff --git a/arch/arm64/include/asm/mman.h b/arch/arm64/include/asm/mman.h index 5966ee4a61542edf..6d4940342ba73060 100644 --- a/arch/arm64/include/asm/mman.h +++ b/arch/arm64/include/asm/mman.h @@ -35,11 +35,40 @@ static inline unsigned long arch_calc_vm_flag_bits(unsigned long flags) } #define arch_calc_vm_flag_bits(flags) arch_calc_vm_flag_bits(flags) +static inline bool arm64_check_wx_prot(unsigned long prot, + struct task_struct *tsk) +{ + /* + * When we are running with SCTLR_ELx.WXN==1, writable mappings are + * implicitly non-executable. This means we should reject such mappings + * when user space attempts to create them using mmap() or mprotect(). + */ + if (arm64_wxn_enabled() && + ((prot & (PROT_WRITE | PROT_EXEC)) == (PROT_WRITE | PROT_EXEC))) { + /* + * User space libraries such as libffi carry elaborate + * heuristics to decide whether it is worth it to even attempt + * to create writable executable mappings, as PaX or selinux + * enabled systems will outright reject it. They will usually + * fall back to something else (e.g., two separate shared + * mmap()s of a temporary file) on failure. + */ + pr_info_ratelimited( + "process %s (%d) attempted to create PROT_WRITE+PROT_EXEC mapping\n", + tsk->comm, tsk->pid); + return false; + } + return true; +} + static inline bool arch_validate_prot(unsigned long prot, unsigned long addr __always_unused) { unsigned long supported = PROT_READ | PROT_WRITE | PROT_EXEC | PROT_SEM; + if (!arm64_check_wx_prot(prot, current)) + return false; + if (system_supports_bti()) supported |= PROT_BTI; @@ -50,6 +79,13 @@ static inline bool arch_validate_prot(unsigned long prot, } #define arch_validate_prot(prot, addr) arch_validate_prot(prot, addr) +static inline bool arch_validate_mmap_prot(unsigned long prot, + unsigned long addr) +{ + return arm64_check_wx_prot(prot, current); +} +#define arch_validate_mmap_prot arch_validate_mmap_prot + static inline bool arch_validate_flags(unsigned long vm_flags) { if (!system_supports_mte()) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 3c80c34f14e152d9..4c20f7fc8abdbef9 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -19,13 +19,41 @@ #include #include #include -#include #include #include #include extern bool rodata_full; +static inline int arch_dup_mmap(struct mm_struct *oldmm, + struct mm_struct *mm) +{ + return 0; +} + +static inline void arch_exit_mmap(struct mm_struct *mm) +{ +} + +static inline void arch_unmap(struct mm_struct *mm, + unsigned long start, unsigned long end) +{ +} + +static inline bool arch_vma_access_permitted(struct vm_area_struct *vma, + bool write, bool execute, bool foreign) +{ + if (IS_ENABLED(CONFIG_ARM64_WXN) && execute && + (vma->vm_flags & (VM_WRITE | VM_EXEC)) == (VM_WRITE | VM_EXEC)) { + pr_warn_ratelimited( + "process %s (%d) attempted to execute from writable memory\n", + current->comm, current->pid); + /* disallow unless the nowxn override is set */ + return !arm64_wxn_enabled(); + } + return true; +} + static inline void contextidr_thread_switch(struct task_struct *next) { if (!IS_ENABLED(CONFIG_PID_IN_CONTEXTIDR)) diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index d0ce3dc4e07aaf4d..662c3d21e150e7f9 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -136,6 +136,7 @@ DEFINE_OVERRIDE(5, smfr0, "id_aa64smfr0", id_aa64smfr0_override, DEFINE_OVERRIDE(6, sw_features, "arm64_sw", arm64_sw_feature_override, FIELD("nokaslr", ARM64_SW_FEATURE_OVERRIDE_NOKASLR), FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF), + FIELD("nowxn", ARM64_SW_FEATURE_OVERRIDE_NOWXN), {}); /* @@ -167,7 +168,8 @@ static const struct { "id_aa64isar2.gpa3=0 id_aa64isar2.apa3=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, - { "rodata=off", "arm64_sw.rodataoff=1" }, + { "rodata=off", "arm64_sw.rodataoff=1 arm64_sw.nowxn=1" }, + { "arm64.nowxn", "arm64_sw.nowxn=1" }, }; static int __init find_field(const char *cmdline, char *opt, int len, diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 4b604b104460c3ef..2bbf017147830bbe 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -242,6 +242,25 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset) idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") disable_wxn(void) +{ + u64 sctlr = read_sysreg(sctlr_el1) & ~SCTLR_ELx_WXN; + + /* + * We cannot safely clear the WXN bit while the MMU and caches are on, + * so turn the MMU off, flush the TLBs and turn it on again but with + * the WXN bit cleared this time. + */ + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %1 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(sctlr)); +} + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) { static char const chosen_str[] __initconst = "/chosen"; @@ -255,6 +274,11 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) /* Parse the command line for CPU feature overrides */ init_feature_override(boot_status, fdt, chosen); + if (IS_ENABLED(CONFIG_ARM64_WXN) && + cpuid_feature_extract_unsigned_field(arm64_sw_feature_override.val, + ARM64_SW_FEATURE_OVERRIDE_NOWXN)) + disable_wxn(); + /* * The virtual KASLR displacement modulo 2MiB is decided by the * physical placement of the image, as otherwise, we might not be able diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index b596a39394ba5363..9d8d9d637105c200 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -465,6 +465,12 @@ SYM_FUNC_START(__cpu_setup) * Prepare SCTLR */ mov_q x0, INIT_SCTLR_EL1_MMU_ON +#ifdef CONFIG_ARM64_WXN + ldr_l x1, arm64_sw_feature_override + FTR_OVR_VAL_OFFSET + tst x1, #0xf << ARM64_SW_FEATURE_OVERRIDE_NOWXN + orr x1, x0, #SCTLR_ELx_WXN + csel x0, x0, x1, ne +#endif ret // return to head.S .unreq mair