From patchwork Wed Sep 23 05:37:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gavin Shan X-Patchwork-Id: 11793869 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35C04618 for ; Wed, 23 Sep 2020 05:38:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F0DC721D43 for ; Wed, 23 Sep 2020 05:38:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Cayq9/w6"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NbGeR41o" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F0DC721D43 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=gVXnV9335INgrOhxf6lnhuuKbKVq1hC6QBUdxz8/rRQ=; b=Cayq9/w6xBc0H4D1uqZ1/pmK1 glEfYiUTaQd4w2lcmx5C7SqrgoCgRmM51HiczBqsVDn2la0Oo+H2dFTRH2PKtWf+w7Bm73CgFzoF4 vnpkyFBeZ/sd0nfBISZ3Zbx75VFXcMlXAIujyX4vk0jIoNDXOWCFCxrpn2tTcUFw+/g2W3InxbvB+ 5W8uFMGdBkBz6vAg5M8FhL+alwW8tfpdPYZApFYLSnXPLFSrAyF9YuzDq5fRwmKvnJAPFvjfy8lc/ r0NC3EQ5GQDnbNfU3+urfvDZCu81y3JFWIE9nTlVzEbY8Iyq9tDE5ZZUR5CmdjnjENklrXb15RSNd p5QsdVB9Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kKxTn-0006mx-TS; Wed, 23 Sep 2020 05:38:11 +0000 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kKxTe-0006l9-TX for linux-arm-kernel@lists.infradead.org; Wed, 23 Sep 2020 05:38:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600839482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=havTeoYgtA9GOaGDzgKjIxi/dmZGW3kqKm8c6KarYTo=; b=NbGeR41o3cig9MsiN6YRSt01kvh5MsGzTtueaOv/6Sow8hHirEhbMh5hnPOKzcQYl7AXZ3 9KFtUYz30bINuApxp847rxXQCRzTzzzYdIQ6vA00jxoon9+FxFGS0sbTT25bdV7V9UORyT pSqn9LUd7vA5+4BFGJCBEFYovTlm4ZM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-596-Au2kdizPN4qDlg-Ig2h7AQ-1; Wed, 23 Sep 2020 01:38:00 -0400 X-MC-Unique: Au2kdizPN4qDlg-Ig2h7AQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3432F1868410; Wed, 23 Sep 2020 05:37:59 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-30.bne.redhat.com [10.64.54.30]) by smtp.corp.redhat.com (Postfix) with ESMTP id 21FBE19C4F; Wed, 23 Sep 2020 05:37:56 +0000 (UTC) From: Gavin Shan To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 1/2] arm64/mm: Introduce zero PGD table Date: Wed, 23 Sep 2020 15:37:20 +1000 Message-Id: <20200923053721.28873-2-gshan@redhat.com> In-Reply-To: <20200923053721.28873-1-gshan@redhat.com> References: <20200923053721.28873-1-gshan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200923_013802_981503_8C4F547F X-CRM114-Status: GOOD ( 15.90 ) X-Spam-Score: -1.4 (-) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-1.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [63.128.21.124 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H5 RBL: Excellent reputation (+5) [63.128.21.124 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders -1.2 DKIMWL_WL_HIGH DKIMwl.org - High trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, anshuman.khandual@arm.com, robin.murphy@arm.com, linux-kernel@vger.kernel.org, shan.gavin@gmail.com, catalin.marinas@arm.com, will@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The zero PGD table is used when TTBR_EL1 is changed. It's exactly the zero page. As the zero page(s) will be allocated dynamically when colored zero page feature is enabled in subsequent patch. the zero page(s) aren't usable during early boot stage. This introduces zero PGD table, which is decoupled from the zero page(s). Signed-off-by: Gavin Shan --- arch/arm64/include/asm/mmu_context.h | 6 +++--- arch/arm64/include/asm/pgtable.h | 2 ++ arch/arm64/kernel/setup.c | 2 +- arch/arm64/kernel/vmlinux.lds.S | 4 ++++ arch/arm64/mm/proc.S | 2 +- 5 files changed, 11 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index f2d7537d6f83..6dbc5726fd56 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -36,11 +36,11 @@ static inline void contextidr_thread_switch(struct task_struct *next) } /* - * Set TTBR0 to empty_zero_page. No translations will be possible via TTBR0. + * Set TTBR0 to zero_pg_dir. No translations will be possible via TTBR0. */ static inline void cpu_set_reserved_ttbr0(void) { - unsigned long ttbr = phys_to_ttbr(__pa_symbol(empty_zero_page)); + unsigned long ttbr = phys_to_ttbr(__pa_symbol(zero_pg_dir)); write_sysreg(ttbr, ttbr0_el1); isb(); @@ -189,7 +189,7 @@ static inline void update_saved_ttbr0(struct task_struct *tsk, return; if (mm == &init_mm) - ttbr = __pa_symbol(empty_zero_page); + ttbr = __pa_symbol(zero_pg_dir); else ttbr = virt_to_phys(mm->pgd) | ASID(mm) << 48; diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index d5d3fbe73953..6953498f4d40 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -474,6 +474,8 @@ static inline bool pud_table(pud_t pud) { return true; } PUD_TYPE_TABLE) #endif +extern pgd_t zero_pg_dir[PTRS_PER_PGD]; +extern pgd_t zero_pg_end[]; extern pgd_t init_pg_dir[PTRS_PER_PGD]; extern pgd_t init_pg_end[]; extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 53acbeca4f57..7e83eaed641e 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -366,7 +366,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) * faults in case uaccess_enable() is inadvertently called by the init * thread. */ - init_task.thread_info.ttbr0 = __pa_symbol(empty_zero_page); + init_task.thread_info.ttbr0 = __pa_symbol(zero_pg_dir); #endif if (boot_args[1] || boot_args[2] || boot_args[3]) { diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 7cba7623fcec..3d3c155d10a4 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -137,6 +137,10 @@ SECTIONS /* everything from this point to __init_begin will be marked RO NX */ RO_DATA(PAGE_SIZE) + zero_pg_dir = .; + . += PAGE_SIZE; + zero_pg_end = .; + idmap_pg_dir = .; . += IDMAP_DIR_SIZE; idmap_pg_end = .; diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 796e47a571e6..90b135c366b3 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -163,7 +163,7 @@ SYM_FUNC_END(cpu_do_resume) .pushsection ".idmap.text", "awx" .macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2 - adrp \tmp1, empty_zero_page + adrp \tmp1, zero_pg_dir phys_to_ttbr \tmp2, \tmp1 offset_ttbr1 \tmp2, \tmp1 msr ttbr1_el1, \tmp2 From patchwork Wed Sep 23 05:37:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gavin Shan X-Patchwork-Id: 11793871 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B566618 for ; Wed, 23 Sep 2020 05:39:43 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D49F221D43 for ; Wed, 23 Sep 2020 05:39:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="quH7ROnK"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KADPERAb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D49F221D43 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RSQTC2KBl4unhFwBpPKUMfMIp8hV/Mw2lB4bCp8pc8g=; b=quH7ROnK5xHywAYrGTioBvjlt 1TkdIk9ymJHOkjfp/U0jjpp5OLBfkrEViPpq8ITZiwAyMOqPXCck3rH/cRqP2LZzkw5A+TrEjC5S/ 3OJZ9bLXIsVJilAhy2z54v7kjcQOD605QCTTeMT0GKSBOmFUARp3/9dchQ9cy/6E6PEOFkGN3DWIP xfvmSCTdQhqVEvFFgBaZND0x8GkiPRmn+r9E6//Zpmjbb9dMlScRqq9+Z6DPA9N/RkjYmHN+2iGXz KSiHgKl7oHswMvJrf/iUGCycD+iLnmWxLKsbKxt8uhUJoQdmxZnKo8A8sdPlbsfoyuMBYvNpTfd/j 9p1reFVcA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kKxTr-0006nn-4D; Wed, 23 Sep 2020 05:38:15 +0000 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kKxTi-0006lv-Ke for linux-arm-kernel@lists.infradead.org; Wed, 23 Sep 2020 05:38:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600839485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OIT5FLk1kB3hxvsgpWAANVEK8Q3p4nFvDJWfSqbLRho=; b=KADPERAbeNwVwjkjCwNC+vH40zv9rwhbTef5WHJ33WWUw+0hoz7UDSa6daWIcxL0FA4luM ietY41unwlbdXKkSjnjmBWkVNAZEmoBhAaTpCuU62AHxO5bVsyA3GrdGr21T5WrcixVapW qICMOXMEvPejbuYadayY4/i4vixs8Jo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-67--edWawqzMBGJv_hnQVqstw-1; Wed, 23 Sep 2020 01:38:03 -0400 X-MC-Unique: -edWawqzMBGJv_hnQVqstw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B96CC1074641; Wed, 23 Sep 2020 05:38:01 +0000 (UTC) Received: from gshan.redhat.com (vpn2-54-30.bne.redhat.com [10.64.54.30]) by smtp.corp.redhat.com (Postfix) with ESMTP id AB06719C4F; Wed, 23 Sep 2020 05:37:59 +0000 (UTC) From: Gavin Shan To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 2/2] arm64/mm: Enable color zero pages Date: Wed, 23 Sep 2020 15:37:21 +1000 Message-Id: <20200923053721.28873-3-gshan@redhat.com> In-Reply-To: <20200923053721.28873-1-gshan@redhat.com> References: <20200923053721.28873-1-gshan@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200923_013806_723029_D554715D X-CRM114-Status: GOOD ( 28.43 ) X-Spam-Score: -1.4 (-) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-1.4 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [216.205.24.124 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H5 RBL: Excellent reputation (+5) [216.205.24.124 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders -1.2 DKIMWL_WL_HIGH DKIMwl.org - High trust sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, anshuman.khandual@arm.com, robin.murphy@arm.com, linux-kernel@vger.kernel.org, shan.gavin@gmail.com, catalin.marinas@arm.com, will@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This enables color zero pages by allocating contiguous page frames for it. The number of pages for this is determined by L1 dCache (or iCache) size, which is probbed from the hardware. * Export cache_setup_of_node() so that the cache topology could be parsed from device-tree. * Add cache_get_info() so that L1 dCache size can be retrieved. * Implement setup_zero_pages(), which is called after the page allocator begins to work, to allocate the contiguous pages needed by color zero page. * Reworked ZERO_PAGE() and define __HAVE_COLOR_ZERO_PAGE. Signed-off-by: Gavin Shan --- arch/arm64/include/asm/cache.h | 3 ++ arch/arm64/include/asm/pgtable.h | 9 ++++- arch/arm64/kernel/cacheinfo.c | 67 ++++++++++++++++++++++++++++++++ arch/arm64/mm/init.c | 37 ++++++++++++++++++ arch/arm64/mm/mmu.c | 7 ---- drivers/base/cacheinfo.c | 3 +- include/linux/cacheinfo.h | 6 +++ 7 files changed, 121 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index a4d1b5f771f6..a42dbcc6b484 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -89,6 +89,9 @@ static inline int cache_line_size_of_cpu(void) } int cache_line_size(void); +unsigned int cache_get_info(unsigned int level, unsigned int type, + unsigned int *sets, unsigned int *ways, + unsigned int *cl_size); /* * Read the effective value of CTR_EL0. diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6953498f4d40..5cb5f8bb090d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -54,8 +54,13 @@ extern void __pgd_error(const char *file, int line, unsigned long val); * ZERO_PAGE is a global shared page that is always zero: used * for zero-mapped memory areas etc.. */ -extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; -#define ZERO_PAGE(vaddr) phys_to_page(__pa_symbol(empty_zero_page)) +extern unsigned long empty_zero_page; +extern unsigned long zero_page_mask; + +#define __HAVE_COLOR_ZERO_PAGE +#define ZERO_PAGE(vaddr) \ + (virt_to_page((void *)(empty_zero_page + \ + (((unsigned long)(vaddr)) & zero_page_mask)))) #define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte)) diff --git a/arch/arm64/kernel/cacheinfo.c b/arch/arm64/kernel/cacheinfo.c index 7fa6828bb488..c13b8897323f 100644 --- a/arch/arm64/kernel/cacheinfo.c +++ b/arch/arm64/kernel/cacheinfo.c @@ -43,6 +43,73 @@ static void ci_leaf_init(struct cacheinfo *this_leaf, this_leaf->type = type; } +unsigned int cache_get_info(unsigned int level, unsigned int type, + unsigned int *sets, unsigned int *ways, + unsigned int *cl_size) +{ + int ret, i, cpu = smp_processor_id(); + enum cache_type t; + struct cpu_cacheinfo *this_cpu_ci = get_cpu_cacheinfo(cpu); + struct cacheinfo ci, *p = NULL; + + /* Sanity check */ + if (type != CACHE_TYPE_INST && type != CACHE_TYPE_DATA) + return 0; + + /* Fetch the cache information if it has been populated */ + if (this_cpu_ci->num_leaves) { + for (i = 0; i < this_cpu_ci->num_leaves; i++) { + p = &this_cpu_ci->info_list[i]; + if (p->level == level && + (p->type == type || p->type == CACHE_TYPE_UNIFIED)) + break; + } + + ret = (i < this_cpu_ci->num_leaves) ? 0 : -ENOENT; + goto out; + } + + /* + * The cache information isn't populated yet, we have to + * retrieve it from ACPI or device tree. + */ + t = get_cache_type(level); + if (t == CACHE_TYPE_NOCACHE || + (t != CACHE_TYPE_SEPARATE && t != type)) { + ret = -EINVAL; + goto out; + } + + p = &ci; + p->type = type; + p->level = level; + this_cpu_ci->info_list = p; + this_cpu_ci->num_levels = 1; + this_cpu_ci->num_leaves = 1; + if (!acpi_disabled) + ret = cache_setup_acpi(cpu); + else if (of_have_populated_dt()) + ret = cache_setup_of_node(cpu); + else + ret = -EPERM; + + memset(this_cpu_ci, 0, sizeof(*this_cpu_ci)); + +out: + if (!ret) { + if (sets) + *sets = p->number_of_sets; + if (ways) + *ways = p->ways_of_associativity; + if (cl_size) + *cl_size = p->coherency_line_size; + + return p->size; + } + + return 0; +} + static int __init_cache_level(unsigned int cpu) { unsigned int ctype, level, leaves, fw_level; diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 481d22c32a2e..330a9f610f28 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -69,6 +70,11 @@ EXPORT_SYMBOL(vmemmap); phys_addr_t arm64_dma_phys_limit __ro_after_init; static phys_addr_t arm64_dma32_phys_limit __ro_after_init; +unsigned long empty_zero_page; +EXPORT_SYMBOL(empty_zero_page); +unsigned long zero_page_mask; +EXPORT_SYMBOL(zero_page_mask); + #ifdef CONFIG_KEXEC_CORE /* * reserve_crashkernel() - reserves memory for crash kernel @@ -507,6 +513,36 @@ static void __init free_unused_memmap(void) } #endif /* !CONFIG_SPARSEMEM_VMEMMAP */ +static void __init setup_zero_pages(void) +{ + struct page *page; + unsigned int size; + int order, i; + + size = cache_get_info(1, CACHE_TYPE_DATA, NULL, NULL, NULL); + order = size > 0 ? get_order(PAGE_ALIGN(size)) : 0; + order = min(order, MAX_ORDER - 1); + + do { + empty_zero_page = __get_free_pages(GFP_KERNEL | __GFP_ZERO, + order); + if (empty_zero_page) + break; + } while (--order >= 0); + + if (!empty_zero_page) + panic("%s: out of memory\n", __func__); + + page = virt_to_page((void *) empty_zero_page); + split_page(page, order); + for (i = 1 << order; i > 0; i--) { + mark_page_reserved(page); + page++; + } + + zero_page_mask = ((PAGE_SIZE << order) - 1) & PAGE_MASK; +} + /* * mem_init() marks the free areas in the mem_map and tells us how much memory * is free. This is done after various parts of the system have claimed their @@ -527,6 +563,7 @@ void __init mem_init(void) #endif /* this will put all unused low memory onto the freelists */ memblock_free_all(); + setup_zero_pages(); mem_init_print_info(NULL); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 75df62fea1b6..736939ab3b4f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -49,13 +49,6 @@ EXPORT_SYMBOL(vabits_actual); u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); -/* - * Empty_zero_page is a special page that is used for zero-initialized data - * and COW. - */ -unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)] __page_aligned_bss; -EXPORT_SYMBOL(empty_zero_page); - static pte_t bm_pte[PTRS_PER_PTE] __page_aligned_bss; static pmd_t bm_pmd[PTRS_PER_PMD] __page_aligned_bss __maybe_unused; static pud_t bm_pud[PTRS_PER_PUD] __page_aligned_bss __maybe_unused; diff --git a/drivers/base/cacheinfo.c b/drivers/base/cacheinfo.c index 8d553c92cd32..f0dc66fc24f1 100644 --- a/drivers/base/cacheinfo.c +++ b/drivers/base/cacheinfo.c @@ -153,7 +153,7 @@ static void cache_of_set_props(struct cacheinfo *this_leaf, cache_associativity(this_leaf); } -static int cache_setup_of_node(unsigned int cpu) +int cache_setup_of_node(unsigned int cpu) { struct device_node *np; struct cacheinfo *this_leaf; @@ -195,7 +195,6 @@ static int cache_setup_of_node(unsigned int cpu) return 0; } #else -static inline int cache_setup_of_node(unsigned int cpu) { return 0; } static inline bool cache_leaves_are_shared(struct cacheinfo *this_leaf, struct cacheinfo *sib_leaf) { diff --git a/include/linux/cacheinfo.h b/include/linux/cacheinfo.h index 46b92cd61d0c..f13d625d3e76 100644 --- a/include/linux/cacheinfo.h +++ b/include/linux/cacheinfo.h @@ -100,6 +100,12 @@ struct cpu_cacheinfo *get_cpu_cacheinfo(unsigned int cpu); int init_cache_level(unsigned int cpu); int populate_cache_leaves(unsigned int cpu); int cache_setup_acpi(unsigned int cpu); +#ifdef CONFIG_OF +int cache_setup_of_node(unsigned int cpu); +#else +static inline int cache_setup_of_node(unsigned int cpu) { return 0; } +#endif + #ifndef CONFIG_ACPI_PPTT /* * acpi_find_last_cache_level is only called on ACPI enabled