From patchwork Sun Nov 1 07:46:17 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jungseok Lee X-Patchwork-Id: 7531641 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id D37FB9F399 for ; Sun, 1 Nov 2015 07:49:01 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A8473206FA for ; Sun, 1 Nov 2015 07:49:00 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7D647206F9 for ; Sun, 1 Nov 2015 07:48:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1ZsnMP-0003xx-Hn; Sun, 01 Nov 2015 07:47:29 +0000 Received: from mail-pa0-x22f.google.com ([2607:f8b0:400e:c03::22f]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1ZsnM3-0003oO-9m for linux-arm-kernel@lists.infradead.org; Sun, 01 Nov 2015 07:47:10 +0000 Received: by pasz6 with SMTP id z6so115596649pas.2 for ; Sun, 01 Nov 2015 00:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v69fUBpgL0qFMkEd0PaF6T8AKif+e8zrT68uMcwOoRE=; b=MlI2az+zPhN4R6tvh+Baoe/h0CVL8vUVm+hl2Eg9hFgBHh7WLgTfXRWhGawhzgnWlP r2QcDZzik+to7fOtgr3Jw5hM93yU075Gx4e2mmJt69mO+cqSmoCaz+MzzagjO7bYsXiq 8nNzsG/sjjwo1pWcdwgUDQ80A0NvIw009vJfNjWQfswCQGQ9Urfkp6YwmYh/SgCaiG2h A4tPdLIk2Fqt3ODYVrCdrdRgjVs6yzarAdfly5oatuFUBpEhfrYj6a1gDbr9mMZValyz MsG1nUd3DAhyz92bGXX0TdcVzoIZfsEyagozD3AKAOPMlnqDPPj6554gJAIUYlwNIUsg rJtA== X-Received: by 10.66.142.164 with SMTP id rx4mr618472pab.25.1446364006662; Sun, 01 Nov 2015 00:46:46 -0700 (PDT) Received: from ip-10-189-3-143.ap-northeast-1.compute.internal (ec2-54-238-87-144.ap-northeast-1.compute.amazonaws.com. [54.238.87.144]) by smtp.gmail.com with ESMTPSA id yq2sm17285199pbb.39.2015.11.01.00.46.43 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 01 Nov 2015 00:46:46 -0700 (PDT) From: Jungseok Lee To: catalin.marinas@arm.com, will.deacon@arm.com, cl@linux.com, tj@kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH v6 3/3] arm64: Introduce IRQ stack Date: Sun, 1 Nov 2015 07:46:17 +0000 Message-Id: <1446363977-23656-4-git-send-email-jungseoklee85@gmail.com> X-Mailer: git-send-email 2.5.0 In-Reply-To: <1446363977-23656-1-git-send-email-jungseoklee85@gmail.com> References: <1446363977-23656-1-git-send-email-jungseoklee85@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20151101_004707_594078_67A9D5A0 X-CRM114-Status: GOOD ( 20.43 ) X-Spam-Score: -2.5 (--) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, takahiro.akashi@linaro.org, barami97@gmail.com, james.morse@arm.com, linux-kernel@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, T_DKIM_INVALID, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, kernel context and interrupts are handled using a single kernel stack navigated by sp_el1. This forces a system to use 16KB stack, not 8KB one. This restriction can make low memory platforms suffer from memory pressure accompanied by performance degradation. This patch addresses the issue as introducing a separate percpu IRQ stack to handle both hard and soft interrupts with two ground rules: - Utilize sp_el0 in EL1 context, which is not used currently - Do not complicate current_thread_info calculation It is a core concept to directly retrieve struct thread_info from sp_el0. This approach helps to prevent text section size from being increased largely as removing masking operation using THREAD_SIZE in tons of places. [Thanks to James Morse for his valuable feedbacks which greatly help to figure out a better implementation. - Jungseok] Cc: AKASHI Takahiro Tested-by: James Morse Signed-off-by: Jungseok Lee --- Note that this change has been tested with 4 different combos: - THREAD_SIZE = 16KB, IRQ_STACK_SIZE = 16KB - THREAD_SIZE = 16KB, IRQ_STACK_SIZE = 8KB - THREAD_SIZE = 8KB, IRQ_STACK_SIZE = 16KB - THREAD_SIZE = 8KB, IRQ_STACK_SIZE = 8KB I've reviwed the approach using do_softirq_own_stack() which Catalin mentioned, but it is questionable to reduce max stack depth highly. That is why this hunk does not change its direction. arch/arm64/Kconfig | 1 + arch/arm64/include/asm/irq.h | 6 +++ arch/arm64/include/asm/percpu.h | 6 +++ arch/arm64/include/asm/thread_info.h | 10 +++- arch/arm64/kernel/entry.S | 42 ++++++++++++++-- arch/arm64/kernel/head.S | 5 ++ arch/arm64/kernel/irq.c | 2 + arch/arm64/kernel/sleep.S | 3 ++ arch/arm64/kernel/smp.c | 4 +- 9 files changed, 70 insertions(+), 9 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 4d8a5b2..de4e4c9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -69,6 +69,7 @@ config ARM64 select HAVE_FUNCTION_GRAPH_TRACER select HAVE_GENERIC_DMA_COHERENT select HAVE_HW_BREAKPOINT if PERF_EVENTS + select HAVE_IRQ_EXIT_ON_IRQ_STACK select HAVE_MEMBLOCK select HAVE_PATA_PLATFORM select HAVE_PERF_EVENTS diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index 0916929..d4c23bd 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -1,6 +1,11 @@ #ifndef __ASM_IRQ_H #define __ASM_IRQ_H +#define IRQ_STACK_SIZE 16384 +#define IRQ_STACK_START_SP (IRQ_STACK_SIZE - 16) + +#ifndef __ASSEMBLY__ + #include #include @@ -21,3 +26,4 @@ static inline void acpi_irq_init(void) #define acpi_irq_init acpi_irq_init #endif +#endif diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 0a456be..c581ed4 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -16,6 +16,12 @@ #ifndef __ASM_PERCPU_H #define __ASM_PERCPU_H +#ifdef CONFIG_ARM64_4K_PAGES +#include + +#define PERCPU_ATOM_SIZE IRQ_STACK_SIZE +#endif + static inline void set_my_cpu_offset(unsigned long off) { asm volatile("msr tpidr_el1, %0" :: "r" (off) : "memory"); diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 90c7ff2..abd64bd 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -73,10 +73,16 @@ register unsigned long current_stack_pointer asm ("sp"); */ static inline struct thread_info *current_thread_info(void) __attribute_const__; +/* + * struct thread_info can be accessed directly via sp_el0. + */ static inline struct thread_info *current_thread_info(void) { - return (struct thread_info *) - (current_stack_pointer & ~(THREAD_SIZE - 1)); + unsigned long sp_el0; + + asm ("mrs %0, sp_el0" : "=r" (sp_el0)); + + return (struct thread_info *)sp_el0; } #define thread_saved_pc(tsk) \ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 7ed3d75..d24acfc 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -27,6 +27,7 @@ #include #include #include +#include #include #include @@ -88,7 +89,8 @@ .if \el == 0 mrs x21, sp_el0 - get_thread_info tsk // Ensure MDSCR_EL1.SS is clear, + mov tsk, sp + and tsk, tsk, #~(THREAD_SIZE - 1) // Ensure MDSCR_EL1.SS is clear, ldr x19, [tsk, #TI_FLAGS] // since we can unmask debug disable_step_tsk x19, x20 // exceptions when scheduling. .else @@ -108,6 +110,13 @@ .endif /* + * Set sp_el0 to current thread_info. + */ + .if \el == 0 + msr sp_el0, tsk + .endif + + /* * Registers that may be useful after this macro is invoked: * * x21 - aborted SP @@ -164,8 +173,28 @@ alternative_endif .endm .macro get_thread_info, rd - mov \rd, sp - and \rd, \rd, #~(THREAD_SIZE - 1) // top of stack + mrs \rd, sp_el0 + .endm + + .macro irq_stack_entry + adr_l x19, irq_stacks + mrs x20, tpidr_el1 + add x20, x19, x20 + mov x23, sp + and x23, x23, #~(IRQ_STACK_SIZE - 1) + cmp x20, x23 // check irq re-entrance + mov x19, sp + mov x23, #IRQ_STACK_START_SP + add x23, x20, x23 // x23 = top of irq stack + csel x23, x19, x23, eq + mov sp, x23 + .endm + + /* + * x19 is preserved between irq_stack_entry and irq_stack_exit. + */ + .macro irq_stack_exit + mov sp, x19 .endm /* @@ -183,10 +212,11 @@ tsk .req x28 // current thread_info * Interrupt handling. */ .macro irq_handler - adrp x1, handle_arch_irq - ldr x1, [x1, #:lo12:handle_arch_irq] + ldr_l x1, handle_arch_irq mov x0, sp + irq_stack_entry blr x1 + irq_stack_exit .endm .text @@ -599,6 +629,8 @@ ENTRY(cpu_switch_to) ldp x29, x9, [x8], #16 ldr lr, [x8] mov sp, x9 + and x9, x9, #~(THREAD_SIZE - 1) + msr sp_el0, x9 ret ENDPROC(cpu_switch_to) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 514c1cc..353376e 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -424,6 +424,9 @@ __mmap_switched: b 1b 2: adr_l sp, initial_sp, x4 + mov x4, sp + and x4, x4, #~(THREAD_SIZE - 1) + msr sp_el0, x4 // Save thread_info str_l x21, __fdt_pointer, x5 // Save FDT pointer str_l x24, memstart_addr, x6 // Save PHYS_OFFSET mov x29, #0 @@ -604,6 +607,8 @@ ENDPROC(secondary_startup) ENTRY(__secondary_switched) ldr x0, [x21] // get secondary_data.stack mov sp, x0 + and x0, x0, #~(THREAD_SIZE - 1) + msr sp_el0, x0 // save thread_info mov x29, #0 b secondary_start_kernel ENDPROC(__secondary_switched) diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index 9f17ec0..943d106 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -30,6 +30,8 @@ unsigned long irq_err_count; +DEFINE_PER_CPU(char [IRQ_STACK_SIZE], irq_stacks) __aligned(IRQ_STACK_SIZE); + int arch_show_interrupts(struct seq_file *p, int prec) { show_ipi_list(p, prec); diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index f586f7c..e33fe33 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -173,6 +173,9 @@ ENTRY(cpu_resume) /* load physical address of identity map page table in x1 */ adrp x1, idmap_pg_dir mov sp, x2 + /* save thread_info */ + and x2, x2, #~(THREAD_SIZE - 1) + msr sp_el0, x2 /* * cpu_do_resume expects x0 to contain context physical address * pointer and x1 to contain physical address of 1:1 page tables diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 2bbdc0e..4908923 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -91,8 +91,8 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) int ret; /* - * We need to tell the secondary core where to find its stack and the - * page tables. + * We need to tell the secondary core where to find its process stack + * and the page tables. */ secondary_data.stack = task_stack_page(idle) + THREAD_START_SP; __flush_dcache_area(&secondary_data, sizeof(secondary_data));