From patchwork Tue Dec 12 21:34:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13490009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C27DAC4332F for ; Tue, 12 Dec 2023 21:36:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62B666B00F0; Tue, 12 Dec 2023 16:36:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DB486B00F6; Tue, 12 Dec 2023 16:36:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 455426B0285; Tue, 12 Dec 2023 16:36:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 312AC6B00F0 for ; Tue, 12 Dec 2023 16:36:04 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 04F94140B0C for ; Tue, 12 Dec 2023 21:36:03 +0000 (UTC) X-FDA: 81559474248.02.4DA04F1 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf22.hostedemail.com (Postfix) with ESMTP id 1BFADC000A for ; Tue, 12 Dec 2023 21:36:01 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=pRGr84jE; spf=pass (imf22.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702416962; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BLS75Mo77acz6k4jx++nul4Z+bTVnVv2nm8SBcxb6zo=; b=0Fde38KpVhXf6fRWqoCZP1mTT/PKWQXo+lkZu1nRGA8EQp2pT8VjvX2sm66QS8WF6xmyzF O/iP8K2CBisLg4ko8PS8C+eAWNFNyQeAZhm6HsH83s2MTK8xNMwnMuMzrI2ecpL5rLLykV bMB3z3CFR4BdFdkg0cwjXPol1nWw1X0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=pRGr84jE; spf=pass (imf22.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702416962; a=rsa-sha256; cv=none; b=oV5VH1QJ7+lvuNces2rVfu9uO8EC4td/1uMupHXifPte9eRU7iXf03ljYVbAN4GHTf7SMu 0laNZ1b5DpPW3373T0suuYVqSifpvYj6WhtbJDshe48cmDCM4rwvc+3O76kyXmu08mwqWb fiLEkUwvg652509kn5igMBPzMdrzZoA= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-40c2d50bfbfso30669055e9.0 for ; Tue, 12 Dec 2023 13:36:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1702416961; x=1703021761; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BLS75Mo77acz6k4jx++nul4Z+bTVnVv2nm8SBcxb6zo=; b=pRGr84jEaBpM4g0q/6oWjNBkfQeqtB+hgUWhV5v27OCnsTJEE+XWgNAs3BGQl+gz3b JeccMCLgvRdaWnek4+wlv33E21SRSs6uR9Akod305pjzDTsahhoLE2eeNTJgUH74FM05 /CmB9kW0G7pWO+sH+2038HZqyerD8z86ZnUxWNTfsm1EtLpfkA6TOYaoaIPCw3fOfWJX hHLnAZY5XKjeIkOxwRHYh5BaXX+KiucojMSGpcLx+SOmK0N+/jG0KiK+s/57G7Zq4m+m 7OQSThNLz2HzJebOmYMKdYUzEbh+vbJQ75/H9zLfBZ1wPmXqbN1qGN6qgUfOZrxe5IqX mEfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702416961; x=1703021761; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BLS75Mo77acz6k4jx++nul4Z+bTVnVv2nm8SBcxb6zo=; b=nfs/C+unC04hzQzC1w1iv8WFGVTMBemDO37MJQE2humuUEQXz5kosD1eNJAP4ETAcN fcekRqzRBs9+E13iUcRFnp3GFue7cjKj88+HmuElaMYwfi167lPFkbeaC5tPjk9PQUvv PKXtvclmAPaR18CHGiHbDTqMcvpb0wxQ56ktn6/KDskJQp2UVHWFS1uSFtz4cbag7n4P KFB4B0HE0eZlaR5SnjIb34d4Emy0gNfEoGI9ynDs3dXrps/01dQCsp91cfiNigBiXdMe nMXPRae76SEhGGWBbiNcCJSyysLMAm2aKsLXLQxsJYoy9v+GYZ7gapjH1SOWALjrA1AM SVAQ== X-Gm-Message-State: AOJu0Yx4/lc56+BzFVWreS9603AUsjghWIkc/uRfmJJryfA0pFFTVEZk LOXkI4Uzf2NknWTXeB7Kh7m0bXdaixa7BiY7ImU= X-Google-Smtp-Source: AGHT+IESLuoBQNsAkgxVuWPb1AKal27QjvlJbqXJPQE4veBnAzTcYr3uBF9MZ4rqSOkrlsCnVLUuRA== X-Received: by 2002:a7b:c456:0:b0:40b:5e4a:2365 with SMTP id l22-20020a7bc456000000b0040b5e4a2365mr3632834wmi.103.1702416960633; Tue, 12 Dec 2023 13:36:00 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id u21-20020a05600c139500b00405d9a950a2sm19994483wmf.28.2023.12.12.13.35.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Dec 2023 13:36:00 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 1/2] mm: Introduce flush_cache_vmap_early() Date: Tue, 12 Dec 2023 22:34:56 +0100 Message-Id: <20231212213457.132605-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231212213457.132605-1-alexghiti@rivosinc.com> References: <20231212213457.132605-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1BFADC000A X-Rspam-User: X-Stat-Signature: hm45ax17i5kd1xbdhdrr7bpi6w3irbby X-Rspamd-Server: rspam01 X-HE-Tag: 1702416961-546454 X-HE-Meta: U2FsdGVkX1+emirlpe5e23tGiAuVR1TZ8/pSz7KMnRhoKrltHNPxbDSvxbjzGj3VWeqgzYKzoaN8K+m2hJ6dHnqreNOtyecpAnhjWJneRA3O6cVcZYuyTVxgzUWhXNvSsDjgB1XWhMokZ+tRD2l3uL+UEuTCfA7OWbbuOqfKAodFxuUsOzIBG+MifWnzQVltPRj96qKcZCkJkDtP236x9KpRasjN1wZjzS/mePLrSN1sD9SGuUyYUL3W/WlV/dx+EvUOhQhFzJ21JE1gyY7tk/dUqAeQblTVAQ3mLO4E/xLxwsJLLMNOW9I9eaOvYMArKFBkYHT0gSakxkdevfM1PAe/OR100FFKXFA9lh8dBPsiQyezZO1IO4C05GEqdSwxlCSFT+Y5imvYKVsRhSadANL+sYFDMlIUYQOIVD2beXr31JmY+rLb2NVrhCIW24V3vJP2FgQTxlZ/jWIm5wKWmonIhU/nXq20EmO0Jz8tmN12YeZqXnkaWiGQ8VjHP7cyvbbgnYSK7H5HhBMk8ZRcLZ0Q4MXjxBVGBsr0DPW7APbjiYDLy81nh8ipGxV14BgBFO1fOWgoA7OXA39IeeXTUWZ/TIqT3gTcftatnyUWlfIQIH5RT2ZSfOruAh0CJnZzFdyHrxf3LV8RRIWRQ9IIzsUiPpNbZecqSnpR7NFdueMAYfs508YomCuaXtzmQ07ikT/ZnB3Vl0uf8N3TVIx+wkSSFuIVtkTtlvgqIH+VSHtUiLsHYF9B+Fc7uZyOSPs7GPbb68IjVCpe9qOpSiaWoMqV0ZmtFM8gUqQzjWw6v3Rbq9RyjGpCYdqU5QgG0oA1e9nNR5cLvMWfrBGFpjuE3Fgkj70axmFtf7ohvj6XGP91ZXwJy3amGfcu/KiUdbAJSsQus4dB4NXfJTbyfRzDtr1W5+BYhA1/vXuwLNLTssKosLg9oML7K3nFG2/0umKViM3+tP+xu4WrkrwLq45 nGFp6JpU OjMiuX2S58UbE2QGTE1GtTFTaGwTdXeEMJoJ+rqxegNc7alZf3UkPUb1h32+n9Ty6j10H+ddX4gmQu/lt7s+gPwyuzX5KuJbINZheNSWpqmYbncvieM8UIrs9sVvRmRqyUHz1HdQWsqw6ss9Q9HamLv76mLpSdc8pbwQc6Aj61CIm6yCAX0QJSwcHJyekOij3qGHam8H8bl4zphQMli4lndiNQj8O3vLP8J//MFcmi8gwnpIMIJ3QW/7pDWf5pEPBTtVUvAmQaUOeEiqvtxhRZo1RjLBGZS0SJZf6ZkCzTiBOTGAlmjJqV2Nu1ouPGxDKY8Cjp8tqyzat3yIBk4pXToK0Xywn1OUyZZBfx/BfUdc1NfDiEPlzoi9DvMK5iRZmviozuFwKCTeSJok02cS9f4EV1Z7uypVOYETLLOwSMXVoW8V3YTF9IhqjUlqD646nIT0llwBtI3NKHUu4zgsrKw1rcBIOn/q1Y4L5O+GZcm/q4OmtPkSrtOERNTKHwfWMx6djLF51VVh55rqh/5YmNWHb4ViDzLDXwH3+uvEiPczwf8Ckzl9yshs41zeOWZg/F2V0d0+VDYUOniiKk8MHfqlE+oPdUUuJZcPcJ2PV4M4pfcU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pcpu setup when using the page allocator sets up a new vmalloc mapping very early in the boot process, so early that it cannot use the flush_cache_vmap() function which may depend on structures not yet initialized (for example in riscv, we currently send an IPI to flush other cpus TLB). But on some architectures, we must call flush_cache_vmap(): for example, in riscv, some uarchs can cache invalid TLB entries so we need to flush the new established mapping to avoid taking an exception. So fix this by introducing a new function flush_cache_vmap_early() which is called right after setting the new page table entry and before accessing this new mapping. This new function implements a local flush tlb on riscv and is no-op for other architectures (same as today). Signed-off-by: Alexandre Ghiti Acked-by: Geert Uytterhoeven --- arch/arc/include/asm/cacheflush.h | 1 + arch/arm/include/asm/cacheflush.h | 2 ++ arch/csky/abiv1/inc/abi/cacheflush.h | 1 + arch/csky/abiv2/inc/abi/cacheflush.h | 1 + arch/m68k/include/asm/cacheflush_mm.h | 1 + arch/mips/include/asm/cacheflush.h | 2 ++ arch/nios2/include/asm/cacheflush.h | 1 + arch/parisc/include/asm/cacheflush.h | 1 + arch/riscv/include/asm/cacheflush.h | 3 ++- arch/riscv/include/asm/tlbflush.h | 1 + arch/riscv/mm/tlbflush.c | 5 +++++ arch/sh/include/asm/cacheflush.h | 1 + arch/sparc/include/asm/cacheflush_32.h | 1 + arch/sparc/include/asm/cacheflush_64.h | 1 + arch/xtensa/include/asm/cacheflush.h | 6 ++++-- include/asm-generic/cacheflush.h | 6 ++++++ mm/percpu.c | 8 +------- 17 files changed, 32 insertions(+), 10 deletions(-) diff --git a/arch/arc/include/asm/cacheflush.h b/arch/arc/include/asm/cacheflush.h index bd5b1a9a0544..6fc74500a9f5 100644 --- a/arch/arc/include/asm/cacheflush.h +++ b/arch/arc/include/asm/cacheflush.h @@ -40,6 +40,7 @@ void dma_cache_wback(phys_addr_t start, unsigned long sz); /* TBD: optimize this */ #define flush_cache_vmap(start, end) flush_cache_all() +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) flush_cache_all() #define flush_cache_dup_mm(mm) /* called on fork (VIVT only) */ diff --git a/arch/arm/include/asm/cacheflush.h b/arch/arm/include/asm/cacheflush.h index f6181f69577f..1075534b0a2e 100644 --- a/arch/arm/include/asm/cacheflush.h +++ b/arch/arm/include/asm/cacheflush.h @@ -340,6 +340,8 @@ static inline void flush_cache_vmap(unsigned long start, unsigned long end) dsb(ishst); } +#define flush_cache_vmap_early(start, end) do { } while (0) + static inline void flush_cache_vunmap(unsigned long start, unsigned long end) { if (!cache_is_vipt_nonaliasing()) diff --git a/arch/csky/abiv1/inc/abi/cacheflush.h b/arch/csky/abiv1/inc/abi/cacheflush.h index 908d8b0bc4fd..d011a81575d2 100644 --- a/arch/csky/abiv1/inc/abi/cacheflush.h +++ b/arch/csky/abiv1/inc/abi/cacheflush.h @@ -43,6 +43,7 @@ static inline void flush_anon_page(struct vm_area_struct *vma, */ extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #define flush_cache_vmap(start, end) cache_wbinv_all() +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) cache_wbinv_all() #define flush_icache_range(start, end) cache_wbinv_range(start, end) diff --git a/arch/csky/abiv2/inc/abi/cacheflush.h b/arch/csky/abiv2/inc/abi/cacheflush.h index 40be16907267..6513ac5d2578 100644 --- a/arch/csky/abiv2/inc/abi/cacheflush.h +++ b/arch/csky/abiv2/inc/abi/cacheflush.h @@ -41,6 +41,7 @@ void flush_icache_mm_range(struct mm_struct *mm, void flush_icache_deferred(struct mm_struct *mm); #define flush_cache_vmap(start, end) do { } while (0) +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) do { } while (0) #define copy_to_user_page(vma, page, vaddr, dst, src, len) \ diff --git a/arch/m68k/include/asm/cacheflush_mm.h b/arch/m68k/include/asm/cacheflush_mm.h index ed12358c4783..9a71b0148461 100644 --- a/arch/m68k/include/asm/cacheflush_mm.h +++ b/arch/m68k/include/asm/cacheflush_mm.h @@ -191,6 +191,7 @@ extern void cache_push_v(unsigned long vaddr, int len); #define flush_cache_all() __flush_cache_all() #define flush_cache_vmap(start, end) flush_cache_all() +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) flush_cache_all() static inline void flush_cache_mm(struct mm_struct *mm) diff --git a/arch/mips/include/asm/cacheflush.h b/arch/mips/include/asm/cacheflush.h index f36c2519ed97..1f14132b3fc9 100644 --- a/arch/mips/include/asm/cacheflush.h +++ b/arch/mips/include/asm/cacheflush.h @@ -97,6 +97,8 @@ static inline void flush_cache_vmap(unsigned long start, unsigned long end) __flush_cache_vmap(); } +#define flush_cache_vmap_early(start, end) do { } while (0) + extern void (*__flush_cache_vunmap)(void); static inline void flush_cache_vunmap(unsigned long start, unsigned long end) diff --git a/arch/nios2/include/asm/cacheflush.h b/arch/nios2/include/asm/cacheflush.h index 348cea097792..81484a776b33 100644 --- a/arch/nios2/include/asm/cacheflush.h +++ b/arch/nios2/include/asm/cacheflush.h @@ -38,6 +38,7 @@ void flush_icache_pages(struct vm_area_struct *vma, struct page *page, #define flush_icache_pages flush_icache_pages #define flush_cache_vmap(start, end) flush_dcache_range(start, end) +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) flush_dcache_range(start, end) extern void copy_to_user_page(struct vm_area_struct *vma, struct page *page, diff --git a/arch/parisc/include/asm/cacheflush.h b/arch/parisc/include/asm/cacheflush.h index b4006f2a9705..ba4c05bc24d6 100644 --- a/arch/parisc/include/asm/cacheflush.h +++ b/arch/parisc/include/asm/cacheflush.h @@ -41,6 +41,7 @@ void flush_kernel_vmap_range(void *vaddr, int size); void invalidate_kernel_vmap_range(void *vaddr, int size); #define flush_cache_vmap(start, end) flush_cache_all() +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) flush_cache_all() void flush_dcache_folio(struct folio *folio); diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 3cb53c4df27c..a129dac4521d 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -37,7 +37,8 @@ static inline void flush_dcache_page(struct page *page) flush_icache_mm(vma->vm_mm, 0) #ifdef CONFIG_64BIT -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +#define flush_cache_vmap_early(start, end) local_flush_tlb_kernel_range(start, end) #endif #ifndef CONFIG_SMP diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..a60416bbe190 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -41,6 +41,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); void flush_tlb_kernel_range(unsigned long start, unsigned long end); +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..8aadc5f71c93 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,11 @@ static inline void local_flush_tlb_range_asid(unsigned long start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } +void local_flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + local_flush_tlb_range_asid(start, end, PAGE_SIZE, FLUSH_TLB_NO_ASID); +} + static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); diff --git a/arch/sh/include/asm/cacheflush.h b/arch/sh/include/asm/cacheflush.h index 878b6b551bd2..51112f54552b 100644 --- a/arch/sh/include/asm/cacheflush.h +++ b/arch/sh/include/asm/cacheflush.h @@ -90,6 +90,7 @@ extern void copy_from_user_page(struct vm_area_struct *vma, unsigned long len); #define flush_cache_vmap(start, end) local_flush_cache_all(NULL) +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) local_flush_cache_all(NULL) #define flush_dcache_mmap_lock(mapping) do { } while (0) diff --git a/arch/sparc/include/asm/cacheflush_32.h b/arch/sparc/include/asm/cacheflush_32.h index f3b7270bf71b..9fee0ccfccb8 100644 --- a/arch/sparc/include/asm/cacheflush_32.h +++ b/arch/sparc/include/asm/cacheflush_32.h @@ -48,6 +48,7 @@ static inline void flush_dcache_page(struct page *page) #define flush_dcache_mmap_unlock(mapping) do { } while (0) #define flush_cache_vmap(start, end) flush_cache_all() +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) flush_cache_all() /* When a context switch happens we must flush all user windows so that diff --git a/arch/sparc/include/asm/cacheflush_64.h b/arch/sparc/include/asm/cacheflush_64.h index 0e879004efff..2b1261b77ecd 100644 --- a/arch/sparc/include/asm/cacheflush_64.h +++ b/arch/sparc/include/asm/cacheflush_64.h @@ -75,6 +75,7 @@ void flush_ptrace_access(struct vm_area_struct *, struct page *, #define flush_dcache_mmap_unlock(mapping) do { } while (0) #define flush_cache_vmap(start, end) do { } while (0) +#define flush_cache_vmap_early(start, end) do { } while (0) #define flush_cache_vunmap(start, end) do { } while (0) #endif /* !__ASSEMBLY__ */ diff --git a/arch/xtensa/include/asm/cacheflush.h b/arch/xtensa/include/asm/cacheflush.h index 785a00ce83c1..38bcecb0e457 100644 --- a/arch/xtensa/include/asm/cacheflush.h +++ b/arch/xtensa/include/asm/cacheflush.h @@ -116,8 +116,9 @@ void flush_cache_page(struct vm_area_struct*, #define flush_cache_mm(mm) flush_cache_all() #define flush_cache_dup_mm(mm) flush_cache_mm(mm) -#define flush_cache_vmap(start,end) flush_cache_all() -#define flush_cache_vunmap(start,end) flush_cache_all() +#define flush_cache_vmap(start,end) flush_cache_all() +#define flush_cache_vmap_early(start,end) do { } while (0) +#define flush_cache_vunmap(start,end) flush_cache_all() void flush_dcache_folio(struct folio *folio); #define flush_dcache_folio flush_dcache_folio @@ -140,6 +141,7 @@ void local_flush_cache_page(struct vm_area_struct *vma, #define flush_cache_dup_mm(mm) do { } while (0) #define flush_cache_vmap(start,end) do { } while (0) +#define flush_cache_vmap_early(start,end) do { } while (0) #define flush_cache_vunmap(start,end) do { } while (0) #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 0 diff --git a/include/asm-generic/cacheflush.h b/include/asm-generic/cacheflush.h index 84ec53ccc450..7ee8a179d103 100644 --- a/include/asm-generic/cacheflush.h +++ b/include/asm-generic/cacheflush.h @@ -91,6 +91,12 @@ static inline void flush_cache_vmap(unsigned long start, unsigned long end) } #endif +#ifndef flush_cache_vmap_early +static inline void flush_cache_vmap_early(unsigned long start, unsigned long end) +{ +} +#endif + #ifndef flush_cache_vunmap static inline void flush_cache_vunmap(unsigned long start, unsigned long end) { diff --git a/mm/percpu.c b/mm/percpu.c index 7b97d31df767..4e11fc1e6def 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3333,13 +3333,7 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t if (rc < 0) panic("failed to map percpu area, err=%d\n", rc); - /* - * FIXME: Archs with virtual cache should flush local - * cache for the linear mapping here - something - * equivalent to flush_cache_vmap() on the local cpu. - * flush_cache_vmap() can't be used as most supporting - * data structures are not set up yet. - */ + flush_cache_vmap_early(unit_addr, unit_addr + ai->unit_size); /* copy static data */ memcpy((void *)unit_addr, __per_cpu_load, ai->static_size); From patchwork Tue Dec 12 21:34:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13490010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C277C4332F for ; Tue, 12 Dec 2023 21:37:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FA518D0005; Tue, 12 Dec 2023 16:37:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A8AA8D0001; Tue, 12 Dec 2023 16:37:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1494F8D0005; Tue, 12 Dec 2023 16:37:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 00E9D8D0001 for ; Tue, 12 Dec 2023 16:37:04 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CE5A31A0A9B for ; Tue, 12 Dec 2023 21:37:04 +0000 (UTC) X-FDA: 81559476768.25.54D8095 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf01.hostedemail.com (Postfix) with ESMTP id F01A340028 for ; Tue, 12 Dec 2023 21:37:02 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=VtzUvGmb; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702417023; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qCOh69caXbVfDD+nz3SkSSNJ4LOnzQLxIG6SJxHDatw=; b=mxN2YXYt5Pf7jjwEv/ohwi5oaXCT/IO3thKfvb2LFePz2hYTeoQAnBjBo/J4A4wRWzQmuk zjMQYQbU+QRWiDCJZ3kvmZb1ebNGrXmr0o/IRKloxB6VZSwZ9HaKQeDh880fuYS3WY9A9D YkY+9V6j0KWKsmEHwlIt7iuxB52k4hM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=VtzUvGmb; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702417023; a=rsa-sha256; cv=none; b=U+1wanvT12HyJU6NlIIjP2K6uzySZ+Qb9ybl/NJwdBVRg5/qs4rE1bnWU3Es6GRIcRQvL2 Pk5BAowQA8gC/zZPrBBwCU9QSZCzlmGorHuWLWdF2oVnRMJs0sbiZzzMOLMDxQmdN6lVIf X/i+osS4JvoetBmkecBARuZOnMlhUsI= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-40c2bb872e2so58038935e9.3 for ; Tue, 12 Dec 2023 13:37:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1702417021; x=1703021821; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qCOh69caXbVfDD+nz3SkSSNJ4LOnzQLxIG6SJxHDatw=; b=VtzUvGmbV5FDawSGrZujQz1ei4mqJFJrD9rbOn1wm0lDDfTLK8Vjo7/ypm53V5Lzww 4pbXCLjwvYgxslq1OMclup0t+DuQHMeKw3qjKoP9x34J08z3xlfYsBom6w83MpfaKUL0 bzzWQ+ciSJU4GDjjZhIZnOiA89E5jrx0OgGVnexmkB3yP4K5fNe4fV0l3M2w/eKGTQAI Nz1TNuUVo1YI0ChbYO6wZVmWeTd0OSeKeGDy38XfWfn9OKob3E0xY4uMfx+xLlHzbsHt ajQCBXJwRbaXPWdDNFFpkyLkI4f3BWOSSSODZITXPVzOYvDeJaTKNc4fJaTR1BjlG02L Gt+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702417021; x=1703021821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qCOh69caXbVfDD+nz3SkSSNJ4LOnzQLxIG6SJxHDatw=; b=wbSZAmV5IhSJ11LmOWRKlaasoNEoaWf4GACGXEA2PYZLXhhdvsciioQOeeGz5afewy 7dX7nuUBE7oe9T/AzuqTkbqL5xebJH2+mPDpRI8UprWfKxgE8kMiUaUhwK79ZElvi28n L+TtqIBO/CL2oNNMPOAdyLuv3dtMdFas2YWveHe9dYnqePo5V1nHxaCWIaUIQylEWXzs oH46RYjAKgVx+fbhvrkoySu+Ajw6/2s+QmmqDyrUpUiLENl7m6PArk37U0V+dT55GgDt 3GU4LvIYy/5FNxFgd5LhKlJYmE5DNUXr7lduv2oeniI6TAaYlJRMYT/B1A45y+wiaYHg 5oWQ== X-Gm-Message-State: AOJu0Yxn5JODN2uowodtucZELleC/9hTiMcaTAh7/18BaTALOao7xlzG zEoloQyVpF8VSpWakVqs78Wksw== X-Google-Smtp-Source: AGHT+IFqqgEgVUuuLcp/jxl4AbRQVAxPSPDTSvhp/ckLGpwai+H0FWmy2ZTFwQBh1m466TRnZ7G6vg== X-Received: by 2002:a05:600c:2814:b0:40b:5f03:b43f with SMTP id m20-20020a05600c281400b0040b5f03b43fmr1885174wmb.353.1702417021655; Tue, 12 Dec 2023 13:37:01 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ay35-20020a05600c1e2300b0040b2b38a1fasm17954734wmb.4.2023.12.12.13.37.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Dec 2023 13:37:01 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Arnd Bergmann , Dennis Zhou , Tejun Heo , Christoph Lameter , Andrew Morton , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-arch@vger.kernel.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 2/2] riscv: Enable pcpu page first chunk allocator Date: Tue, 12 Dec 2023 22:34:57 +0100 Message-Id: <20231212213457.132605-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231212213457.132605-1-alexghiti@rivosinc.com> References: <20231212213457.132605-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F01A340028 X-Rspam-User: X-Stat-Signature: 9mcxumrraasss1fcqcibnt6fsb1z3gx5 X-Rspamd-Server: rspam01 X-HE-Tag: 1702417022-811886 X-HE-Meta: U2FsdGVkX19R6Se01LTQjxx5JAU8tZj8Jq8tmAbeF6Gz08F2/UTtRx6p2fY0pQJO5ORqNG/WybGpeqRTCWhP429K3GSQo7wBMLMrh4WD1aox+q5URxPOoCrDJqpuiB2aTGpBwx77UhAuKTjL3b5luhZOpvOo5jrwILCN/Jipw5Ebk8GUhA/jkFgjTHs2n2R3h6aU8BwwuD61/MkD3cuIGyAPMFu5vjzFlvYKd4+IzP70nYAGXa016CPxfr3xz0SEt7oHpK9D7jsBFbWZ4tLbHfamNACIVnU+hx3RIbeMhKq8b46dL27BghsavSjWoEVQ2GOCwSxF6smfg+Y5c5jHcbAN9hqORZToxIGNUS+Wvo9aYlU6TgyiEmjtqIxXH/yH0WDQm2LPdER3JCYWXQ1V6yT4DJjSMnqEngYnyVeME5R+13O+3BnGbtoFqU9JAapDnmCSLbIs1V3z4cVUjU23Oyd4CRfLGNMjqEnEj6PL1D6T+ui60woKbMrLApnbweuP8uWr9TVYxlVSojxSFpZibJk2t6i6gheFMFJk/CVmQ22ba+LRlUgDdQ0PINdvraimHH0iJDTeIAO3AIqu+wWlNswntySSFqAxUnGYCIZFhGOFoOmMVADkgInuHrOX+w7GDkK6NeL/C75s2LLlxq839Pno2SX8/vHV9z7BOwtg4wwZ+doOUB4fbTbS14cgHsNN+0SnlOA3dHMuCvKOXVNHWkpUHsUEKiYOmOWGMdA8PQ7VOsmq9wCfcBSz0T0qvrSb7RfZnauigdLBUyc33IN55exYJhQyqGeCt2SsOAo8UWfwfPSx3jbx6K1IG91JdW7znJQK5JdfOZqYPulGX2tE2SQjwy4TVPhhJO/VYlIyEhUBzghcLV/g6a+Ro3UEKleyFJjsZ4CKjWeXhF/AAukJezXJlZg34pybcOzIo3CesTx6q92d4igd+xnT+yDBDWtHdCgia+pPDQ1KdLNiNfR DxxFatwH KiZKI5BnnWtspjzqjgKtDjChw8HffiQEIaTBQjAXEpWqlV82pypZg6db7UCvKfpeGd3NhynFeT/0LZhCjYNSSIcpL0x+3v8v2Slgk0fGWsNaHwBJkGLha5oE7okXwGyO9PaXke3S9ktsjEf5PFANAiOI9x1WlK1FpMhIEA+nZIXfDIApBTURKhjTb5mZO7VCAaH7rLiBQ9DGXjTgolqi7n3A9B3kefzUgiGsBlvtNo0W9ifGEba7YP5bN2pmDLi+8zIDDl6mh25ReGhysLgBNXXoXtcyKBYr6wx0JrK/eMM09QpRs2+9sD/etgvoQkhyskX6ud6/kABuLxZRQbLDoKqdlTfz99233BK5qfFfrXp+qBLAlVl4CQZN02ouDSfToVu0Rw52VMvxDJZrAN7rePQ7FHzk8vthMnsxSPHBGGwjXmZ0y5cj+6uJF+UHbVgamC03kbfo03M1PgAs+5W/IMoJjf01k4KSEZ/RJdTYzjOEFRDpTUvtCxUDVkCe+1R5xDvbB5DMHQ2+8bVRtNu/jqS9jGYNXoxdQyFtVFr2+L5wBri0ggU6/ZFYkIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As explained in commit 6ea529a2037c ("percpu: make embedding first chunk allocator check vmalloc space size"), the embedding first chunk allocator needs the vmalloc space to be larger than the maximum distance between units which are grouped into NUMA nodes. On a very sparse NUMA configurations and a small vmalloc area (for example, it is 64GB in sv39), the allocation of dynamic percpu data in the vmalloc area could fail. So provide the pcpu page allocator as a fallback in case we fall into such a sparse configuration (which happened in arm64 as shown by commit 09cea6195073 ("arm64: support page mapping percpu first chunk allocator")). Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 2 ++ arch/riscv/mm/kasan_init.c | 8 ++++++++ 2 files changed, 10 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 7603bd8ab333..8ba4a63e0ae5 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -415,7 +415,9 @@ config NUMA depends on SMP && MMU select ARCH_SUPPORTS_NUMA_BALANCING select GENERIC_ARCH_NUMA + select HAVE_SETUP_PER_CPU_AREA select NEED_PER_CPU_EMBED_FIRST_CHUNK + select NEED_PER_CPU_PAGE_FIRST_CHUNK select OF_NUMA select USE_PERCPU_NUMA_NODE_ID help diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 5e39dcf23fdb..4c9a2c527f08 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -438,6 +438,14 @@ static void __init kasan_shallow_populate(void *start, void *end) kasan_shallow_populate_pgd(vaddr, vend); } +#ifdef CONFIG_KASAN_VMALLOC +void __init kasan_populate_early_vm_area_shadow(void *start, unsigned long size) +{ + kasan_populate(kasan_mem_to_shadow(start), + kasan_mem_to_shadow(start + size)); +} +#endif + static void __init create_tmp_mapping(void) { void *ptr;