From patchwork Thu Dec 5 10:37:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9C250E77170 for ; Thu, 5 Dec 2024 10:44:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tGON1NteLGDVWP516gtVCgFhnyyC+09X4HTwA+SV1Ck=; b=yze6NxXr1vJ/ko CgA9EGx/uFZueEJgzwzg0knp/HjV1Im+swiodkojYh6IrrE4kEU38fvDjIjB+onapamzl+ZjhAPSq 7Lo84r584JTVaaQRMMI0VFJCYXUTozcwiofyoydRRcrO0MbY4x+WVm8YGZR/eYcPZgsw7IcBBfGBT hyg4BOPI5nBrxeK9N43KY6gz3KRlvG5qIoPOnXPrdnn04eqnuxxQjiiVSRifrzez5P5Rhl23NtQGZ KPns/MIecNLRERECaCahjy2wfzDiUSerWwqX4zNhR9GibbmowjMIv5eBmifpDvfJM4JZbQSRIR4Jk +B2tMSAxdeTKQ9VvbPhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ll-0000000FaMX-3mcx; Thu, 05 Dec 2024 10:44:49 +0000 Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ew-0000000FXx8-2LiG for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:37:47 +0000 Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-7253bc4d25eso622038b3a.0 for ; Thu, 05 Dec 2024 02:37:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395063; x=1733999863; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KWkBqx6Qlg0tee9JR/yzxG5KwDs2r7O63kRKnpevvpY=; b=HAsGr5IoCMGD/VTpFNANpzIu8y41Y7EDWOr2dcDRtyuwImM9w8pSfgUkq+KYW46efD bgkZoQZ3XmMiYRpXNUOPoi2rePAPl/IEhflmOQgL5CXqUx/0k6qXHVEUjoFXvFjbbfNh KZ2tSeXppstidxG1lTHWqxQO8pHLptwVwDd3wQvTRb2NVyLP6qjQtbKmNvPnMr5NIMIx z6WBev0EjXDt+Sq+bIR7hu8dFMrDHaoNndtW8K+9yWPpkVZOrXB5q/4kGZ2lHGLQbSug iJpNlbYUznrBf91V+6dDMnMTPzYETDPPGSLtwNJoRU84JI2RYK3mZuJzD44hmq6Q5LhN 5gaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395063; x=1733999863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KWkBqx6Qlg0tee9JR/yzxG5KwDs2r7O63kRKnpevvpY=; b=MRjR1ohrybl/28QIyRJO+88yMaQayi/D6rpGofH+Tx/Lg5vyy3O0Dk3W8Q3imGkrki V4GPokGR97nQnBnu5bp/0llnU6OirE9YQG9M68l7jPhaRfHIVdH/FrQGNbf3D3cTmnam DsSiv7NwzvH8y3FaaK2PAbpSduy7/7W1KC5L5wWbZEfPo+j1rf3E63EnS4nHYLlxvNBW 3xizoZcLw+jpO9XB0uOmRhey2exaFSlCsx+sntQjWqMLuyxog+uaqmlF4zJOSg64G1Hh lDUSnNuLarY/kWpjzddXyqFDxyzwqbQPJs8Xx1U8S/k977DI2T39OIhsaCMXnU7Gykxk YWwA== X-Forwarded-Encrypted: i=1; AJvYcCUJiIHZPJGgzgwHupec33pVe1J+7n9ZsZ4G1t8ojN8JzicUJy2qkx3uEHZwjKiauMr8JkrGAAtCOPCIQQ==@lists.infradead.org X-Gm-Message-State: AOJu0Yy5YUvAseq7RB7lZiHfGAX0v75JLD8OXlexs/UBnEaRppMlQ3k7 ob0AdE39ZdmVpzJc1AEkGq+7OSVBKOGD4d7asAfDUf8aGidc6hfNy5ELMhKKTRw= X-Gm-Gg: ASbGncsoZglZ2vwLaLao7ZFiInCdPcLYpoF/AjCq1nmxCsh6BmwVJp1mCtxvJiinI/0 LHqeFF2sn94rya0m/smrOKrz/ulOM8hsn1S9M9/1V5FjCATLUSTtI9G3z2Sccn0wdJrCACdPGFQ IUBqKXtRSpfPgMhNeiWjEAbE9Uj07wlxufJctvk7fPOG+0yQN6p6DNj4ndXgQkBuWyMgyrrB2Xt hcEg3fce9quW/hiexTYbLpGweciYFgvARXOhMlfzn7DxG2+/z+C4glRoI23DUFmcgUvsbSNjxTC KUvh+05ViLqQx1KFnRhs6j/6m7K+r0jj X-Google-Smtp-Source: AGHT+IEvp+5QW/iv1/V1tpDVHfeo0s8LNYsKCxdjWx85NP0irBg++mJLHUVM5oudhxdOyXEvXSVVbg== X-Received: by 2002:a17:902:e00c:b0:215:7136:f7e3 with SMTP id d9443c01a7336-215f3ce4f83mr33803375ad.19.1733395062764; Thu, 05 Dec 2024 02:37:42 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:42 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 01/21] riscv: mm: Distinguish hardware base page and software base page Date: Thu, 5 Dec 2024 18:37:09 +0800 Message-Id: <20241205103729.14798-2-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023746_595882_1E359ED0 X-CRM114-Status: GOOD ( 15.99 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The key idea to implement larger base page based on MMU that only supports 4K page is to decouple the MMU page from the software page in view of kernel mm. In contrary to software page, we denote the MMU page as hardware page. To decouple these two kinds of pages, we should manage, allocate and map memory at a granularity of software page, which is exactly what existing mm code does. The page table operations, however, should configure page table entries at a granularity of hardware page, which is the responsibility of arch code. This commit introduces the concept of hardware base page for RISCV. Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 10 ++++++++++ arch/riscv/include/asm/page.h | 7 +++++++ arch/riscv/include/asm/pgtable-32.h | 5 +++-- arch/riscv/include/asm/pgtable-64.h | 5 +++-- arch/riscv/include/asm/pgtable-bits.h | 3 ++- arch/riscv/include/asm/pgtable.h | 1 + 6 files changed, 26 insertions(+), 5 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index fa8f2da87a0a..2c0cb175a92a 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -289,6 +289,16 @@ config PAGE_OFFSET default 0xc0000000 if 32BIT default 0xff60000000000000 if 64BIT +config RISCV_HW_PAGE_SHIFT + int + default 12 + +config RISCV_USE_SW_PAGE + bool + depends on 64BIT + depends on RISCV_HW_PAGE_SHIFT != PAGE_SHIFT + default n + config KASAN_SHADOW_OFFSET hex depends on KASAN_GENERIC diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 32d308a3355f..7c581a3e057b 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -12,6 +12,10 @@ #include #include +#define HW_PAGE_SHIFT CONFIG_RISCV_HW_PAGE_SHIFT +#define HW_PAGE_SIZE (_AC(1, UL) << HW_PAGE_SHIFT) +#define HW_PAGE_MASK (~(HW_PAGE_SIZE - 1)) + #define PAGE_SHIFT CONFIG_PAGE_SHIFT #define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT) #define PAGE_MASK (~(PAGE_SIZE - 1)) @@ -185,6 +189,9 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #define __pa(x) __virt_to_phys((unsigned long)(x)) #define __va(x) ((void *)__pa_to_va_nodebug((phys_addr_t)(x))) +#define pfn_to_hwpfn(pfn) (pfn << (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define hwpfn_to_pfn(hwpfn) (hwpfn >> (PAGE_SHIFT - HW_PAGE_SHIFT)) + #define phys_to_pfn(phys) (PFN_DOWN(phys)) #define pfn_to_phys(pfn) (PFN_PHYS(pfn)) diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/pgtable-32.h index 00f3369570a8..159a668c3dd8 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -20,9 +20,10 @@ /* * rv32 PTE format: * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 - * PFN reserved for SW D A G U X W R V + * HW_PFN reserved for SW D A G U X W R V */ -#define _PAGE_PFN_MASK GENMASK(31, 10) +#define _PAGE_HW_PFN_MASK GENMASK(31, 10) +#define _PAGE_PFN_MASK GENMASK(31, (10 + PAGE_SHIFT - HW_PAGE_SHIFT)) #define _PAGE_NOCACHE 0 #define _PAGE_IO 0 diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 0897dd99ab8d..963aa4be9eed 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -72,9 +72,10 @@ typedef struct { /* * rv64 PTE format: * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 - * N MT RSV PFN reserved for SW D A G U X W R V + * N MT RSV HW_PFN reserved for SW D A G U X W R V */ -#define _PAGE_PFN_MASK GENMASK(53, 10) +#define _PAGE_HW_PFN_MASK GENMASK(53, 10) +#define _PAGE_PFN_MASK GENMASK(53, (10 + PAGE_SHIFT - HW_PAGE_SHIFT)) /* * [63] Svnapot definitions: diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h index a8f5205cea54..e5bb6a805505 100644 --- a/arch/riscv/include/asm/pgtable-bits.h +++ b/arch/riscv/include/asm/pgtable-bits.h @@ -31,7 +31,8 @@ /* Used for swap PTEs only. */ #define _PAGE_SWP_EXCLUSIVE _PAGE_ACCESSED -#define _PAGE_PFN_SHIFT 10 +#define _PAGE_HWPFN_SHIFT 10 +#define _PAGE_PFN_SHIFT (_PAGE_HWPFN_SHIFT + (PAGE_SHIFT - HW_PAGE_SHIFT)) /* * when all of R/W/X are zero, the PTE is a pointer to the next level diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index e79f15293492..9d6d0ff86c76 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -114,6 +114,7 @@ #include #include +#define __page_val_to_hwpfn(_val) (((_val) & _PAGE_HW_PFN_MASK) >> _PAGE_HWPFN_SHIFT) #define __page_val_to_pfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT) #ifdef CONFIG_64BIT From patchwork Thu Dec 5 10:37:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FEEEE7716C for ; Thu, 5 Dec 2024 10:44:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yEXztEQru495sbpjDgV9UGiOS7KrZQxVQIeum8FhVwY=; b=oz8+oO3UizPlFb 4KGcPA485yUFZaz1/5CCxZxYhsOKfCGdtWRrfuyVQmF2vOuPpdFj7fc1XnRGKWjryLwjhOSnY6bOR bKz7+rRsxLf3Fz5z7l9wfQHC0l5jcbNK1ftmm7odwdgM2h9VEr0mveC1WHwWiiUI770rVd3kNKJXz vjK0IaMdqI2HHbEjwad7j0PYM/TCIR54433hZ4X0vw0LLznMu7qDcWU9MKlIlcjmSm+FfseKeYqft GTWgetQ7p7Iue6rGgCKwWd94vsX8TlzIBfadjOkk36rYmPgJNf24LfWKrxq60cwVxv64xgbtEClml oya5lPsDvdSFKvOJo5zg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lm-0000000FaMi-25cH; Thu, 05 Dec 2024 10:44:50 +0000 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ey-0000000FY0N-3HnX for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:37:49 +0000 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-215666ea06aso15455225ad.0 for ; Thu, 05 Dec 2024 02:37:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395068; x=1733999868; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UNKH3fzWVL17kpFj96xVkU+IFapkCvyeY63L7HJogMk=; b=dNpRxqiqOciAPJY4zFX9tRcz0qx82pgIW3XvUjbpNJaHSWyZ7w6cGU7AvV6e9rbA8f u0v2Ry8UwQUhxYdKbiGXlHBEtdIAGcb82O86tkGrJgxEQFgKRBd205Hz1XQoLcfnDM3K 11fsnbGyGIWJmf9Hm3aTewoNyq/kcdcDTl7liM20J0Kn/wLvKdMQR4eHcgM2d3qfzVtJ u4DHd/4eqixPmABPVHOSLXm+ROsBMZYytmL1zfQ1RJ6cp7SD5h1IUjJxYEqUFuoqCCq8 t55jVda7kFx8Cgd6bOPPa/FIQ2niBZvsp3uq2rP9gAzFj3CaUZ/+vEJGkNw6SuWpSlst nQiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395068; x=1733999868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UNKH3fzWVL17kpFj96xVkU+IFapkCvyeY63L7HJogMk=; b=L5wN0n2nVf9bpDFcEvz48LLDXd+WI3RXeCGVz1yXPe3M44dvwgbd6JWSKhzSiF8HN9 3n1PBseMqA0wiBRc528UmwrmN1AOwD+eSJBIzXxei9j7x8BQKpMfFLXhWxbEmA2/Ic5e tq1MS4cGpU/crQsyik5DGyFwCAEB4OAc6AoeTXzxJ1RakQR2SfuIsgfkFg4d5VjmvYT2 M8O7wxkB8Ts/kPiEnaivPrg7HwL+Q9dLpPhzLNaVRdS/+KuXd1yZXQomdxGjbSoLOTs5 4lHj6DNoF4QUIUdIwREQh0GaZBiExXHshBasHzSPBoQvEzF3igV1rmG3/dtHKVhun+EC Z2+w== X-Forwarded-Encrypted: i=1; AJvYcCU8G3Tl/129PuU1Zea2g0d1sxUgvMrQU91NxCpbD1BNIPvNusug9BEFBAgzuPG9oVC8vwNK7NW+aBZt5g==@lists.infradead.org X-Gm-Message-State: AOJu0YyaCGA/V5ypOA0IXKNeANOCxI4zqzN5Qj3OA//4eBRRQorj4yN0 W8A459hP6fKWYirWUvzvBbYpUrEAvzff1Hmk04ynqBaa/Acqt0wMoMKnXjaHVY85UFDA8K0zXce vXQ0= X-Gm-Gg: ASbGnctjRsd51Sycd95USXEm3AdQc0DheVhMqSAVtKMcQbzefIqc6ZCUT5aiO1Rgd25 1W7bJfIclgnbfUde3nmuE8ZeEFUT/zr6SqHAdClnHxQD/wwv67sRnQI4XOoq3La2dgVSxzjgRz0 j91s2CBa2IRowogFaX2XTU7FKXZoLOd88ZnVaBZhdcdcFOSMOEZTIbh/7wCu5wlH3Efzn6Bi5NJ KSXTQxyrwEGJEftMDz/3OZSRmmm4gbL/9fFRSGBHcDiH6Ode/3x9QsXcWq/hM1H5Fgug5jl32qW kNpZjaTKinU2UwajB992BUU57vY+Ggvh X-Google-Smtp-Source: AGHT+IGkn+Ya+kK82LZNSNu9BskAONVJ6LJWivbKYKO/n1RCbEvXyllqnQCmoUfNjTenfPmHl113mQ== X-Received: by 2002:a17:902:d48a:b0:215:8112:2fb with SMTP id d9443c01a7336-215f3c7ec62mr46538845ad.13.1733395066881; Thu, 05 Dec 2024 02:37:46 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:46 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 02/21] riscv: mm: Configure satp with hw page pfn Date: Thu, 5 Dec 2024 18:37:10 +0800 Message-Id: <20241205103729.14798-3-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023748_822614_7E158836 X-CRM114-Status: GOOD ( 15.32 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The control register CSR_SATP on RISC-V, which points to the root page table page, is used by MMU to translate va to pa when TLB miss happens. Thus it should be encoded at a granularity of hardware page, while existing code usually encodes it via software page frame number. This commit corrects encoding operations of CSR_SATP register. To get developers rid of the annoying encoding format of CSR_SATP and the conversion between sw pfn and hw pfn, we abstract the encoding operations of CSR_SATP into a specific function. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 14 ++++++++++++++ arch/riscv/kernel/head.S | 4 ++-- arch/riscv/kernel/hibernate.c | 3 ++- arch/riscv/mm/context.c | 7 +++---- arch/riscv/mm/fault.c | 2 +- arch/riscv/mm/init.c | 7 +++++-- arch/riscv/mm/kasan_init.c | 7 +++++-- 7 files changed, 32 insertions(+), 12 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9d6d0ff86c76..9d3947ec3523 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -206,6 +206,20 @@ extern pgd_t swapper_pg_dir[]; extern pgd_t trampoline_pg_dir[]; extern pgd_t early_pg_dir[]; +static inline unsigned long make_satp(unsigned long pfn, + unsigned long asid, unsigned long satp_mode) +{ + return (pfn_to_hwpfn(pfn) | + ((asid & SATP_ASID_MASK) << SATP_ASID_SHIFT) | satp_mode); +} + +static inline unsigned long satp_pfn(unsigned long satp) +{ + unsigned long hwpfn = satp & SATP_PPN; + + return hwpfn_to_pfn(hwpfn); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline int pmd_present(pmd_t pmd) { diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 356d5397b2a2..b8568e3ddefa 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -86,7 +86,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a2 /* Compute satp for kernel page tables, but don't load it yet */ - srl a2, a0, PAGE_SHIFT + srl a2, a0, HW_PAGE_SHIFT la a1, satp_mode XIP_FIXUP_OFFSET a1 REG_L a1, 0(a1) @@ -100,7 +100,7 @@ relocate_enable_mmu: */ la a0, trampoline_pg_dir XIP_FIXUP_OFFSET a0 - srl a0, a0, PAGE_SHIFT + srl a0, a0, HW_PAGE_SHIFT or a0, a0, a1 sfence.vma csrw CSR_SATP, a0 diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 671b686c0158..155be6b1d32c 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -395,7 +395,8 @@ int swsusp_arch_resume(void) if (ret) return ret; - hibernate_restore_image(resume_hdr.saved_satp, (PFN_DOWN(__pa(resume_pg_dir)) | satp_mode), + hibernate_restore_image(resume_hdr.saved_satp, + make_satp(PFN_DOWN(__pa(resume_pg_dir)), 0, satp_mode), resume_hdr.restore_cpu_addr); return 0; diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 4abe3de23225..229c78d9ad3a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -189,9 +189,8 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) raw_spin_unlock_irqrestore(&context_lock, flags); switch_mm_fast: - csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - (cntx2asid(cntx) << SATP_ASID_SHIFT) | - satp_mode); + csr_write(CSR_SATP, make_satp(virt_to_pfn(mm->pgd), cntx2asid(cntx), + satp_mode)); if (need_flush_tlb) local_flush_tlb_all(); @@ -200,7 +199,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ - csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); + csr_write(CSR_SATP, make_satp(virt_to_pfn(mm->pgd), 0, satp_mode)); local_flush_tlb_all_asid(0); } diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index a9f2b4af8f3f..4772152be0f9 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -133,7 +133,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * of a task switch. */ index = pgd_index(addr); - pfn = csr_read(CSR_SATP) & SATP_PPN; + pfn = satp_pfn(csr_read(CSR_SATP)); pgd = (pgd_t *)pfn_to_virt(pfn) + index; pgd_k = init_mm.pgd + index; diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 0e8c20adcd98..f9334aab45a6 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -836,7 +836,7 @@ static __init void set_satp_mode(uintptr_t dtb_pa) (uintptr_t)early_p4d : (uintptr_t)early_pud, PGDIR_SIZE, PAGE_TABLE); - identity_satp = PFN_DOWN((uintptr_t)&early_pg_dir) | satp_mode; + identity_satp = make_satp(PFN_DOWN((uintptr_t)&early_pg_dir), 0, satp_mode); local_flush_tlb_all(); csr_write(CSR_SATP, identity_satp); @@ -1316,6 +1316,8 @@ static void __init create_linear_mapping_page_table(void) static void __init setup_vm_final(void) { + unsigned long satp; + /* Setup swapper PGD for fixmap */ #if !defined(CONFIG_64BIT) /* @@ -1349,7 +1351,8 @@ static void __init setup_vm_final(void) clear_fixmap(FIX_P4D); /* Move to swapper page table */ - csr_write(CSR_SATP, PFN_DOWN(__pa_symbol(swapper_pg_dir)) | satp_mode); + satp = make_satp(PFN_DOWN(__pa_symbol(swapper_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); local_flush_tlb_all(); pt_ops_set_late(); diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index c301c8d291d2..3eee1665358e 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -482,11 +482,13 @@ static void __init create_tmp_mapping(void) void __init kasan_init(void) { + unsigned long satp; phys_addr_t p_start, p_end; u64 i; create_tmp_mapping(); - csr_write(CSR_SATP, PFN_DOWN(__pa(tmp_pg_dir)) | satp_mode); + satp = make_satp(PFN_DOWN(__pa(tmp_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); kasan_early_clear_pgd(pgd_offset_k(KASAN_SHADOW_START), KASAN_SHADOW_START, KASAN_SHADOW_END); @@ -531,6 +533,7 @@ void __init kasan_init(void) memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); init_task.kasan_depth = 0; - csr_write(CSR_SATP, PFN_DOWN(__pa(swapper_pg_dir)) | satp_mode); + satp = make_satp(PFN_DOWN(__pa(swapper_pg_dir)), 0, satp_mode); + csr_write(CSR_SATP, satp); local_flush_tlb_all(); } From patchwork Thu Dec 5 10:37:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894982 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FB3AE77175 for ; Thu, 5 Dec 2024 10:44:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HahNfaI24yWErTy2Q/nwRTjmlL/hgb9L+P40xw/k4cg=; b=SG+AFuOxeoGgRH 0KLMgu1LtC1jDrssGJkLwREd6NsKB6mkQ3suobsiTeiRl1hjIhfM2Bvw2te2F+0oeDh8rDTdPHS7U yetlE4jkNYmgZKaqaN66WVUUrk0wnu/epi/fy3va+ALneddMUUqr/YymR51fJhYGyr+i/5FcCrRdc 0le5jeDmcfsgC9vqZAjbO6AIKOh+DUZLVNinkMB8NJGgXLyqgbkZyJvM5OCaeMoiKGcai/MfjuFzk VMKGvwFHmEPC6zqBNXYI1HO4IvLRj4JTyb3qEJEQenw+QEaQOFe9nk0DzzSWkPpzRSvBqqP8LtBFG sbwnIjB2cFsfulT/749w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ln-0000000FaO1-1BE1; Thu, 05 Dec 2024 10:44:51 +0000 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9F3-0000000FY38-1Gqx for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:37:54 +0000 Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-7ee11ff7210so653135a12.1 for ; Thu, 05 Dec 2024 02:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395072; x=1733999872; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=teCsoKxDGeiN3+DzdoxY3wVIl2UTRwagjcvIS6Z3bMs=; b=EgnmuJS6kv+hVuokJZNgCmjyeGmhwpHhLqIdyp5qdQKKYiA4u5EMA8QBf/riR7i1bU MjEKH8CZoHaXzkaUYMtQW+aLdSReSjEZs+G+vY66ysWAKbQBvlfHRaA0vvSSXlfzDX5w 0MQfNAx4hNgjMJQi8i44POAt9hAi7BioLMF1wvk2CDHZI1bx+JNbwiuG/eAfFigtv/ZA KtBNR5we0LqGSPpTizc7KSqj6KqGIxHVQ57Dcy1aHfWTQ5sQpGxid98h4OOOA+jnu0z9 UYSAAZC0tI7MXS+craHUgR2c4QfDvKmdhHaGvfeRK2k7heEXa8wf3bFVUUqXZ5rQLBi9 4EUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395072; x=1733999872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=teCsoKxDGeiN3+DzdoxY3wVIl2UTRwagjcvIS6Z3bMs=; b=MCDgwmE5wY+riY4ofBm0cJU4JzaSB3ERlhTNJ9xa0Fqy8PYGCkAXJboMtAJm98XlFy 8955TN1cY30BrW+QmFRLVPV4vY/ltIEBWFerEGUUiYBkkjJSC4stKNI+CwJxVLTg1DdF 5Z2M07PKlIcl0PosYXo9OTBTXi4d5iYrugj6dYZSizF4o0rTp2+9iYC28ywyRFN8Zazj 2Y7H5u1s97Gmkn4SjXxCdWHwaql+Du23Lj70TAqVLiFLHmQwusfrRYAq3qluCQKBKb0v hg/s1l0OJMbRFudDPMjVdA34z/p/6PLLX/EhDHZWRr1m14+L7HxYdM3+C+tuotzUzFby cmpA== X-Forwarded-Encrypted: i=1; AJvYcCWjI7EGYHuQQbzIi4icRvDkj6m7ii6C1wlSnebRhVBtz/EFwZ5HQeKa1feO9XR+B4QAJ3xIAOEsJPndGQ==@lists.infradead.org X-Gm-Message-State: AOJu0Yz0cDptPvFb5LLxz8mwQ2KgQTa2gNVtYLltWjff3eUeqZ4vYdPu iwWwNsCAHO6Yqf23lP/hLwMZaRV+1/i/0+Efn5APR6kKMjQhE1fb8vxDaIZCvj0= X-Gm-Gg: ASbGncujEkCI5SwB8xUyw5B/U5fdT3P62NbIo6TksubBGdaFMMWytzm2Z3Z2X4RsbJA l58GMbnWeuwyHpPYYgPoE6iE3v3d8vuZUkgYoNywuylYsd97PMGkbh8OiY2baNMAE2BaCm7LfD1 P3nVGiylj+Bvi58zus5UX/eT2d/cdpErbI2CPp8ADEycl5h2wzgfdgB1+i5ahPf0GyoDodoKVTW p4nbbyAfZ9Af8PuLjb8g08H/leiTxXp9VAseeZehTnSpqD/q/WBe3p1KSjtCfvGvSyP/8218mle a8wM3lvTeJT95/X+W46hv/7+Jgi2dJ1+ X-Google-Smtp-Source: AGHT+IHv/nBjYCvlcRUcvZSgl+N0//z6T9cTlerWiKA9ANKuf1lPaFN2gvQdQxXb4nAsIJJZSEYzng== X-Received: by 2002:a05:6a20:1d98:b0:1db:efab:390c with SMTP id adf61e73a8af0-1e1653f3f22mr17696126637.35.1733395071049; Thu, 05 Dec 2024 02:37:51 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:50 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 03/21] riscv: mm: Reimplement page table entry structures Date: Thu, 5 Dec 2024 18:37:11 +0800 Message-Id: <20241205103729.14798-4-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023753_340239_AE806CD8 X-CRM114-Status: GOOD ( 16.12 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After decoupling hardware base page and software base page, each software page can consists of several hardware base pages now. The pte struct should be turned to an array of mapping entires to map the software page. For example, in 64K Page Size kernel, each software page consists of 16 contiguous hardware pages. Thus the pte struct should contains 16 mapping entries to map 16 hardware pages. This commit reimplements pte structure. Signed-off-by: Xu Lu --- arch/riscv/include/asm/page.h | 43 +++++++++++++++++++++++---- arch/riscv/include/asm/pgtable-64.h | 41 +++++++++++++++++++++---- arch/riscv/include/asm/pgtable.h | 23 +++++++++++++-- arch/riscv/mm/pgtable.c | 46 +++++++++++++++++++++++++++++ 4 files changed, 141 insertions(+), 12 deletions(-) diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 7c581a3e057b..9bc908d94c7a 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -63,6 +63,36 @@ void clear_page(void *page); * Use struct definitions to apply C type checking */ +#ifdef CONFIG_RISCV_USE_SW_PAGE + +#define HW_PAGES_PER_PAGE (1 << (PAGE_SHIFT - HW_PAGE_SHIFT)) + +struct page_table_entry { + union { + unsigned long pgds[HW_PAGES_PER_PAGE]; + unsigned long p4ds[HW_PAGES_PER_PAGE]; + unsigned long puds[HW_PAGES_PER_PAGE]; + unsigned long pmds[HW_PAGES_PER_PAGE]; + unsigned long ptes[HW_PAGES_PER_PAGE]; + }; +}; + +/* Page Global Directory entry */ +typedef struct page_table_entry pgd_t; + +/* Page Table entry */ +typedef struct page_table_entry pte_t; + +#define pte_val(x) ((x).ptes[0]) +#define pgd_val(x) ((x).pgds[0]) + +pte_t __pte(unsigned long pteval); +pgd_t __pgd(unsigned long pgdval); +#define __pte __pte +#define __pgd __pgd + +#else /* CONFIG_RISCV_USE_SW_PAGE */ + /* Page Global Directory entry */ typedef struct { unsigned long pgd; @@ -73,18 +103,21 @@ typedef struct { unsigned long pte; } pte_t; +#define pte_val(x) ((x).pte) +#define pgd_val(x) ((x).pgd) + +#define __pte(x) ((pte_t) { (x) }) +#define __pgd(x) ((pgd_t) { (x) }) + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + typedef struct { unsigned long pgprot; } pgprot_t; typedef struct page *pgtable_t; -#define pte_val(x) ((x).pte) -#define pgd_val(x) ((x).pgd) #define pgprot_val(x) ((x).pgprot) - -#define __pte(x) ((pte_t) { (x) }) -#define __pgd(x) ((pgd_t) { (x) }) #define __pgprot(x) ((pgprot_t) { (x) }) #ifdef CONFIG_64BIT diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 963aa4be9eed..e736873d7768 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -41,6 +41,35 @@ extern bool pgtable_l5_enabled; #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE - 1)) +#ifdef CONFIG_RISCV_USE_SW_PAGE + +/* Page 4th Directory entry */ +typedef struct page_table_entry p4d_t; + +#define p4d_val(x) ((x).p4ds[0]) +p4d_t __p4d(unsigned long p4dval); +#define __p4d __p4d +#define PTRS_PER_P4D (PAGE_SIZE / sizeof(p4d_t)) + +/* Page Upper Directory entry */ +typedef struct page_table_entry pud_t; + +#define pud_val(x) ((x).puds[0]) +pud_t __pud(unsigned long pudval); +#define __pud __pud +#define PTRS_PER_PUD (PAGE_SIZE / sizeof(pud_t)) + +/* Page Middle Directory entry */ +typedef struct page_table_entry pmd_t; + +#define pmd_val(x) ((x).pmds[0]) +pmd_t __pmd(unsigned long pmdval); +#define __pmd __pmd + +#define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t)) + +#else /* CONFIG_RISCV_USE_SW_PAGE */ + /* Page 4th Directory entry */ typedef struct { unsigned long p4d; @@ -69,6 +98,8 @@ typedef struct { #define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t)) +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + /* * rv64 PTE format: * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 @@ -98,7 +129,7 @@ enum napot_cont_order { #define for_each_napot_order_rev(order) \ for (order = NAPOT_ORDER_MAX - 1; \ order >= NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order(val) (__builtin_ctzl((val.pte >> _PAGE_PFN_SHIFT) << 1)) +#define napot_cont_order(val) (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1)) #define napot_cont_shift(order) ((order) + PAGE_SHIFT) #define napot_cont_size(order) BIT(napot_cont_shift(order)) @@ -279,7 +310,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) if (pgtable_l4_enabled) WRITE_ONCE(*p4dp, p4d); else - set_pud((pud_t *)p4dp, (pud_t){ p4d_val(p4d) }); + set_pud((pud_t *)p4dp, __pud(p4d_val(p4d))); } static inline int p4d_none(p4d_t p4d) @@ -327,7 +358,7 @@ static inline pud_t *p4d_pgtable(p4d_t p4d) if (pgtable_l4_enabled) return (pud_t *)pfn_to_virt(__page_val_to_pfn(p4d_val(p4d))); - return (pud_t *)pud_pgtable((pud_t) { p4d_val(p4d) }); + return (pud_t *)pud_pgtable(__pud(p4d_val(p4d))); } #define p4d_page_vaddr(p4d) ((unsigned long)p4d_pgtable(p4d)) @@ -346,7 +377,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) if (pgtable_l5_enabled) WRITE_ONCE(*pgdp, pgd); else - set_p4d((p4d_t *)pgdp, (p4d_t){ pgd_val(pgd) }); + set_p4d((p4d_t *)pgdp, __p4d(pgd_val(pgd))); } static inline int pgd_none(pgd_t pgd) @@ -384,7 +415,7 @@ static inline p4d_t *pgd_pgtable(pgd_t pgd) if (pgtable_l5_enabled) return (p4d_t *)pfn_to_virt(__page_val_to_pfn(pgd_val(pgd))); - return (p4d_t *)p4d_pgtable((p4d_t) { pgd_val(pgd) }); + return (p4d_t *)p4d_pgtable(__p4d(pgd_val(pgd))); } #define pgd_page_vaddr(pgd) ((unsigned long)pgd_pgtable(pgd)) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9d3947ec3523..f9aed43809b3 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -574,6 +574,25 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) +{ + unsigned int i; + + if (pte_present(pte) && !pte_napot(pte)) + for (i = 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] += nr << _PAGE_PFN_SHIFT; + + return pte; +} +#else +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) +{ + return __pte(pte_val(pte) + (nr << _PAGE_PFN_SHIFT)); +} +#endif +#define pte_advance_pfn pte_advance_pfn + static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) { @@ -584,7 +603,7 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; + pteval = pte_advance_pfn(pteval, 1); } } #define set_ptes set_ptes @@ -882,7 +901,7 @@ extern pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, ((offset) << __SWP_OFFSET_SHIFT) }) #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) }) -#define __swp_entry_to_pte(x) ((pte_t) { (x).val }) +#define __swp_entry_to_pte(x) (__pte((x).val)) static inline int pte_swp_exclusive(pte_t pte) { diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 4ae67324f992..0c6b2fc6be58 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,6 +5,52 @@ #include #include +#ifdef CONFIG_RISCV_USE_SW_PAGE + +pte_t __pte(unsigned long pteval) +{ + pte_t pte; + + return pte; +} +EXPORT_SYMBOL(__pte); + +pgd_t __pgd(unsigned long pgdval) +{ + pgd_t pgd; + + return pgd; +} +EXPORT_SYMBOL(__pgd); + +#ifdef CONFIG_64BIT +p4d_t __p4d(unsigned long p4dval) +{ + p4d_t p4d; + + return p4d; +} +EXPORT_SYMBOL(__p4d); + +pud_t __pud(unsigned long pudval) +{ + pud_t pud; + + return pud; +} +EXPORT_SYMBOL(__pud); + +pmd_t __pmd(unsigned long pmdval) +{ + pmd_t pmd; + + return pmd; +} +EXPORT_SYMBOL(__pmd); +#endif /* CONFIG_64BIT */ + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) From patchwork Thu Dec 5 10:37:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894978 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C9B0E77173 for ; Thu, 5 Dec 2024 10:44:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SeYPshsaITiobVFpceZH3OQo/gDnUSXAtOpMWUnMxI8=; b=1xNHV3LK1m8cNl rwDK2tdUUKTS2surieb358F7fHWk1w6S2NmFjEVWHh8uNls98veLBdgdyuZ91Go0XRqgLp+cqZcwh Zdp7y+K6q3vLAj9IIo+dl8jZT9epTDu6MZJqzO9vDlTDDA8w+CIIEnp2c3tfXkd6IUrtdPPJjh1Un kf0s05btFG+4iEYQxDDUnFnCgYRerMoNlxk1pA4CKwqn27ikymzRIWYh5juZDESZBteWS0S58xA0W vvLezidNEXxbHEptR9UDVoYlIIpP9GLm6BrmpqvKsTM9Uh/SXtz+3YTGpFMvdKmYt4hT5UHabWgMJ Ai/2+VMrqhBE5PVCCykw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ln-0000000FaOp-3sUA; Thu, 05 Dec 2024 10:44:51 +0000 Received: from mail-pg1-x52b.google.com ([2607:f8b0:4864:20::52b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9F6-0000000FY43-0J1W for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:37:57 +0000 Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-7fbbe0fb0b8so533702a12.0 for ; Thu, 05 Dec 2024 02:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395075; x=1733999875; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cc/dZWEu2lOmGL5teCDDaBtpVgBp3/NGadgIOM//MYE=; b=MsJ5Tb0el1cllLtBi1j9fZkXnH78+PK14xOJXUOZtjTOI8PBdzpeHVCqoniLqG11TQ n/oBCKp7Rp+rQ/8tIkM9LC4tUDZOavrO5p3uPTZr+D9fArCpHaX99fCp3csd6/teXamz HDxJkB2kHEbil7McZATvB7oQwXqVx2c59zsjKdv7v2HoV4HpxJ61Ye7/Gzx9m80UdWMV xc8F4IOU/ygcx5vkcGCaJL0Ysury7Ln7MG9aWXKLIYuqV6VklrWWhtEE+hPC3i1nOII+ m7pd124yCMHR++AAbYu7kJZkLR6dcd8lAqGhDqRdPnJZQvywXmki9BUarQUkFsqUmHOe YM7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395075; x=1733999875; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cc/dZWEu2lOmGL5teCDDaBtpVgBp3/NGadgIOM//MYE=; b=T8d6c+XGhVL6QNjkUuzF+g/pQxeaa62FC7SIprwjxTQnXOLn1qTGHFZiV4+oCJcoXU FlpSWsau+29WZXsmkCE7natDz+ZsIrEuCO2nQLDs8PM/MVFh+ee0Py7yOq6sWfTISkKu 4MHa5iE7OPX/fmBIvOXqS6TouJwP3+jSxAVxVAEpZnSYNYkLOS4wgHyxWPttsfCz++gv QNK2RGFboQfZ3vRfJwbM9zEppHRXWJH/Ze2xTq7p05Zu1DVk65jQZusaJzKm6kpQ5fIw skVESJwiycj1Ah9oR/M/9dVWWb0k3e5kbRhHY4VyjFs+5jbK4edkZWwIq255AmiIeHxt eACg== X-Forwarded-Encrypted: i=1; AJvYcCVWUOSeh18m6dmbYNK+bSH+Q6lo2mbdNwR+u1KACLk4oAOVkk3lA3HCiHhQJfZkXFHCndex3oQKqLHtvQ==@lists.infradead.org X-Gm-Message-State: AOJu0YyywLaLxmqPszX9IByir8uV8D06d//T3CsrIuAINyCMj8kn9wKb vmZkQIXA0HDwLDy+IMp0POVxyFvQUbeDz4Qd07XAzY+rkklpiVhXUh5k2fBO/6I= X-Gm-Gg: ASbGncvR80rZjSu5g5+SfPmmIInDWKmvXXpKdY91c//b9wf+DCQBkYOXIXL4Va7sOks Mirc5efFHlMVTAAaTmBPsJdTS4JKQVUWFZYy/ZJIXtIortyV036s+Pbeb9jwXFFFH/+VmYJrqAC U5gIA0NP1IbwOEsDtUisCf4TxjVaFWyS6TWELNhydI2Xr55O32TW+M0xhtUpfDaNciKKB2RpOXd QWqbSj73F4stPr/MzhyAmnJbeZifyQoaQbBZ6rJ02OBAKzgKHnHO4cKtiwyOfcihLnxoKPc1wK7 AxvPGRzhh6Lnj0fGWtGUQKdy5zTvxpDz X-Google-Smtp-Source: AGHT+IEA0IERrJMVRep0HIEJfScJ16JhQtj/WdB3oMlrbTyvrSoqe//NvEXdVU3Seu11zobAb4MPFg== X-Received: by 2002:a05:6a21:3989:b0:1db:e3f6:55f with SMTP id adf61e73a8af0-1e1653b992amr15407579637.18.1733395075109; Thu, 05 Dec 2024 02:37:55 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:54 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 04/21] riscv: mm: Reimplement page table entry constructor function Date: Thu, 5 Dec 2024 18:37:12 +0800 Message-Id: <20241205103729.14798-5-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023756_110360_A4B2E99F X-CRM114-Status: GOOD ( 14.36 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This commit reimplements the page table entry constructor. As each page can contains several hardware pages now, the pte constructor need to initialize all mapping entries of these hardware pages. Note that the step path between mapping entries differs in different page table entry levels. For example, in PTE level, the step path between hardware mapping entries is hardware page size (aka 4K). In PMD level, the step path is (2 ^ 9) * hardware page size (aka 2M), etc. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-32.h | 5 +++ arch/riscv/include/asm/pgtable-64.h | 41 +++++++++++++++++++--- arch/riscv/include/asm/pgtable.h | 54 ++++++++++++++++++++++++----- arch/riscv/mm/pgtable.c | 47 +++++++++++++++++++++++++ 4 files changed, 133 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/pgtable-32.h index 159a668c3dd8..2959ab72f926 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -37,4 +37,9 @@ static const __maybe_unused int pgtable_l4_enabled; static const __maybe_unused int pgtable_l5_enabled; +static inline int __pgd_present(unsigned long pgdval) +{ + return pgdval & _PAGE_PRESENT; +} + #endif /* _ASM_RISCV_PGTABLE_32_H */ diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index e736873d7768..efcf63667f93 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -204,9 +204,14 @@ static inline u64 riscv_page_io(void) _PAGE_USER | _PAGE_GLOBAL | \ _PAGE_MTMASK)) +static inline int __pud_present(unsigned long pudval) +{ + return pudval & _PAGE_PRESENT; +} + static inline int pud_present(pud_t pud) { - return (pud_val(pud) & _PAGE_PRESENT); + return __pud_present(pud_val(pud)); } static inline int pud_none(pud_t pud) @@ -219,11 +224,16 @@ static inline int pud_bad(pud_t pud) return !pud_present(pud); } -#define pud_leaf pud_leaf +static inline bool __pud_leaf(unsigned long pudval) +{ + return __pud_present(pudval) && (pudval & _PAGE_LEAF); +} + static inline bool pud_leaf(pud_t pud) { - return pud_present(pud) && (pud_val(pud) & _PAGE_LEAF); + return __pud_leaf(pud_val(pud)); } +#define pud_leaf pud_leaf static inline int pud_user(pud_t pud) { @@ -321,14 +331,30 @@ static inline int p4d_none(p4d_t p4d) return 0; } +static inline int __p4d_present(unsigned long p4dval) +{ + return p4dval & _PAGE_PRESENT; +} + static inline int p4d_present(p4d_t p4d) { if (pgtable_l4_enabled) - return (p4d_val(p4d) & _PAGE_PRESENT); + return __p4d_present(p4d_val(p4d)); return 1; } +static inline int __p4d_leaf(unsigned long p4dval) +{ + return 0; +} + +static inline int p4d_leaf(p4d_t p4d) +{ + return __p4d_leaf(p4d_val(p4d)); +} +#define p4d_leaf p4d_leaf + static inline int p4d_bad(p4d_t p4d) { if (pgtable_l4_enabled) @@ -388,10 +414,15 @@ static inline int pgd_none(pgd_t pgd) return 0; } +static inline int __pgd_present(unsigned long pgdval) +{ + return pgdval & _PAGE_PRESENT; +} + static inline int pgd_present(pgd_t pgd) { if (pgtable_l5_enabled) - return (pgd_val(pgd) & _PAGE_PRESENT); + return __pgd_present(pgd_val(pgd)); return 1; } diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index f9aed43809b3..1d5f533edbd5 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -220,8 +220,19 @@ static inline unsigned long satp_pfn(unsigned long satp) return hwpfn_to_pfn(hwpfn); } +static inline int __pgd_leaf(unsigned long pgdval) +{ + return __pgd_present(pgdval) && (pgdval & _PAGE_LEAF); +} + +static inline int pgd_leaf(pgd_t pgd) +{ + return __pgd_leaf(pgd_val(pgd)); +} +#define pgd_leaf pgd_leaf + #ifdef CONFIG_TRANSPARENT_HUGEPAGE -static inline int pmd_present(pmd_t pmd) +static inline int __pmd_present(unsigned long pmdval) { /* * Checking for _PAGE_LEAF is needed too because: @@ -229,15 +240,20 @@ static inline int pmd_present(pmd_t pmd) * the present bit, in this situation, pmd_present() and * pmd_trans_huge() still needs to return true. */ - return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE | _PAGE_LEAF)); + return (pmdval & (_PAGE_PRESENT | _PAGE_PROT_NONE | _PAGE_LEAF)); } #else -static inline int pmd_present(pmd_t pmd) +static inline int __pmd_present(unsigned long pmdval) { - return (pmd_val(pmd) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); + return (pmdval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); } #endif +static inline int pmd_present(pmd_t pmd) +{ + return __pmd_present(pmd_val(pmd)); +} + static inline int pmd_none(pmd_t pmd) { return (pmd_val(pmd) == 0); @@ -248,11 +264,16 @@ static inline int pmd_bad(pmd_t pmd) return !pmd_present(pmd) || (pmd_val(pmd) & _PAGE_LEAF); } -#define pmd_leaf pmd_leaf +static inline bool __pmd_leaf(unsigned long pmdval) +{ + return __pmd_present(pmdval) && (pmdval & _PAGE_LEAF); +} + static inline bool pmd_leaf(pmd_t pmd) { - return pmd_present(pmd) && (pmd_val(pmd) & _PAGE_LEAF); + return __pmd_leaf(pmd_val(pmd)); } +#define pmd_leaf pmd_leaf static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { @@ -306,9 +327,14 @@ static __always_inline bool has_svnapot(void) return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT); } +static inline unsigned long __pte_napot(unsigned long val) +{ + return val & _PAGE_NAPOT; +} + static inline unsigned long pte_napot(pte_t pte) { - return pte_val(pte) & _PAGE_NAPOT; + return __pte_napot(pte_val(pte)); } static inline pte_t pte_mknapot(pte_t pte, unsigned int order) @@ -324,11 +350,16 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned int order) static __always_inline bool has_svnapot(void) { return false; } -static inline unsigned long pte_napot(pte_t pte) +static inline unsigned long __pte_napot(unsigned long pteval) { return 0; } +static inline unsigned long pte_napot(pte_t pte) +{ + return __pte_napot(pte_val(pte)); +} + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */ @@ -356,9 +387,14 @@ static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot) #define mk_pte(page, prot) pfn_pte(page_to_pfn(page), prot) +static inline int __pte_present(unsigned long pteval) +{ + return (pteval & (_PAGE_PRESENT | _PAGE_PROT_NONE)); +} + static inline int pte_present(pte_t pte) { - return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); + return __pte_present(pte_val(pte)); } #define pte_accessible pte_accessible diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 0c6b2fc6be58..f57ada26a183 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -10,6 +10,13 @@ pte_t __pte(unsigned long pteval) { pte_t pte; + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + pte.ptes[i] = pteval; + if (__pte_present(pteval) && !__pte_napot(pteval)) + pteval += 1 << _PAGE_HWPFN_SHIFT; + } return pte; } @@ -18,6 +25,16 @@ EXPORT_SYMBOL(__pte); pgd_t __pgd(unsigned long pgdval) { pgd_t pgd; + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + pgd.pgds[i] = pgdval; + if (__pgd_leaf(pgdval)) + pgdval += (1 << (PGDIR_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pgd_present(pgdval)) + pgdval += 1 << _PAGE_HWPFN_SHIFT; + } return pgd; } @@ -27,6 +44,16 @@ EXPORT_SYMBOL(__pgd); p4d_t __p4d(unsigned long p4dval) { p4d_t p4d; + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + p4d.p4ds[i] = p4dval; + if (__p4d_leaf(p4dval)) + p4dval += (1 << (P4D_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__p4d_present(p4dval)) + p4dval += 1 << _PAGE_HWPFN_SHIFT; + } return p4d; } @@ -35,6 +62,16 @@ EXPORT_SYMBOL(__p4d); pud_t __pud(unsigned long pudval) { pud_t pud; + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + pud.puds[i] = pudval; + if (__pud_leaf(pudval)) + pudval += (1 << (PUD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pud_present(pudval)) + pudval += 1 << _PAGE_HWPFN_SHIFT; + } return pud; } @@ -43,6 +80,16 @@ EXPORT_SYMBOL(__pud); pmd_t __pmd(unsigned long pmdval) { pmd_t pmd; + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + pmd.pmds[i] = pmdval; + if (__pmd_leaf(pmdval)) + pmdval += (1 << (PMD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pmd_present(pmdval)) + pmdval += 1 << _PAGE_HWPFN_SHIFT; + } return pmd; } From patchwork Thu Dec 5 10:37:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DED7EE77178 for ; Thu, 5 Dec 2024 10:44:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YVRWtYPn+O5EuIIzyp/dQBF7T+nLs6Kcsi3VFQ/9J7E=; b=WcsW2v7F62pKT7 XEru7AkLaAnvwNJI4n7MDTKMlAqzCeFG4Hmd51X7QSfeYnseP/bjAKrxmX6/GnCoXoJ7LgyKyzkKJ Yz2ZqIrYbPqMe0uj5aXzsNtz/fjcJ8rq22iySi5lJIS5WDO55C/UIPaG/psEPgResl17YQq3Zejct VD6MqraoSfE7rmek7Kd2fzBoBya3DcrcDk3FkZWp/+g60+8Wf3LuaG0TtaI8N15Amw0sy1Icjw0yW NMTNWaGB8YZZBs1ObSwHOnvSQN4JrbURyAUEPK0H/n7APeUTqLjo6snAhsm6U/lORVnwDsXY+42K/ UVmnr2yMcmRO1CMspDbw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lo-0000000FaPX-2qlS; Thu, 05 Dec 2024 10:44:52 +0000 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FA-0000000FY5v-0K1J for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:01 +0000 Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-7fc41b4c78bso490104a12.3 for ; Thu, 05 Dec 2024 02:37:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395079; x=1733999879; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7qR1jHqy2cDl06AgpV8oSKqeRVPpykz0hiLJxVMX/VM=; b=X6pApK3iCm1bCMhRnGzLX7zyFWoOUrBIqsR2A4wjWDPZ4AGeUK9h8pWs1WIWRe2PSi ugoO+QLn0w8y61L16aHPdOwY/ks3h6kkRCuCEieduZXysQf8LpmjWnbG3LaiM9hhhbpI hmp8J+aEwJ7M6WC4hQGlIbTTLZteuqzhNOPIj4gRzAWQZjM2lkRrU9tcsKjg182hjF5j z3O6rDKz2vQ76p7EiF+158x6tAEIBVdqTfy4AviVpNzqW+Ger5dhiKiaZ0UVxC6K4cdj 9/TVKJ8+fl0OFYv7w35NfmQm/VoSJI9wbPlRI9c/A4ShmbX13nb8/Bd+ubC3fv8K2QIw esgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395079; x=1733999879; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7qR1jHqy2cDl06AgpV8oSKqeRVPpykz0hiLJxVMX/VM=; b=QxG++onNug8IQCyO6DPECQvZt1m0z8YiQE+BU8wROTQjQoYdKhSt/kbn4nco9NgA6p kXj0346S0KSp1StEMWoPg0j2+n/mq7Rziip/bUVdo5CDNtgEGYYOnJluHjPO2DAIGwoB Vj0oFQBgKMGsZrr/yrX7QGq09qsVl9o2hTqUYBetbJhGgT7MF4Grnv/lEOj4bmPJB/9n jKYfEv2h1NQ61MbCxcvRCsE25GTPnztsRRzax8EDDdYLO417yhzuLStUSifTo9Pq9BNm /fOoklqpDNsO9Q77b4TqKsaOtKcyOT5Y1gbP87x+ozTFihXGPLZ0LLO8DK+K+L8wP6iB n0rw== X-Forwarded-Encrypted: i=1; AJvYcCXR5hKhr8klwJ3NqZ15rrjIKEjdbek06McclJcQClOclA0g9p5Sxrp6hl94kaMRg/0jDFtyfU/QvtvBDQ==@lists.infradead.org X-Gm-Message-State: AOJu0YyBAHFi6/Jg0DlfeddOnodNkQOjHS5/N+qE7XZ9t7zhFerXzAPj tXDP6uNeEBtB9xuwXdVkzUvlZMvs462+QQstCpam6QhimpAvPOOGVyFxZeeD4c265L16itx1zwY ZW8c= X-Gm-Gg: ASbGncvqggGbtkeKqt47b5dmZUOcjHOc/S+1Opwej32kxV/WVhhCTMV8OQ6IWg0Inrf Z8baoyiR7ANbHO6bkbNJBuhNnP/vuK5v3yf1waubVckl47EBXvsRtUzVbft+NoV7Jbs0DFvSFCf S9zm+JWunZNceuhLT08cZEWWLinz1y0uu/5666uitEVZQfBkyMr9KxbzkgOhPYGca5d2TLtrXZ3 73gcafKkK3D+lwNLdFmv1q+/i5qtrh/3miTMYskVSt/iV0wYGkVLjgSMBD/CwKMQaJrV6DWssla 47JbGuJmkx5dZx/GAjDo6dwm3qbocEvI X-Google-Smtp-Source: AGHT+IEUKSwiJ64FZ3Pe2Otsaz2xFzyoQCTrZnVdkwgamPJ8aR60BE+SnfSU6DdJI6PGfNndTdnNkA== X-Received: by 2002:a05:6a21:99a2:b0:1db:e358:48a8 with SMTP id adf61e73a8af0-1e1653b91d0mr15785669637.10.1733395079129; Thu, 05 Dec 2024 02:37:59 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:37:58 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 05/21] riscv: mm: Reimplement conversion functions between page table entry Date: Thu, 5 Dec 2024 18:37:13 +0800 Message-Id: <20241205103729.14798-6-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023800_112200_054076D7 X-CRM114-Status: UNSURE ( 7.70 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Some code will convert high level pte into low level pte to reuse pte functions. For example, pmd_dirty() will convert pmd struct into pte struct to check whether it is dirty using pte_dirty(). As pte struct at different level has different constructor now, we can not apply pte constructor during conversion. Thus, this commit converts ptes by directly converting structure type. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 81 ++++++++++++++++++++++++++++++-- 1 file changed, 76 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 1d5f533edbd5..f7b51c52b815 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -309,6 +309,50 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) return (unsigned long)pfn_to_virt(__page_val_to_pfn(pmd_val(pmd))); } +#ifdef CONFIG_RISCV_USE_SW_PAGE + +static inline pte_t pmd_pte(pmd_t pmd) +{ + return (pte_t)pmd; +} + +static inline pte_t pud_pte(pud_t pud) +{ + return (pte_t)pud; +} + +static inline pte_t p4d_pte(p4d_t p4d) +{ + return (pte_t)p4d; +} + +static inline pte_t pgd_pte(pgd_t pgd) +{ + return (pte_t)pgd; +} + +static inline pmd_t pte_pmd(pte_t pte) +{ + return (pmd_t)pte; +} + +static inline pud_t pte_pud(pte_t pte) +{ + return (pud_t)pte; +} + +static inline p4d_t pte_p4d(pte_t pte) +{ + return (p4d_t)pte; +} + +static inline pgd_t pte_pgd(pte_t pte) +{ + return (pgd_t)pte; +} + +#else /* CONFIG_RISCV_USE_SW_PAGE */ + static inline pte_t pmd_pte(pmd_t pmd) { return __pte(pmd_val(pmd)); @@ -319,6 +363,38 @@ static inline pte_t pud_pte(pud_t pud) return __pte(pud_val(pud)); } +static inline pte_t p4d_pte(p4d_t p4d) +{ + return __pte(p4d_val(p4d)); +} + +static inline pte_t pgd_pte(pgd_t pgd) +{ + return __pte(pgd_val(pgd)); +} + +static inline pmd_t pte_pmd(pte_t pte) +{ + return __pmd(pte_val(pte)); +} + +static inline pud_t pte_pud(pte_t pte) +{ + return __pud(pte_val(pte)); +} + +static inline p4d_t pte_p4d(pte_t pte) +{ + return __p4d(pte_val(pte)); +} + +static inline pgd_t pte_pgd(pte_t pte) +{ + return __pgd(pte_val(pte)); +} + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #ifdef CONFIG_RISCV_ISA_SVNAPOT #include @@ -728,11 +804,6 @@ static inline pgprot_t pgprot_writecombine(pgprot_t _prot) /* * THP functions */ -static inline pmd_t pte_pmd(pte_t pte) -{ - return __pmd(pte_val(pte)); -} - static inline pmd_t pmd_mkhuge(pmd_t pmd) { return pmd; From patchwork Thu Dec 5 10:37:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894980 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C7855E77177 for ; Thu, 5 Dec 2024 10:44:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1ptaG+VnMnjTQeeyso1HzXfmIIfAzCjB888kqgUXwnw=; b=T2ISPciSAof3AM uMY7I/+zTniLtmzl/Wd7xlvzrOMbe0U8bi7xeBvY9zPK2XEaf5WHvL8yi4HvEVI8PLatm1JSJKSxR i9oFcgwrd88SsxNkaBSQp7HdIX16tjchz981pHC4uzpx3FFu7QKbbd1jQPNGV7YhL/6cCfKdZtlB9 zkPGNWRuZ20637Hy5w2YymKD/8COxN1Lf/K758YZVRvLkMTFLieP4VERNOyWNYw87Cuuaszgl++m2 a8rhjDYVTAwz0hI5lVvHFN5Gg9KXQ2uGY7xOQ2EhrVJN4jO4kzv18sfOeI0XcO6Tk0zS2ssGv1b52 zsocioIeR/MeixSWfLzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lp-0000000FaQ2-1gd9; Thu, 05 Dec 2024 10:44:53 +0000 Received: from mail-ot1-x334.google.com ([2607:f8b0:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FE-0000000FY6u-2Oa7 for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:05 +0000 Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-71d46ace74aso371755a34.3 for ; Thu, 05 Dec 2024 02:38:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395083; x=1733999883; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XbXb8+gIbpPc0seecsAjhzRbpNpiwhPRJnpv+Yg3m9U=; b=ZaDCvF2/vyrKnkBLdXgUd6iarWLH6eJYKG7yrjddEjYv3WxcMW7+15gNKRcsWogUBf +I8X+RMgT+kHpG2qOHfltNIBCnwgf+tcitr2Asa2eIHYYc0d2TO+mkxsx8ZMVKUOkQ+U bmO5rNwHb3J+7LLPOVTYfB4BNhVVYdKB6aSSayPsDkjpBEDugVKOmVj2+2fXLdrpgZ7N lsOhc+4Sy2xO/d5g94vtSi60qfvKDfSy6ekKIH84tb0JdzwQFb54JzTFLahTAvZ6s7EW PRUkr225XdBMFapI0Mgk9gkJ69kHumLqbG7W0po/xdE1kCSE77l+QjPD9xhCn3ycirHL Ozxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395083; x=1733999883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XbXb8+gIbpPc0seecsAjhzRbpNpiwhPRJnpv+Yg3m9U=; b=GzUZ7BVVLIFgB8FOybcwTLdnbeOgWvlERce0MMDqMUijdnitCB7+dkx3EVU68LKDky +qdBomRiDdkxCpQtQeKBJSTEUXPvat6/1gbWLH9mBvViIKj4eyygiwcjmg5644mm+foB rnWC01wedWSbfPzcsfYg7OkQy10e37Vt27sG579PDcZ0M03rqMA0CXxnnyGsb4Ll5WPY orETLs14jlmD9TQ9S7Oe9XUpRZ++3PUEREMV35Pjb839la953ICiNs/i3LeXg8GJw2BB uxZRBSMnsbXAhQ9SZP7ouMKlMcDrezGN2ABsTa0bMY+eaEPo0iGeyHNFLLh2vxQxErBL Eing== X-Forwarded-Encrypted: i=1; AJvYcCX9gxm7xvPrl6SPo56dbwS6YDILiu9hEbV00CoyZisfeZMKaZnKgIPe78k5TQYGINfZHZcKHiGhu7qy6g==@lists.infradead.org X-Gm-Message-State: AOJu0YwXm3KVyS8C8FF2oX+TuKtzV4h38VB03Eu3XM5418KvcIQeLo+I qwe7hYdQE/qdzcN/X+EqLvEg9hVsmDc9sOBftn7TRVN3rm6UTZiHb+a4p9G8WsE= X-Gm-Gg: ASbGnctw8G0njrn1pVai3hsOv8lf46tE8Crd2A/jmXb1/N0PK9HPcB/f1JUelQNQSFO TAQjv2NPXP92IO7DeSteYHtd6uual1PanQgLzAuxTVP+mD8ZJO/O1fvDX2I35Ja+v6trisvPyJp 2MZoZt1A1Om7oSC9EgcP3YNm+iBQAVsqs78LddesTi0BYkscTfjhAu0Xa2oM+F+XlevyN0qUB0f QhRO2u8ukftW04pYs+5b+dpvRWgPw2u0CmIzEbQY7+Nhuwz8rSvdDAy4RJyoLkhLrvxmwfaygtV qhJ+vFTeO1y+VNsGlZYll4NX+r3E9FEp X-Google-Smtp-Source: AGHT+IEx3vS74gMj6raF1dIqh7j7ldd5iu4wb4DU/N4Y4QqLjAI1/NclGnjqcHYWmLe9eo91DygJAQ== X-Received: by 2002:a05:6830:d1a:b0:71a:6c7a:e23b with SMTP id 46e09a7af769-71dad652d39mr12397021a34.16.1733395083206; Thu, 05 Dec 2024 02:38:03 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.37.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:02 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 06/21] riscv: mm: Avoid pte constructor during pte conversion Date: Thu, 5 Dec 2024 18:37:14 +0800 Message-Id: <20241205103729.14798-7-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023804_606582_44407173 X-CRM114-Status: UNSURE ( 9.65 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This commit converts ptes at different level via directly converting pte type instead of using pte constructor, as ptes from different levels has different constructors. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 52 +++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index f7b51c52b815..d3da8aee213c 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -351,6 +351,26 @@ static inline pgd_t pte_pgd(pte_t pte) return (pgd_t)pte; } +static inline pte_t pte_set_flag(pte_t pte, unsigned long flag) +{ + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] |= flag; + + return pte; +} + +static inline pte_t pte_clear_flag(pte_t pte, unsigned long flag) +{ + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] &= (~flag); + + return pte; +} + #else /* CONFIG_RISCV_USE_SW_PAGE */ static inline pte_t pmd_pte(pmd_t pmd) @@ -393,6 +413,16 @@ static inline pgd_t pte_pgd(pte_t pte) return __pgd(pte_val(pte)); } +static inline pte_t pte_set_flag(pte_t pte, unsigned long flag) +{ + return __pte(pte_val(pte) | flag); +} + +static inline pte_t pte_clear_flag(pte_t pte, unsigned long flag) +{ + return __pte(pte_val(pte) & (~flag)); +} + #endif /* CONFIG_RISCV_USE_SW_PAGE */ #ifdef CONFIG_RISCV_ISA_SVNAPOT @@ -537,46 +567,46 @@ static inline int pte_devmap(pte_t pte) static inline pte_t pte_wrprotect(pte_t pte) { - return __pte(pte_val(pte) & ~(_PAGE_WRITE)); + return pte_clear_flag(pte, _PAGE_WRITE); } /* static inline pte_t pte_mkread(pte_t pte) */ static inline pte_t pte_mkwrite_novma(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_WRITE); + return pte_set_flag(pte, _PAGE_WRITE); } /* static inline pte_t pte_mkexec(pte_t pte) */ static inline pte_t pte_mkdirty(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_DIRTY); + return pte_set_flag(pte, _PAGE_DIRTY); } static inline pte_t pte_mkclean(pte_t pte) { - return __pte(pte_val(pte) & ~(_PAGE_DIRTY)); + return pte_clear_flag(pte, _PAGE_DIRTY); } static inline pte_t pte_mkyoung(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_ACCESSED); + return pte_set_flag(pte, _PAGE_ACCESSED); } static inline pte_t pte_mkold(pte_t pte) { - return __pte(pte_val(pte) & ~(_PAGE_ACCESSED)); + return pte_clear_flag(pte, _PAGE_ACCESSED); } static inline pte_t pte_mkspecial(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_SPECIAL); + return pte_set_flag(pte, _PAGE_SPECIAL); } static inline pte_t pte_mkdevmap(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_DEVMAP); + return pte_set_flag(pte, _PAGE_DEVMAP); } static inline pte_t pte_mkhuge(pte_t pte) @@ -612,7 +642,7 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) ALT_THEAD_PMA(newprot_val); - return __pte((pte_val(pte) & _PAGE_CHG_MASK) | newprot_val); + return pte_set_flag(pte_clear_flag(pte, ~_PAGE_CHG_MASK), newprot_val); } #define pgd_ERROR(e) \ @@ -1017,12 +1047,12 @@ static inline int pte_swp_exclusive(pte_t pte) static inline pte_t pte_swp_mkexclusive(pte_t pte) { - return __pte(pte_val(pte) | _PAGE_SWP_EXCLUSIVE); + return pte_set_flag(pte, _PAGE_SWP_EXCLUSIVE); } static inline pte_t pte_swp_clear_exclusive(pte_t pte) { - return __pte(pte_val(pte) & ~_PAGE_SWP_EXCLUSIVE); + return pte_clear_flag(pte, _PAGE_SWP_EXCLUSIVE); } #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION From patchwork Thu Dec 5 10:37:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1EB8FE77173 for ; Thu, 5 Dec 2024 10:45:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wEBTppFczb4UjCaFNYwoejfKFUZZmnewfTtY6Bi1vgI=; b=FB23QjpaePU96n J2rwUD4xAALe0k3+xkHw/7QGgY+iMn8wVakYFQ2u3VCM1Jq/NNw5adgqoMW2vKqq368WBgEHx8yx6 cT4ZhKW6VgWtBLcTvNcBreLIECHM8BLd4/G61RITvitCbvEYCivCfBddqwde9KF7MBckBEuCAeHA7 NovpIEPTwg/fckWJNoVCW2unTvo5x05NCS9dJGckW2kCxaDoaugfmiqEWyhfL3ExtZ/Ij8OlX8Uzh 6pt12cSNF+YwdK/zzF1ZH2qmodLcQ8arvArtiu73WdXJHx0MNB6ECax+7+Q6XScvmkyLQ9ROI9+Ph Ct9Wikncr2Mk+TRLJ91Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lq-0000000FaQu-1fGT; Thu, 05 Dec 2024 10:44:54 +0000 Received: from mail-ot1-x32d.google.com ([2607:f8b0:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FI-0000000FY8Q-1NI6 for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:09 +0000 Received: by mail-ot1-x32d.google.com with SMTP id 46e09a7af769-71d41932d32so398834a34.0 for ; Thu, 05 Dec 2024 02:38:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395087; x=1733999887; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cw560nhmcCgMtoVMwQuAveKPR5/TXRAfvCK89A67aqo=; b=Ide1+Jy8jWNoOb0gJpo54CFdQKtXkvyVYyqc3i6vqZi82ZB9DtlIzzrXU0F85xcYcu m1AcLd2aBDmZTY0wNvj3qNH0Onw38VfbxCgavyZ8AQLEHPUGhI6gmZBxP6a1mgQU9DTn WYFv8zWOjowPbI2RRhgVXIjo7ShqJl45GaM9tmsj1R5qQ9GigWA/uL0Hk2IarNsz5yCI jFJPRt3n0Ctxl7r7k4cmGNLt6Xk3869Zh8fv3pSGcx2LhnVHOcls6cdfGGxBS/k051ll fdncAZNUchDj6GO3xsER7HnB9rCCGhNTZlL3ZlrbcNzOSRyzNIfnsxIlrmnfB9UjWRgW jA4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395087; x=1733999887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cw560nhmcCgMtoVMwQuAveKPR5/TXRAfvCK89A67aqo=; b=dzRefiDJqtH81JVj8pSwdLeeB5tNQvAtr0br3iFRJSg/HuAhN14jFVeBztSPLOKZNk VZoiMZEUPmyH9qKUpjk5arPVwySWl9wOOcG67qzB8xy9KWZSbxeylsUqA/zgkX/0xGyu YfwYMSi2gmTdirAo3XytWsimGrNzufNamxHBQMEEd5223U3+aJGG9/Zr8eQSDbcvtuiB ZWb6ZW4BurNQ1MAn2yMG04Dd3c5tgAXbbOAccHh1bseK4ra6Ee8bCVg3Ip01tPEbbJyP 9Hjq1ftXch4mS7n5cfvnEwXDZaGsIvap59o0Akw22DlN7kRTJOu4Ue39Mn7ytLrtYQiP q52A== X-Forwarded-Encrypted: i=1; AJvYcCW9PlXObrriFcDa/JicpjoVpnvy/cxHw7xBYKJchxy1UShzI2gD7FIs5ehX2h5v2pqGSzDgqoe9KJkCaw==@lists.infradead.org X-Gm-Message-State: AOJu0YyzQwgRdzvKryK0eaURhqY1lPfeS1NGUaUvOz4FzCRCHzyrOlk1 CVHvI2U5BIBpAcL9dJ/VTIjFj9touifWkNV0QN5aKhpQkkNh8xJR5Yx2hKMmy1Q= X-Gm-Gg: ASbGncvU+AXMajZcP1oleGdFVRpZHFteecdJ17a8zh5NbQFyqD7jG5xMcpww6aRLD4t 7fAQzzvHo0On/5pAasbkyEcdGFFEZT/QSxV/tR9CtPClwXNs+rqXSVbONDB5aWB3pCm+Jl+OW/x blRu/fJoUzWVFgUlGsuYw+hA39P0P88wWczUdgu/WXG96+SuSy4/P0X7Bre8h7mp3sckgrqBc/l CChNkUjM2qkb/CZvmh1ciAE5lNRiiFKnXFdLUtARttRzdsM6FQmqKcDWU2OazirpQtEqE+L+abO d1tIMveVJ2VfwsHgsonYgubnfv13Ao8m X-Google-Smtp-Source: AGHT+IFgLX/MNdvVLEUe2bcmja2NfdcbWMv5BT39DW17XBNTzPIKVvGRlTw6+0l7AjGg6yRiky8Jng== X-Received: by 2002:a05:6830:6016:b0:71d:54fb:da56 with SMTP id 46e09a7af769-71dad5f33bbmr12858019a34.2.1733395087266; Thu, 05 Dec 2024 02:38:07 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:07 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 07/21] riscv: mm: Reimplement page table entry get function Date: Thu, 5 Dec 2024 18:37:15 +0800 Message-Id: <20241205103729.14798-8-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023808_363922_B9214FB8 X-CRM114-Status: UNSURE ( 5.94 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This commit reimplements ptep_get/pmdp_get/... functions. As pte structures now contains multiple mapping entries, we can not use READ_ONCE to fetch its value. Instead, we use traditional dereference way. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d3da8aee213c..ba4a083b7210 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -704,6 +704,36 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) WRITE_ONCE(*ptep, pteval); } +static inline pte_t ptep_get(pte_t *ptep) +{ + return *ptep; +} +#define ptep_get ptep_get + +static inline pmd_t pmdp_get(pmd_t *pmdp) +{ + return *pmdp; +} +#define pmdp_get pmdp_get + +static inline pud_t pudp_get(pud_t *pudp) +{ + return *pudp; +} +#define pudp_get pudp_get + +static inline p4d_t p4dp_get(p4d_t *p4dp) +{ + return *p4dp; +} +#define p4dp_get p4dp_get + +static inline pgd_t pgdp_get(pgd_t *pgdp) +{ + return *pgdp; +} +#define pgdp_get pgdp_get + void flush_icache_pte(struct mm_struct *mm, pte_t pte); static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) From patchwork Thu Dec 5 10:37:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7EBC4E77176 for ; Thu, 5 Dec 2024 10:45:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mBJaklUnQYDg8CHFK41N+FdH/3af5awmPagoOyrbmaM=; b=Oy7NMOHF3NqDuu MKesXDzq5P0TQ6wJfnlxApy1H5sLzSjei669eZwN3rgQFUIWvMDw1YunMlsTNSicXKDDbjsd8eEAY KWwmjnMkFQ7elIHsD86LSs1XKRNWlQM/b1tt5wWLWy/vxVtrDVq9CYLKd8rcxW6TTgqqGNgzYCJLY euaclKo1H7TRc4n8OzIqot8vGBt1jrC0cbUayVLCuH98gVLHGEA/mN+kiLs6pDozbL/HbCyoSn+vF 9yYveaseZWsMAVMMzs+j4U6cmjvhZyMia5jc6RwVkt+pXmDBQVEHsNZJg37t2/1vkLaq0vC3zjqDC Ta/UwS70doxH6C1Ot/WA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lr-0000000FaRR-0woM; Thu, 05 Dec 2024 10:44:55 +0000 Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FM-0000000FY9t-0mSu for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:13 +0000 Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-7fcfb7db9bfso560804a12.1 for ; Thu, 05 Dec 2024 02:38:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395091; x=1733999891; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O7hmUlID2KY6paYgoqiid5lfsbGFxDDCAN4DBvbC41c=; b=Db69tBvLGPpWgeE+FsWEKDnqqcFC8qjmfJL7zzIPl9qh/AredMThlb5hKDK1suPJ/m 7hiE3+px+EzAUelV6vCKRrlHHcwLlQUA6P3CpJ+w6NTP+//VQJf+L7f8tjt0EVsnWzVo w0Ba8KB7wRRF75Iqry4oaL1QJpJaChaSzPMcRW6GAC31LPfESRbWuSWxv21n5f3kKc92 1rJeMjcCaCBN3DR31+si8AGpX7ak52N0LVeNxNVAGtIJQFPF4X9mlyTyoHSuIxtTIZvI NtTw+hbx13Rj5vXEZ3oz2Qhj18VUgHTJXsp2gMtHLtutdOFXsdDqgGT7JRiUCT/Zqh2U wKxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395091; x=1733999891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O7hmUlID2KY6paYgoqiid5lfsbGFxDDCAN4DBvbC41c=; b=PVsCGHTuhuQ4RW9Bj0jUauNuMEkKo4c9oPln2cq7NKjjqxN4dJ8EtsLs416TJQQQuW 0YL6NEZiO5UL6mFl+xR6mLBQpYW20iPc+YllrUUYPqtYanuyUlz86QwFAQirykJLNXZ6 /LofzAe6DsE0UrkwmJQiNj3AeLrUkykV05IgWYnq3k6UMv22LK5tjJo3sfPxCC+jAndv XYDlrjn5p8MpmVF6NxTwM69YlxqiWoe8zAOnyhXRuWTToEq9qOowoM/0633+HTwz9uUa Adhfl9xEIy6yG0l5hKVhliIMxkONxqYz6Cy5xHlOBhJrV2HlUbgCto5pUmdlpZPAp4vr 4tpQ== X-Forwarded-Encrypted: i=1; AJvYcCU1/FLTapKdG8kQGelJF8uHQnxJvOxROhKhOi2Pvy2c7yCz3TElt90tbIslC2mwYzUV7rwre5JcAaolRw==@lists.infradead.org X-Gm-Message-State: AOJu0YyfZVg202j2p0f3fPsvMb6v+ZuEeOkMIl16w9CVc0X9DAnDr/7E LyAJoNb/yDvztik1PLAH8lsQNTiwr9+lqM4HlYRSIDOESGjL+KVeB+nlFX1/gK8= X-Gm-Gg: ASbGncuueSyHvHIZql/Thi5iVHi7BfkuNIrRkzbpNz00PJ7lEnJJR5MbbcnJxIfdmdE oP4R3D7AhictJzW1QXWdFsZ8kpD3vPlGZfuMN9uCfwvvDR01fUvtsu1pFGiMv7v9RuTPkNgrYYS szXd5g84WDcBMTJN9XghNuuJUe//JMVW9EgbD0N5TAzBdL4YYp+zLAcCQS5ensuTffePyQ2IyPe yXiq2hBg4m5pF+nB+TOmLS7uMj4Engp8EiT+f0bUKrjSo3jxn5PyYfw5ku4V2vE5nKM4i5EIo6d VmU+zv5SUY86767VfRHk4nl0d+pyYSTn X-Google-Smtp-Source: AGHT+IHbEOK4fVgmqFCn4T84ALhR/xBKYInyH1RV8Ugvahqt2YHHrGpIWPwGC34ww4f3ko0WEWYg3w== X-Received: by 2002:a05:6a20:cfa4:b0:1e0:d104:4dbd with SMTP id adf61e73a8af0-1e16542f238mr16645992637.46.1733395091287; Thu, 05 Dec 2024 02:38:11 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:11 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 08/21] riscv: mm: Reimplement page table entry atomic get function Date: Thu, 5 Dec 2024 18:37:16 +0800 Message-Id: <20241205103729.14798-9-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023812_224404_C127579A X-CRM114-Status: GOOD ( 11.77 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This commit implements lockless functions to atomically fetch pte's value. For each pte structure, we atomically fetch the first mapping entry, and then fetch the following entries and compare them with the first mappin entry plus certain step path in a loop. If we find any difference in their pfns or prots, then the pte structure has been modified and need to be reloaded. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 156 +++++++++++++++++++++++++++++++ include/linux/pgtable.h | 21 +++++ 2 files changed, 177 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ba4a083b7210..fe42afb4441e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -220,6 +220,18 @@ static inline unsigned long satp_pfn(unsigned long satp) return hwpfn_to_pfn(hwpfn); } +static inline unsigned long __pte_pgprot(unsigned long pteval) +{ + unsigned long prot_mask = GENMASK(_PAGE_HWPFN_SHIFT - 1, 0); + + return pteval & prot_mask; +} + +static inline pgprot_t pte_pgprot(pte_t pte) +{ + return __pgprot(__pte_pgprot(pte_val(pte))); +} + static inline int __pgd_leaf(unsigned long pgdval) { return __pgd_present(pgdval) && (pgdval & _PAGE_LEAF); @@ -734,6 +746,150 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) } #define pgdp_get pgdp_get +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + unsigned long pteval; + pte_t pte; + int i; + +retry: + pteval = READ_ONCE(ptep->ptes[0]); + pte = *ptep; + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pteval) != + __page_val_to_pfn(pte.ptes[i])) + goto retry; + if ((__pte_pgprot(pteval) | _PAGE_DIRTY | _PAGE_ACCESSED) != + (__pte_pgprot(pte.ptes[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pte_present(pteval) && !__pte_napot(pteval)) + pteval += 1 << _PAGE_HWPFN_SHIFT; + } + + return pte; +} +#define ptep_get_lockless ptep_get_lockless + +static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) +{ + unsigned long pmdval; + pmd_t pmd; + int i; + +retry: + pmdval = READ_ONCE(pmdp->pmds[0]); + pmd = *pmdp; + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pmdval) != + __page_val_to_pfn(pmd.pmds[i])) + goto retry; + if ((__pte_pgprot(pmdval) | _PAGE_DIRTY | _PAGE_ACCESSED) != + (__pte_pgprot(pmd.pmds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pmd_leaf(pmdval)) + pmdval += (1 << (PMD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pmd_present(pmdval)) + pmdval += 1 << _PAGE_HWPFN_SHIFT; + } + + return pmd; +} +#define pmdp_get_lockless pmdp_get_lockless + +static inline void pmdp_get_lockless_sync(void) +{ +} + +static inline pud_t pudp_get_lockless(pud_t *pudp) +{ + unsigned long pudval; + pud_t pud; + int i; + +retry: + pudval = READ_ONCE(pudp->puds[0]); + pud = *pudp; + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pudval) != + __page_val_to_pfn(pud.puds[i])) + goto retry; + if ((__pte_pgprot(pudval) | _PAGE_DIRTY | _PAGE_ACCESSED) != + (__pte_pgprot(pud.puds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pud_leaf(pudval)) + pudval += (1 << (PUD_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pud_present(pudval)) + pudval += 1 << _PAGE_HWPFN_SHIFT; + } + + return pud; +} +#define pudp_get_lockless pudp_get_lockless + +static inline p4d_t p4dp_get_lockless(p4d_t *p4dp) +{ + unsigned long p4dval; + p4d_t p4d; + int i; + +retry: + p4dval = READ_ONCE(p4dp->p4ds[0]); + p4d = *p4dp; + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(p4dval) != + __page_val_to_pfn(p4d.p4ds[i])) + goto retry; + if ((__pte_pgprot(p4dval) | _PAGE_DIRTY | _PAGE_ACCESSED) != + (__pte_pgprot(p4d.p4ds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__p4d_leaf(p4dval)) + p4dval += (1 << (P4D_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__p4d_present(p4dval)) + p4dval += 1 << _PAGE_HWPFN_SHIFT; + } + + return p4d; +} +#define p4dp_get_lockless p4dp_get_lockless + +static inline pgd_t pgdp_get_lockless(pgd_t *pgdp) +{ + unsigned long pgdval; + pgd_t pgd; + int i; + +retry: + pgdval = READ_ONCE(pgdp->pgds[0]); + pgd = *pgdp; + for (i = 0; i < HW_PAGES_PER_PAGE; i++) { + if (__page_val_to_pfn(pgdval) != + __page_val_to_pfn(pgd.pgds[i])) + goto retry; + if ((__pte_pgprot(pgdval) | _PAGE_DIRTY | _PAGE_ACCESSED) != + (__pte_pgprot(pgd.pgds[i]) | _PAGE_DIRTY | _PAGE_ACCESSED)) + goto retry; + + if (__pgd_leaf(pgdval)) + pgdval += (1 << (PGDIR_SHIFT - PAGE_SHIFT)) << + _PAGE_HWPFN_SHIFT; + else if (__pgd_present(pgdval)) + pgdval += 1 << _PAGE_HWPFN_SHIFT; + } + + return pgd; +} +#define pgdp_get_lockless pgdp_get_lockless + +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + void flush_icache_pte(struct mm_struct *mm, pte_t pte); static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8b2ac6bd2ae..b629c48b980b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -598,6 +598,27 @@ static inline void pmdp_get_lockless_sync(void) } #endif +#ifndef pudp_get_lockless +static inline pud_t pudp_get_lockless(pud_t *pudp) +{ + return pudp_get(pudp); +} +#endif + +#ifndef p4dp_get_lockless +static inline p4d_t p4dp_get_lockless(p4d_t *p4dp) +{ + return p4dp_get(p4dp); +} +#endif + +#ifndef pgdp_get_lockless +static inline pgd_t pgdp_get_lockless(pgd_t *pgdp) +{ + return pgdp_get(pgdp); +} +#endif + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, From patchwork Thu Dec 5 10:37:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 123BEE77170 for ; Thu, 5 Dec 2024 10:45:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+rqxzBM2GhBfjuuxT3loWPDbusKa5KkzlT+fUlFXAkw=; b=AuDjc+20ocNm3p bjI4tS3NMv5pCp58g5l1IBSE11vzQ8KVPzoHRUca8XXOAwFqrN929KOUBowIzqdebwpd5ppNV1aHt GGgWPzPPOt/thUobnxadYR5oqsRHGe2L4VP4p+uqRMFi8rK/EaGjc9I8kBaNJJoSfkR7+LYeRCIN/ pWdlb126lscuFMBzUve/CuKi1UHZ6b9kA4pi9c7SxTUWgjRTiD7cD57Xfzbx06rJljthYIZCngftU oBGGCFhxmP8B0WZhLq2WhaB8H0PXFEY0eh3/ekKZBOikcDLACka/rJxdFDCHLGHLa5hmcQe3NzjDC R6ZY49M0szTicsc5UKgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lr-0000000FaS0-3pmX; Thu, 05 Dec 2024 10:44:55 +0000 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FQ-0000000FYBU-14Yc for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:17 +0000 Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-7fd17f230c2so424815a12.0 for ; Thu, 05 Dec 2024 02:38:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395095; x=1733999895; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=83UO5ozW5NdISNbxrvGH5KVgxXxH6sNR5OZieItF5/g=; b=J21zQXG3GLISQbU7dnogfBUBmYYCz55IaXD5ojK2mwDE7jBJ23zIlzaTCZKg3Ih6qK dlnKy/YJw+doP8QQYXeWnazHRxvYmWEMVfaC614bx7OdZXumq+j9zEePDzj0H+gTB2yv yWxZU6YC1gvBRsv6msPIjhB/iyCKdfrXyU2GN/0eCFfoiNRcImXGtx7rM75Gg4djNYFP IAlf5oG+WmYNTUdId/nQ9qgzLkKspTqZtX4u/A4c7qZ84pigV+DL7QJVtEKWloX0WH1Y x/B5FcUGxln0KjuxwgwGwimQBPSsA6cnpPmZwWJegOFo9IrnMkANQX3tM2UfpEkiNyek Qcnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395095; x=1733999895; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=83UO5ozW5NdISNbxrvGH5KVgxXxH6sNR5OZieItF5/g=; b=uqLupPYjr0uQYF/e1Chj5lOnaIDjFecQIaZAu821bZ8Nbs7fqZRPJjV7i5ySJgbaSq QheCxU/EEoa+NLENHmIYuV69f/hjBFPUIpxcA3SZPy5Sl7SmfRT4QJAtPHaLHhDqK2fE HfJrqBX1WUKE89N/pfCrlG3TjPBNcdkK/DoPZmxBetXQh0z0Cpg8mAb6tF3xcB2qxqT3 H/R/44/Rr+nFpVUmK/Y2b3pLa3YBw1lUjy7FBcX/TzVsop4bEr+cfa0vSAME5ShDbgg2 4d4J+uy0T3CWMr3YT0YaX1Yyh2MK+6tnpCSECpSsqNtl0EWu09jPbDIRiyh07P/iYfYL vNfQ== X-Forwarded-Encrypted: i=1; AJvYcCXCfRLEwRRkyupXlosG1ZmPrfl1xnIbWjPaPTC1x9BbbSqiYfflvs9jz5sHIxSW7RjCWOrki6qoE6ygdg==@lists.infradead.org X-Gm-Message-State: AOJu0YzS2EgZftymkO3rgIVy/UCrqpobtfcub2zLDKQfqmNyLPxb6VyW eZMwjc4vOqx8W7xXVAWgQBAZezEicIdDQ2P0L5RZlrw6rByphrwWhFnKOM73o7o= X-Gm-Gg: ASbGncuoNsNRwgAhwdt1ywIoCuUiLM3+vUpXcIDOYjhW+vNLgZSj4JDYVxbtmK4fiRG QVaIacabwqlCuSalajX3ksMnEh2MsBUt+R6gBX/WjHRU6jjht0RXIZh6oNnZ/vQ3TWY0uEZvuWt 3zS0vmigBkYYOwx59qWwjFNAnNU7FWTLpxCEOcK06g3nzvnc+okgk5Iia07hDjYyjYwsOG+ZDK+ 5M+A4ULtyrT4hAFEfluGgqiyxOA0ZDePrZqmE84pkNObqQKBpaEJAJkesamfmiYeKRrlRuad+Mx ltFe/AVyPlPpkInXwUoa2o3oYKaE7mn5 X-Google-Smtp-Source: AGHT+IHonPj5or3FCVlbY151Zkh3FREpG17bX8AKYVEQ/Cqc6X6geX5o43T/JV8/DVhtAO5u57D41Q== X-Received: by 2002:a05:6a21:99a2:b0:1d9:c615:944f with SMTP id adf61e73a8af0-1e1653b9f1bmr15516117637.11.1733395095235; Thu, 05 Dec 2024 02:38:15 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:15 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 09/21] riscv: mm: Replace READ_ONCE with atomic pte get function Date: Thu, 5 Dec 2024 18:37:17 +0800 Message-Id: <20241205103729.14798-10-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023816_296497_FB6A5358 X-CRM114-Status: GOOD ( 16.16 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org READ_ONCE can not be applied to pte structure with multipling mapping entries. This commit replaces READ_ONCE with atomic pte get function. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-64.h | 6 +++--- arch/riscv/include/asm/pgtable.h | 21 +++++++++++++-------- arch/riscv/kernel/hibernate.c | 18 +++++++++--------- arch/riscv/mm/pgtable.c | 12 +++++++++--- kernel/events/core.c | 6 +++--- mm/debug_vm_pgtable.c | 4 ++-- mm/gup.c | 10 +++++----- mm/hmm.c | 2 +- mm/mapping_dirty_helpers.c | 2 +- mm/memory.c | 4 ++-- mm/mprotect.c | 2 +- mm/ptdump.c | 8 ++++---- mm/sparse-vmemmap.c | 2 +- mm/vmscan.c | 2 +- 14 files changed, 55 insertions(+), 44 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index efcf63667f93..2649cc90b14e 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -242,7 +242,7 @@ static inline int pud_user(pud_t pud) static inline void set_pud(pud_t *pudp, pud_t pud) { - WRITE_ONCE(*pudp, pud); + *pudp = pud; } static inline void pud_clear(pud_t *pudp) @@ -318,7 +318,7 @@ static inline unsigned long _pmd_pfn(pmd_t pmd) static inline void set_p4d(p4d_t *p4dp, p4d_t p4d) { if (pgtable_l4_enabled) - WRITE_ONCE(*p4dp, p4d); + *p4dp = p4d; else set_pud((pud_t *)p4dp, __pud(p4d_val(p4d))); } @@ -401,7 +401,7 @@ pud_t *pud_offset(p4d_t *p4d, unsigned long address); static inline void set_pgd(pgd_t *pgdp, pgd_t pgd) { if (pgtable_l5_enabled) - WRITE_ONCE(*pgdp, pgd); + *pgdp = pgd; else set_p4d((p4d_t *)pgdp, __p4d(pgd_val(pgd))); } diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index fe42afb4441e..bf724d006236 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -289,7 +289,7 @@ static inline bool pmd_leaf(pmd_t pmd) static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) { - WRITE_ONCE(*pmdp, pmd); + *pmdp = pmd; } static inline void pmd_clear(pmd_t *pmdp) @@ -713,7 +713,7 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) */ static inline void set_pte(pte_t *ptep, pte_t pteval) { - WRITE_ONCE(*ptep, pteval); + *ptep = pteval; } static inline pte_t ptep_get(pte_t *ptep) @@ -953,10 +953,9 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long a static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); - + pte_t pte = ptep_get(ptep); + pte_clear(mm, address, ptep); page_table_check_pte_clear(mm, pte); - return pte; } @@ -964,7 +963,8 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) { - atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); + pte_t old_pte = ptep_get(ptep); + set_pte(ptep, pte_wrprotect(old_pte)); } #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH @@ -1170,8 +1170,9 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - pmd_t pmd = __pmd(atomic_long_xchg((atomic_long_t *)pmdp, 0)); + pmd_t pmd = pmdp_get(pmdp); + pmd_clear(pmdp); page_table_check_pmd_clear(mm, pmd); return pmd; @@ -1188,8 +1189,12 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { + pmd_t old_pmd = pmdp_get(pmdp); + page_table_check_pmd_set(vma->vm_mm, pmdp, pmd); - return __pmd(atomic_long_xchg((atomic_long_t *)pmdp, pmd_val(pmd))); + set_pmd(pmdp, pmd); + + return old_pmd; } #define pmdp_collapse_flush pmdp_collapse_flush diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 155be6b1d32c..5018d38f5280 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -171,7 +171,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long pte_t *src_ptep; pte_t *dst_ptep; - if (pmd_none(READ_ONCE(*dst_pmdp))) { + if (pmd_none(pmdp_get_lockless(dst_pmdp))) { dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); if (!dst_ptep) return -ENOMEM; @@ -183,7 +183,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long src_ptep = pte_offset_kernel(src_pmdp, start); do { - pte_t pte = READ_ONCE(*src_ptep); + pte_t pte = ptep_get_lockless(src_ptep); if (pte_present(pte)) set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); @@ -200,7 +200,7 @@ static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long pmd_t *src_pmdp; pmd_t *dst_pmdp; - if (pud_none(READ_ONCE(*dst_pudp))) { + if (pud_none(pudp_get_lockless(dst_pudp))) { dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); if (!dst_pmdp) return -ENOMEM; @@ -212,7 +212,7 @@ static int temp_pgtable_map_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long src_pmdp = pmd_offset(src_pudp, start); do { - pmd_t pmd = READ_ONCE(*src_pmdp); + pmd_t pmd = pmdp_get_lockless(src_pmdp); next = pmd_addr_end(start, end); @@ -239,7 +239,7 @@ static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long pud_t *dst_pudp; pud_t *src_pudp; - if (p4d_none(READ_ONCE(*dst_p4dp))) { + if (p4d_none(p4dp_get_lockless(dst_p4dp))) { dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); if (!dst_pudp) return -ENOMEM; @@ -251,7 +251,7 @@ static int temp_pgtable_map_pud(p4d_t *dst_p4dp, p4d_t *src_p4dp, unsigned long src_pudp = pud_offset(src_p4dp, start); do { - pud_t pud = READ_ONCE(*src_pudp); + pud_t pud = pudp_get_lockless(src_pudp); next = pud_addr_end(start, end); @@ -278,7 +278,7 @@ static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long p4d_t *dst_p4dp; p4d_t *src_p4dp; - if (pgd_none(READ_ONCE(*dst_pgdp))) { + if (pgd_none(pgdp_get_lockless(dst_pgdp))) { dst_p4dp = (p4d_t *)get_safe_page(GFP_ATOMIC); if (!dst_p4dp) return -ENOMEM; @@ -290,7 +290,7 @@ static int temp_pgtable_map_p4d(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long src_p4dp = p4d_offset(src_pgdp, start); do { - p4d_t p4d = READ_ONCE(*src_p4dp); + p4d_t p4d = p4dp_get_lockless(src_p4dp); next = p4d_addr_end(start, end); @@ -317,7 +317,7 @@ static int temp_pgtable_mapping(pgd_t *pgdp, unsigned long start, unsigned long unsigned long ret; do { - pgd_t pgd = READ_ONCE(*src_pgdp); + pgd_t pgd = pgdp_get_lockless(src_pgdp); next = pgd_addr_end(start, end); diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index f57ada26a183..150aea8e2d7a 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -128,9 +128,15 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (!pte_young(ptep_get(ptep))) - return 0; - return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); + int r = 1; + pte_t pte = ptep_get(ptep); + + if (!pte_young(pte)) + r = 0; + else + set_pte(ptep, pte_mkold(pte)); + + return r; } EXPORT_SYMBOL_GPL(ptep_test_and_clear_young); diff --git a/kernel/events/core.c b/kernel/events/core.c index df27d08a7232..84d49c60f55b 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7709,7 +7709,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) pte_t *ptep, pte; pgdp = pgd_offset(mm, addr); - pgd = READ_ONCE(*pgdp); + pgd = pgdp_get_lockless(pgdp); if (pgd_none(pgd)) return 0; @@ -7717,7 +7717,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return pgd_leaf_size(pgd); p4dp = p4d_offset_lockless(pgdp, pgd, addr); - p4d = READ_ONCE(*p4dp); + p4d = p4dp_get_lockless(p4dp); if (!p4d_present(p4d)) return 0; @@ -7725,7 +7725,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return p4d_leaf_size(p4d); pudp = pud_offset_lockless(p4dp, p4d, addr); - pud = READ_ONCE(*pudp); + pud = pudp_get_lockless(pudp); if (!pud_present(pud)) return 0; diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index bc748f700a9e..1cec548cc6c7 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -438,7 +438,7 @@ static void __init pmd_huge_tests(struct pgtable_debug_args *args) * X86 defined pmd_set_huge() verifies that the given * PMD is not a populated non-leaf entry. */ - WRITE_ONCE(*args->pmdp, __pmd(0)); + set_pmd(args->pmdp, __pmd(0)); WARN_ON(!pmd_set_huge(args->pmdp, __pfn_to_phys(args->fixed_pmd_pfn), args->page_prot)); WARN_ON(!pmd_clear_huge(args->pmdp)); pmd = pmdp_get(args->pmdp); @@ -458,7 +458,7 @@ static void __init pud_huge_tests(struct pgtable_debug_args *args) * X86 defined pud_set_huge() verifies that the given * PUD is not a populated non-leaf entry. */ - WRITE_ONCE(*args->pudp, __pud(0)); + set_pud(args->pudp, __pud(0)); WARN_ON(!pud_set_huge(args->pudp, __pfn_to_phys(args->fixed_pud_pfn), args->page_prot)); WARN_ON(!pud_clear_huge(args->pudp)); pud = pudp_get(args->pudp); diff --git a/mm/gup.c b/mm/gup.c index ad0c8922dac3..db444d732028 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1004,7 +1004,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; pudp = pud_offset(p4dp, address); - pud = READ_ONCE(*pudp); + pud = pudp_get_lockless(pudp); if (!pud_present(pud)) return no_page_table(vma, flags, address); if (pud_leaf(pud)) { @@ -1029,7 +1029,7 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4d_t *p4dp, p4d; p4dp = p4d_offset(pgdp, address); - p4d = READ_ONCE(*p4dp); + p4d = p4dp_get_lockless(p4dp); BUILD_BUG_ON(p4d_leaf(p4d)); if (!p4d_present(p4d) || p4d_bad(p4d)) @@ -3259,7 +3259,7 @@ static int gup_fast_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, pudp = pud_offset_lockless(p4dp, p4d, addr); do { - pud_t pud = READ_ONCE(*pudp); + pud_t pud = pudp_get_lockless(pudp); next = pud_addr_end(addr, end); if (unlikely(!pud_present(pud))) @@ -3285,7 +3285,7 @@ static int gup_fast_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { - p4d_t p4d = READ_ONCE(*p4dp); + p4d_t p4d = p4dp_get_lockless(p4dp); next = p4d_addr_end(addr, end); if (!p4d_present(p4d)) @@ -3307,7 +3307,7 @@ static void gup_fast_pgd_range(unsigned long addr, unsigned long end, pgdp = pgd_offset(current->mm, addr); do { - pgd_t pgd = READ_ONCE(*pgdp); + pgd_t pgd = pgdp_get_lockless(pgdp); next = pgd_addr_end(addr, end); if (pgd_none(pgd)) diff --git a/mm/hmm.c b/mm/hmm.c index 7e0229ae4a5a..fa56b735883e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -423,7 +423,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, /* Normally we don't want to split the huge page */ walk->action = ACTION_CONTINUE; - pud = READ_ONCE(*pudp); + pud = pudp_get_lockless(pudp); if (!pud_present(pud)) { spin_unlock(ptl); return hmm_vma_walk_hole(start, end, -1, walk); diff --git a/mm/mapping_dirty_helpers.c b/mm/mapping_dirty_helpers.c index 2f8829b3541a..8771432c3300 100644 --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -149,7 +149,7 @@ static int wp_clean_pud_entry(pud_t *pud, unsigned long addr, unsigned long end, struct mm_walk *walk) { #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD - pud_t pudval = READ_ONCE(*pud); + pud_t pudval = pudp_get_lockless(pud); /* Do not split a huge pud */ if (pud_trans_huge(pudval) || pud_devmap(pudval)) { diff --git a/mm/memory.c b/mm/memory.c index bdf77a3ec47b..03ee104cb009 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6428,12 +6428,12 @@ int follow_pfnmap_start(struct follow_pfnmap_args *args) goto out; p4dp = p4d_offset(pgdp, address); - p4d = READ_ONCE(*p4dp); + p4d = p4dp_get_lockless(p4dp); if (p4d_none(p4d) || unlikely(p4d_bad(p4d))) goto out; pudp = pud_offset(p4dp, address); - pud = READ_ONCE(*pudp); + pud = pudp_get_lockless(pudp); if (pud_none(pud)) goto out; if (pud_leaf(pud)) { diff --git a/mm/mprotect.c b/mm/mprotect.c index 6f450af3252e..a165ab597a73 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -447,7 +447,7 @@ static inline long change_pud_range(struct mmu_gather *tlb, break; } - pud = READ_ONCE(*pudp); + pud = pudp_get_lockless(pudp); if (pud_none(pud)) continue; diff --git a/mm/ptdump.c b/mm/ptdump.c index 106e1d66e9f9..b8a2ad43392f 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -30,7 +30,7 @@ static int ptdump_pgd_entry(pgd_t *pgd, unsigned long addr, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st = walk->private; - pgd_t val = READ_ONCE(*pgd); + pgd_t val = pgdp_get_lockless(pgd); #if CONFIG_PGTABLE_LEVELS > 4 && \ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) @@ -53,7 +53,7 @@ static int ptdump_p4d_entry(p4d_t *p4d, unsigned long addr, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st = walk->private; - p4d_t val = READ_ONCE(*p4d); + p4d_t val = p4dp_get_lockless(p4d); #if CONFIG_PGTABLE_LEVELS > 3 && \ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) @@ -76,7 +76,7 @@ static int ptdump_pud_entry(pud_t *pud, unsigned long addr, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st = walk->private; - pud_t val = READ_ONCE(*pud); + pud_t val = pudp_get_lockless(pud); #if CONFIG_PGTABLE_LEVELS > 2 && \ (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) @@ -99,7 +99,7 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long next, struct mm_walk *walk) { struct ptdump_state *st = walk->private; - pmd_t val = READ_ONCE(*pmd); + pmd_t val = pmdp_get_lockless(pmd); #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) if (pmd_page(val) == virt_to_page(lm_alias(kasan_early_shadow_pte))) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index c0388b2e959d..6621fb096fd0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -337,7 +337,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return -ENOMEM; pmd = pmd_offset(pud, addr); - if (pmd_none(READ_ONCE(*pmd))) { + if (pmd_none(pmdp_get_lockless(pmd))) { void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); diff --git a/mm/vmscan.c b/mm/vmscan.c index 28ba2b06fc7d..2bc78c339fd1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3608,7 +3608,7 @@ static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long end, pud = pud_offset(p4d, start & P4D_MASK); restart: for (i = pud_index(start), addr = start; addr != end; i++, addr = next) { - pud_t val = READ_ONCE(pud[i]); + pud_t val = pudp_get_lockless(&pud[i]); next = pud_addr_end(addr, end); From patchwork Thu Dec 5 10:37:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3FDDE7717A for ; Thu, 5 Dec 2024 10:45:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=r2h7Y8gCYhjEiBVfls6o2qzqChFBV8uYwqwd5RyXXzo=; b=2uJu7OI7kbh8Pm QYpSL7eCivKWvYBHWvms8jXRz3JGVbL6YE6LTM0ioexShXNt8hyhrcBqly2M06lkg/3zHLtMU9+qR oXKhfxnKXdvAHzmRJ96A71p7s+ncckgmO30tfVXlq2VcUlpgO1bW7KEuNiV2cT9oVlZs8lJUff99Y 3y1aa2+ZWzad/xrjZDYELRQWRZlqOOOgnraa9s29ufEdZzRaBCP2ZAHTSolIIbsyWC+wgLoBlRR2t gFhaZ86PUIIJf4ZB5K/a/A/kCEwfvHleHUC+vrJLmxakYT+UPd4JKADB5cPWzrQOodbCctZYjOG4a UwM6Vqi5t9W8ldXwjWEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ls-0000000FaSa-2TzL; Thu, 05 Dec 2024 10:44:56 +0000 Received: from mail-ot1-x334.google.com ([2607:f8b0:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FU-0000000FYCS-18cg for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:21 +0000 Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-71d46ace74aso371853a34.3 for ; Thu, 05 Dec 2024 02:38:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395099; x=1733999899; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hfWeT6m9rP2RvyJAYvuGAOSzcqpyDy27Avf8NzKYXNE=; b=hYPxKtkWSsN2rordmFQQRyfMVoh2oYdZROAo4+HjqCuOzSOa9ced63XcW5Kjqf6ywn MHrvhpVS3GkvNYuA9sVOmkf341vHnRqFv3RC+HMp9gfbl+06k9jbGUQsQ7/eWnd4ORjc Q97GmwWD5Rp63Rhw8Vu/g3McosgowbiBOx1wmPgzGkciJjucnfJo1suJ+mDQCuNZf31L /O66CnmOaY3FImO9m3pGtL/4Eudt1gLNHH1na4TJEyXEpcf9h6G9z5K+vx+qPDJQwQZR sBsI+p0NZvR3ZvpumVelp2knN7MM6QgaLyDN1eNnwtC1wyfZm9BuwpQg3dAA1N6T9kfF LZBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395099; x=1733999899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hfWeT6m9rP2RvyJAYvuGAOSzcqpyDy27Avf8NzKYXNE=; b=GFq20yli0O8Q3xR9mILIzgaqmkIrp7TnwbBij9Xr4RN2mn3X2O9L/OWDwQ4CG+x+Sr M1iLYM6WlT2OoJ9vgveY9wyzlAU1pIqmIpksttz9PuxSkitoCwu0zeOtkJkJpSQXvgdc MezmG7XvRsh3BMuWXLFNVwWRhXNumLQtgyTo2NoodeRTQ1YdB6/R0TXVR0w6xLjeZ5xu 2mO369KZMXojGB8PsNsVWXPacTsVc5xFOt7ZcjMp1IlkONMHoaPbfkgMVCXABGth/9cr 9jU3jn+MFYkmZZqNH8PUXslTt3006vXBqF4KMlHJTDD51gdlnTUxNByeWlHMr2dkd2Nb VJOQ== X-Forwarded-Encrypted: i=1; AJvYcCVVd16K5fbcrskRQSnGxXN5mN4Phf1bGJkKmM9Fdu7dzH9nQ8HwzpqNe2DU09rLnkUIZ1dA3+qMAKOtiA==@lists.infradead.org X-Gm-Message-State: AOJu0YzJaEIg/BuEAScax22UWj/YVpM+TxI2jwU6x8aYO0r8RvL0FA/G ZxjIyEor/HsS9KDitT9EJAf5IekSJ0x/0CG54huD/2JSSAJiPSgnLRAvff5gXpE= X-Gm-Gg: ASbGncuh9XzPeqJm2gQwKONpLkDNgZvItazx+AU88U/7m1suY4Qn2I8myt/QN4pdQIe 6J6dTWhWmgaaZecJgbIJQn+11ZVC+CNamWrcV3uGwHYxTURzjEOBpP5Z+Op/RJsQ+Ecujt6FRxP EoD1ZREHRpNGqmKI1e85FeEv5nr80RKNDKMihYxXOQVgrN+zNrVmiZt/TPCwXs07pqiAW/n2NiX 0UnQBL3KRnPS8EkE75/U7mVuDN8yD1tJsJ8VcM5d0wJIoJEXpjCCvJhhXWNYQbS7uZqTGkLLjWI RxCV7gTw+BmK0SIpUizZOv7D2sSJCEpq X-Google-Smtp-Source: AGHT+IHmo8dAIPLaAwW/lT+NI/PJnWblM6aaHV5WtIj+KNZJRyA2qtNhsMIENCAwyOOd5jQe2iKEOQ== X-Received: by 2002:a05:6830:2aa8:b0:71d:4488:19d9 with SMTP id 46e09a7af769-71dad652c1fmr12149732a34.17.1733395099595; Thu, 05 Dec 2024 02:38:19 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:19 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 10/21] riscv: mm: Reimplement PTE A/D bit check function Date: Thu, 5 Dec 2024 18:37:18 +0800 Message-Id: <20241205103729.14798-11-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023820_305878_158D78D5 X-CRM114-Status: UNSURE ( 9.09 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org CPU that supports only 4K MMU usually updates access/dirty bit at 4K pte level. As each software page can contains multiple 4K hardware pages, we need to traverse all mapping entries to check whether corresponding 4K page is accessed or dirty during pte_dirty/pte_access functions. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index bf724d006236..c0f7442c8a9e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -553,6 +553,29 @@ static inline int pte_huge(pte_t pte) return pte_present(pte) && (pte_val(pte) & _PAGE_LEAF); } +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline int pte_dirty(pte_t pte) +{ + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) + if (pte.ptes[i] & _PAGE_DIRTY) + return 1; + + return 0; +} + +static inline int pte_young(pte_t pte) +{ + unsigned int i; + + for (i = 0; i < HW_PAGES_PER_PAGE; i++) + if (pte.ptes[i] & _PAGE_ACCESSED) + return 1; + + return 0; +} +#else static inline int pte_dirty(pte_t pte) { return pte_val(pte) & _PAGE_DIRTY; @@ -562,6 +585,7 @@ static inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED; } +#endif /* CONFIG_RISCV_USE_SW_PAGE */ static inline int pte_special(pte_t pte) { From patchwork Thu Dec 5 10:37:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B49FFE7716C for ; Thu, 5 Dec 2024 10:45:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LPvVXxzJ02U8a1QtqzvgM4QgY3RbcFbxUHJZ4dlb5MY=; b=DC7fHwpvuI7D8C /X4601UxPULito2SA0cdoJUQG+VPc8mq0bgTnV641ftNOg6xFdS+7HWJkpbuuq0BlflmKgnQ+EObl my65Nluth5SsKfMtBmWZW0Nzd23i0vstU3zT0GIlco5F6FkL1qOrfGbTyadZZaw+NkpVkKdPAHIuc wfTy2jXY9VlMwGjE/GYTcBFquK4okOcV3jYO1hLV21FLGPaNdSOfELNoWIl8APN34f/0yhA8sjTLc XzXzm59nX83fSsQQIkrQrcSaQVvbKH5F0gt/uuepSXrua2G3zoIvFN5KPkQRwE4gjddSYJNZkJnmg eRDoUzT+BqAgvX1bkWKA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lt-0000000FaT8-19CT; Thu, 05 Dec 2024 10:44:57 +0000 Received: from mail-oa1-x2a.google.com ([2001:4860:4864:20::2a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9FY-0000000FYDw-4615 for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:26 +0000 Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-29e5c0c46c3so498904fac.3 for ; Thu, 05 Dec 2024 02:38:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395104; x=1733999904; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XIMWQ57ZznUQuD/aiNRYDz3W26VtFUEaOqg8e4FokQk=; b=AFC9mAvx5lIoz+wY+OmDiKtjk1urLrrKShrVjpMq0G+e4z4tiK6LgYHIT3DwV6UrVs QWr1QYk3R7xsRBMbOWCKMHV/uZiMghf9E/x6kruh3aobs3x0l9uXmXeamsNVZYOtv5wt QjB38Zf34f2YQGcfFB8oWAlS+qDzB04Whcz5+Y+rAjgRclyZ2shwPnxhnd1fe8mbwY/L GLfevdBkz19ilRKSjMpIb7A70uc+5fS9tuUgqn6i8AJlHAV0TzN1HOhl8zl8qqFZbBFr e1t6inSP/kjVtDTFnE0NLok03F6ANlj+6z02x+xiQvjJ8m2prKFIfXsM+uOQaNsamYGI cyDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395104; x=1733999904; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XIMWQ57ZznUQuD/aiNRYDz3W26VtFUEaOqg8e4FokQk=; b=T0FBfiVA66ovXm0qp5K/5uPaPHyy+ySL5sl1ia7rj/HTM6hzp0ClaFnAlJ+v0zPRmu VVBxPgk7did9QkEmruQGyaJS9xCOVu2+ABRqTFruFSdmcsYHgxBdLIZSvU6JI8mlX3vh HHGM5Jka/KDr+M2M05AQukEUCqXtlNuC9L+w61S/2rB81ErWCER+X9RJb2k4cN23/Exq MvT/+dqZQYxma+w6T1VDdb1Fz5AT8WpeIzPNyAiqwx6aAIGeZSnllACj4jtMx0gDOmYb RITgYrWvjGgT0FDPSlflJZgwLr9PqGi55+RQB0rO/a1utUonpgKrjNSLuaXGrbMBt9lO 7TsQ== X-Forwarded-Encrypted: i=1; AJvYcCXbXr24bBuF/wyY0nAfqwzzpq0Mx6mR2ov+pN+q22gEjgZgpxdZVcnoM+n7NZU7j00cTqIu1iSD8bWv2g==@lists.infradead.org X-Gm-Message-State: AOJu0YzpsDnYpC3/xAICLmJFoW0+SkSMoaboCSSy55WcLc97tWb3T1f+ mj3m+x0lUkmGyAjj8E+ki+6dzRm3gYBLcDfjFdGXoLgzBkBxtRRCodtJYmZ39G4= X-Gm-Gg: ASbGncugDUNhj2QijCAcBH6+0vVrzL66ZRPVr2TBzrIp8cSnmTfK105L2hToqh7P8mW FBAxsJGHAIND/fpmwF/xD1/v4pC3dF+suolHSO8n7kPaDTPpYOyBMyLYhMTknfH3T5B5+RMKni1 PgPOo4vkqz+3GwWrK4AAAL2mVrYgvA3pe3bCHCz3S/41jT5FVgXGv90NQFDVBBFb8C/nAfYPMBb boZxsffBQsq/IXY6i0aTd3Uh9L2nJmZHfFurFKL9cPlRo/MVO3F1Zy+zcxqj52fMVdPv7IJZzIR PVWr/2LSzeKiA8l8urhguxp0zKuNdMR3 X-Google-Smtp-Source: AGHT+IFGD682lN2KXS9mZyYiAgPgUddfTIrz2tftbpWLW1OwVDgKSjli0UbK/dW/c1XvobnxdhptFQ== X-Received: by 2002:a05:6870:4944:b0:29d:c5e8:e41f with SMTP id 586e51a60fabf-29e88576409mr11051212fac.5.1733395103681; Thu, 05 Dec 2024 02:38:23 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:23 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 11/21] riscv: mm: Reimplement mk_huge_pte function Date: Thu, 5 Dec 2024 18:37:19 +0800 Message-Id: <20241205103729.14798-12-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023825_018591_993025C2 X-CRM114-Status: GOOD ( 13.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Huge pte can be pud, pmd, or svnapot pte. Huge ptes from different page table levels have different pte constructors. This commit reimplements mk_huge_pte function. We take vma struct as argument to check the target huge pte level and applying corresponding constructor. Signed-off-by: Xu Lu --- arch/riscv/include/asm/hugetlb.h | 5 +++++ arch/riscv/mm/hugetlbpage.c | 23 ++++++++++++++++++++++- arch/s390/include/asm/hugetlb.h | 2 +- include/asm-generic/hugetlb.h | 5 ++++- mm/debug_vm_pgtable.c | 2 +- mm/hugetlb.c | 4 ++-- 6 files changed, 35 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index faf3624d8057..eafd00f4b74f 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -51,6 +51,11 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#ifdef CONFIG_RISCV_USE_SW_PAGE +#define __HAVE_ARCH_MK_HUGE_PTE +pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *page, pgprot_t pgprot); +#endif + #include #endif /* _ASM_RISCV_HUGETLB_H */ diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 42314f093922..8896c28ec881 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -2,6 +2,27 @@ #include #include +#ifdef CONFIG_RISCV_USE_SW_PAGE +pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *page, pgprot_t pgprot) +{ + pte_t pte; + unsigned int shift = huge_page_shift(hstate_vma(vma)); + + if (shift == PGDIR_SHIFT) + pte = pgd_pte(pfn_pgd(page_to_pfn(page), pgprot)); + else if (shift == P4D_SHIFT) + pte = p4d_pte(pfn_p4d(page_to_pfn(page), pgprot)); + else if (shift == PUD_SHIFT) + pte = pud_pte(pfn_pud(page_to_pfn(page), pgprot)); + else if (shift == PMD_SHIFT) + pte = pmd_pte(pfn_pmd(page_to_pfn(page), pgprot)); + else + pte = pfn_pte(page_to_pfn(page), pgprot); + + return pte; +} +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #ifdef CONFIG_RISCV_ISA_SVNAPOT pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { @@ -74,7 +95,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, out: if (pte) { - pte_t pteval = ptep_get_lockless(pte); + pte_t pteval = ptep_get(pte); WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval)); } diff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetlb.h index cf1b5d6fb1a6..cea9118d4bba 100644 --- a/arch/s390/include/asm/hugetlb.h +++ b/arch/s390/include/asm/hugetlb.h @@ -79,7 +79,7 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm, __set_huge_pte_at(mm, addr, ptep, pte_wrprotect(pte)); } -static inline pte_t mk_huge_pte(struct page *page, pgprot_t pgprot) +static inline pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *page, pgprot_t pgprot) { return mk_pte(page, pgprot); } diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h index 594d5905f615..90765bc03bba 100644 --- a/include/asm-generic/hugetlb.h +++ b/include/asm-generic/hugetlb.h @@ -5,10 +5,13 @@ #include #include -static inline pte_t mk_huge_pte(struct page *page, pgprot_t pgprot) +#ifndef __HAVE_ARCH_MK_HUGE_PTE +static inline pte_t mk_huge_pte(struct vm_area_struct *vma, struct page *page, + pgprot_t pgprot) { return mk_pte(page, pgprot); } +#endif static inline unsigned long huge_pte_write(pte_t pte) { diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 1cec548cc6c7..24839883d513 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -919,7 +919,7 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) * as it was previously derived from a real kernel symbol. */ page = pfn_to_page(args->fixed_pmd_pfn); - pte = mk_huge_pte(page, args->page_prot); + pte = mk_huge_pte(args->vma, page, args->page_prot); WARN_ON(!huge_pte_dirty(huge_pte_mkdirty(pte))); WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte)))); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 190fa05635f4..2b33eb46408f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5140,10 +5140,10 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page, unsigned int shift = huge_page_shift(hstate_vma(vma)); if (writable) { - entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(page, + entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(vma, page, vma->vm_page_prot))); } else { - entry = huge_pte_wrprotect(mk_huge_pte(page, + entry = huge_pte_wrprotect(mk_huge_pte(vma, page, vma->vm_page_prot)); } entry = pte_mkyoung(entry); From patchwork Thu Dec 5 10:37:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E9FF4E77177 for ; Thu, 5 Dec 2024 10:45:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Jo97bj+gMvLiVCussfpn7l8dgH80Y5ojBs0WObAMwII=; b=AQ9gvfJGZZEkkA 3v8+ZDqKu5OGGSBLrbqsiMZ9QAtpqYUPpOBJDheCnrA+/67IQYJKVZa0Zp+XNuksGqY/jnrNocYVw EV7VLxllGXVMGX7vAlFPs12OZDxER3b7qki9bRPBEPSPdwA7wnU4/IozpSwGfaSPjMCYnJrngQ9al jf4O2Mo0VS2W4txr0waMnq7oa9kkFqjRRrUwlx7wUEcul7gUu6gGJLx2amt89SaEskeOo2dY7dDCu 0+BjrLa5J+AdsEWR6bF5qSdDCvzVbKbrKehhItJUeDdmfQU3aaxNae0L07oCkCM2xFEFx2GGnUTZF DnLUoqMvnwSwty4h2ZvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lt-0000000FaTi-3uW0; Thu, 05 Dec 2024 10:44:57 +0000 Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Fc-0000000FYEl-1Brh for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:29 +0000 Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-7fbd9be84bdso709414a12.1 for ; Thu, 05 Dec 2024 02:38:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395108; x=1733999908; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFQIXohFMC+aCLCj7cIVe0JD4G758F8Hwu3svQYABEY=; b=Kez6TqyAN2R+7ox2x5Svoded4boLTxI9+FQS1zzxYec0KceaDPwn8T22ueNuzZru6d ootlnPW91GBTtkmcnfNzSKihebTfaG82CZ+FZGrLP61nVJEN9aD5xA9qISJ1goz3Hjl1 Z8Chf/sOXfZ+tEqmwSOlfPIP4W6J1BP3c7qiwXVY5gXapGbzFQMIitJKOz71xaIfsNG5 de0izF1d+5/8tWpOVF/gTngM1ulDnJ0j50l4fqcjVmlcko1W3HMHan/mWbuGMPZahEkV vs7jYkHUh8KE14GI0VrFIpGiWmHxw7Nbiw07BLpM1Mae2VJjZp8Yikq6TYWCCYy2107l auRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395108; x=1733999908; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HFQIXohFMC+aCLCj7cIVe0JD4G758F8Hwu3svQYABEY=; b=q7xf3ueQIF7Y9d1msnjtMaky7Gk1rXTada/bcRPaVMaC3Lny7NkYGzjtRz7/Y2XhcZ CQljAmXXFesPqpjlwFpCDeBO4GmQe96zWQJ4JciJn80u8CJOhXZ3mC8hEI+GLnAwrFmX HhSnvzKP3NDRR/eCCskI8Yq7p/xzYdgDcw4hQwS5som0PtK/y5MgMZIaxDI2cw3uAuAk iGFdoVjT10vOxl2xw2A3+3hGmVN3IWVDY/h+O+ZsxTMV2aQ+z/0gca8cXq75R/1JPyZT U0owkzTap23P+Cjb6rwKd2l/DrcvSM+H8DSnmWn05UQL5Nur0de1dQd80n2uZjRvx9au ci9Q== X-Forwarded-Encrypted: i=1; AJvYcCVW2nt6Xf69KTbzmbN9kG2JVD7k6k+EaQHPnu0XchTlS+1uyahV6fTIeFMh8cBLqKYY31XWESrm3SG5lA==@lists.infradead.org X-Gm-Message-State: AOJu0YyAKL4X5mWOV2sQJ5xxHu0WqNgleGVqSCfWBOahISL3f/P726wF yDPooMN7MRt4nmQ8FrPD+vWl1qUmi4j/BrAh06KQOGFoul1tVnNSOzolwe0rwMo= X-Gm-Gg: ASbGncsKYl733jEBZ4lb8K2qQlSk+b6xIHDmEKi/JZV1DmHoFDtfQZmrCO5LZ+pjqm0 6dhxrBY+Xu8UzTD4Pn4vcW8rEu5UCa/C18O+UGFgew1k/UQrFcQUB91IHqQ99Z4frFxjb0kuJME AfShcbNjoAoIBFJwruRwQrjBqM0T8GiosVQcb7A8jGZCceKrz7ri+y6snlu6YNESqvfp/u4k24Z S7/FVwsXIW77OmqDRPMixFpcm1W/KoZAYhFhB01fKCIyKvaW+737e5gP/RH8xVhgZbtbBECBxwM vl9+l6DqWdfX/wsRX5G4TU8r9spQqvPg X-Google-Smtp-Source: AGHT+IEVSD+tgAqJx2WiLXY6iHldH8R0b3LHGoJ+robZaZ03Gbz5dxvb2ZG0GtuQ60x2xrNdVJqFOw== X-Received: by 2002:a05:6300:4041:b0:1e1:1659:82a4 with SMTP id adf61e73a8af0-1e16541341fmr16353259637.41.1733395107751; Thu, 05 Dec 2024 02:38:27 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:27 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 12/21] riscv: mm: Reimplement tlb flush function Date: Thu, 5 Dec 2024 18:37:20 +0800 Message-Id: <20241205103729.14798-13-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023828_323714_A94B0D69 X-CRM114-Status: GOOD ( 15.20 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When tlb flushing a page correponding to a certain address, CPU actually only flushes tlb entries of the first 4K hardware page. This commit reimplements tlb flushing function to flush all tlb entries of hardware pages in the same software page. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 9 ++++++--- arch/riscv/include/asm/tlbflush.h | 26 ++++++++++++++++++++------ arch/riscv/mm/fault.c | 13 +++++++++---- arch/riscv/mm/init.c | 2 +- arch/riscv/mm/tlbflush.c | 31 +++++++++++++++++++++---------- 5 files changed, 57 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index c0f7442c8a9e..9fa16c0c20aa 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -701,7 +701,7 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, * the extra traps reduce performance. So, eagerly SFENCE.VMA. */ while (nr--) - local_flush_tlb_page(address + nr * PAGE_SIZE); + local_flush_tlb_page(address + nr * PAGE_SIZE, PAGE_SIZE); svvptc:; /* @@ -719,9 +719,12 @@ svvptc:; static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pte_t *ptep = (pte_t *)pmdp; + asm goto(ALTERNATIVE("nop", "j %l[svvptc]", 0, RISCV_ISA_EXT_SVVPTC, 1) + : : : : svvptc); - update_mmu_cache(vma, address, ptep); + local_flush_tlb_page(address, PMD_SIZE); + +svvptc:; } #define __HAVE_ARCH_PTE_SAME diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 72e559934952..25cc39ab84d5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -29,18 +29,32 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) } /* Flush one page from local TLB */ -static inline void local_flush_tlb_page(unsigned long addr) +static inline void local_flush_tlb_page(unsigned long addr, + unsigned long page_size) { - ALT_SFENCE_VMA_ADDR(addr); + unsigned int i; + unsigned long hw_page_num = 1 << (PAGE_SHIFT - HW_PAGE_SHIFT); + unsigned long hw_page_size = page_size >> (PAGE_SHIFT - HW_PAGE_SHIFT); + + for (i = 0; i < hw_page_num; i++, addr += hw_page_size) + ALT_SFENCE_VMA_ADDR(addr); } static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long page_size, unsigned long asid) { - if (asid != FLUSH_TLB_NO_ASID) - ALT_SFENCE_VMA_ADDR_ASID(addr, asid); - else - local_flush_tlb_page(addr); + unsigned int i; + unsigned long hw_page_num, hw_page_size; + + if (asid != FLUSH_TLB_NO_ASID) { + hw_page_num = 1 << (PAGE_SHIFT - HW_PAGE_SHIFT); + hw_page_size = page_size >> (PAGE_SHIFT - HW_PAGE_SHIFT); + + for (i = 0; i < hw_page_num; i++, addr += hw_page_size) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + } else + local_flush_tlb_page(addr, page_size); } void flush_tlb_all(void); diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 4772152be0f9..94524e5adc0b 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -118,7 +118,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a pmd_t *pmd_k; pte_t *pte_k; int index; - unsigned long pfn; + unsigned long pfn, page_size; /* User mode accesses just cause a SIGSEGV */ if (user_mode(regs)) @@ -154,8 +154,10 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a no_context(regs, addr); return; } - if (pud_leaf(pudp_get(pud_k))) + if (pud_leaf(pudp_get(pud_k))) { + page_size = PUD_SIZE; goto flush_tlb; + } /* * Since the vmalloc area is global, it is unnecessary @@ -166,8 +168,10 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a no_context(regs, addr); return; } - if (pmd_leaf(pmdp_get(pmd_k))) + if (pmd_leaf(pmdp_get(pmd_k))) { + page_size = PMD_SIZE; goto flush_tlb; + } /* * Make sure the actual PTE exists as well to @@ -180,6 +184,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a no_context(regs, addr); return; } + page_size = PAGE_SIZE; /* * The kernel assumes that TLBs don't cache invalid @@ -188,7 +193,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * necessary even after writing invalid entries. */ flush_tlb: - local_flush_tlb_page(addr); + local_flush_tlb_page(addr, page_size); } static inline bool access_error(unsigned long cause, struct vm_area_struct *vma) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index f9334aab45a6..678b892b4ed8 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -356,7 +356,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot) set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else pte_clear(&init_mm, addr, ptep); - local_flush_tlb_page(addr); + local_flush_tlb_page(addr, PAGE_SIZE); } static inline pte_t *__init get_pte_virt_early(phys_addr_t pa) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 9b6e86ce3867..d5036f2a8244 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -27,7 +27,7 @@ static void local_flush_tlb_range_threshold_asid(unsigned long start, } for (i = 0; i < nr_ptes_in_range; ++i) { - local_flush_tlb_page_asid(start, asid); + local_flush_tlb_page_asid(start, stride, asid); start += stride; } } @@ -36,7 +36,7 @@ static inline void local_flush_tlb_range_asid(unsigned long start, unsigned long size, unsigned long stride, unsigned long asid) { if (size <= stride) - local_flush_tlb_page_asid(start, asid); + local_flush_tlb_page_asid(start, stride, asid); else if (size == FLUSH_TLB_MAX_SIZE) local_flush_tlb_all_asid(asid); else @@ -126,14 +126,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, start, end - start, page_size); } -void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) -{ - __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), - addr, PAGE_SIZE, PAGE_SIZE); -} - -void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, - unsigned long end) +static inline unsigned long local_flush_tlb_page_size(struct vm_area_struct *vma) { unsigned long stride_size; @@ -161,6 +154,24 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, } } + return stride_size; +} + +void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) +{ + unsigned long page_size; + + page_size = local_flush_tlb_page_size(vma); + __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), + addr, page_size, page_size); +} + +void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + unsigned long stride_size; + + stride_size = local_flush_tlb_page_size(vma); __flush_tlb_range(mm_cpumask(vma->vm_mm), get_mm_asid(vma->vm_mm), start, end - start, stride_size); } From patchwork Thu Dec 5 10:37:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 872D0E77175 for ; Thu, 5 Dec 2024 10:45:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jNiZUeBekrJDSLPsn3BorpQ5tGcLjCuOsUEuLYPsBm0=; b=kGeQcerI+pWdmg WxxT3nPI/C94kUN+88kTihI6D4m8BVDIynZ5TwQZlXFkzjzoDTnuZWbTn38cRZsLjLprEwu4ZVE7K nIw1iCpaaYD+a3kqLl5QSp9N8ghTXprLT8NW3mx68FuGZt34cCG44wOpg6WoYV88DoB3S6h5Xk4c2 OWCm/I2mvLSmg8FzV4ACfMe5/HgK+/O5i2N0pcPHsxsg+1q2kSk8qSL/4RcEi+MY1Q4CkpLLo4j5h wG+oms45MY8c2sd4fJiHXVNdbMgZWPqGfPmI/BKErJQP+5vLO+Q13Kb5QwOf8BTa0FXX5mCquVD64 KAd2W/2r+QFiNItnV3YQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lu-0000000FaUF-2u6X; Thu, 05 Dec 2024 10:44:58 +0000 Received: from mail-pf1-x433.google.com ([2607:f8b0:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Fg-0000000FYGG-25eH for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:33 +0000 Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-7258ed68cedso729181b3a.1 for ; Thu, 05 Dec 2024 02:38:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395112; x=1733999912; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=q07gCTIJJE8Xz9bnD//uUjqUF3bkJlTAdnYkbDObJk8=; b=PY0i8n5Pnbt00uUQ4lPjFDGrEjOyV38LMphCXihw0dXUxH9W2dc+wh2vwVVaIcCvF8 ImkhCZ8U8kFQ9B2AtlYR9uRh/ITKG+wUTpdhbrj/H8whhP1Dw/KVwV5vBQgK+dL1J/RQ ZX9ZiPMaajSkIxzY7W9bnb+J81Cgkk0iJdO23V3EftKavXMdRyZmSpPxov5KWHrTpZ4G eHQUTCFQ1bWKBiwrp4QGeq9npf67DAVUCe+dPpmap+6vuHGGsAwiD3p0m1g4IEQV0FjA VjUjoonNU1DZrFVsdteiwT6eKYbx/A0VXNS7pcOt2RftQZzBwKnlno+cj33N+zLu91Fn JFHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395112; x=1733999912; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q07gCTIJJE8Xz9bnD//uUjqUF3bkJlTAdnYkbDObJk8=; b=me6WOl+m/798dC+TKc9z2xV1+z3HZYB1R2f2tZ7jw+OgxwH+3JYzxqoeAFhZ9t/XPG sB5/+d3JGoFmCpAv1gfhkjZIBnrzk/tUTvFOoyR432Pxmn7qZbL0rY5SZ6N5BvdlZ9bd qSIlUqmPdU2O/TfWZ9T0WzTB+NHL/5UfG8VLB4C8wtQYD85YF1rrGBnepmQtFODWJX5w Jghq4/qWZqQ0rZSfl40BVmmh89JoVzOUCcb6XSZcv9pqpJVrFtxbqcmT/aVY9FKw42DK 2MAyahouAXIDQSZERdYoGo4jd4M9zZofcinTWhtzgO32ti4UepRcpGoej9jnxz74gako uBZg== X-Forwarded-Encrypted: i=1; AJvYcCW/Ovt0c56MD3o0SlpfoY6AS2cMUE3tsSiL2TuJEpMcBF5OlzynBqsz3VVgQi2ekhdCFGil9weVmug53Q==@lists.infradead.org X-Gm-Message-State: AOJu0YwiP1ga/a/waV9uN2EDDvk4Q+nO1SXJ5hlEHBXxLHEpqC4YhLtr n8BMBLYv6e66upkgnWnrZkVh7Zahoy/JTF/NuHkNinKgdkxHKkA8+SRdXEKKhrc= X-Gm-Gg: ASbGncv97FDVpJjdKQnXp3v1u3GAaSFkUFvNJnyAr4Ms8Ux/DAJn0Zzg93blYT5QZsl WonEWQW2UFJPWaR/wq98CStfpgFPI12gVFuoElYjaJsZoqB/sI7NsjhLcTvB04PWNiH3DaRmZbB fAKGkqSoUoXJMIkk9ui90ZzY0crfaE4pYw2DuOOpAY+5ZzpA3tZ3aalk259Q1z3QBDyORcQ27X8 Z6DSCRZIf+nNfyPVwpRRzCCRiDfpiYCuQvdTr/HwamdMVpTCJhxZWhwIPMQ5UtID88qUwMn4Bhv K+Qp2FhbLmvZnuRUT/PMo9tnK9zZaIv7 X-Google-Smtp-Source: AGHT+IG3G9tPg6qUeSO1QGI/Mdcn1uGTNUPe0Tsf/e+WSW8vxQ4wgkKlRQB0qkddigD/hXKfbzL+jQ== X-Received: by 2002:a05:6a00:138d:b0:71e:cb5:2219 with SMTP id d2e1a72fcca58-7257fa5e7d6mr10595111b3a.9.1733395111754; Thu, 05 Dec 2024 02:38:31 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:31 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 13/21] riscv: mm: Adjust PGDIR/P4D/PUD/PMD_SHIFT Date: Thu, 5 Dec 2024 18:37:21 +0800 Message-Id: <20241205103729.14798-14-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023832_536211_E688DAD2 X-CRM114-Status: GOOD ( 12.03 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This commit adjusts the SHIFT of pte index bits at each page table level. For example, in SV39, the traditional va behaves as: ---------------------------------------------- | pgd index | pmd index | pte index | offset | ---------------------------------------------- | 38 30 | 29 21 | 20 12 | 11 0 | ---------------------------------------------- When we choose 64K as basic software page, va now behaves as: ---------------------------------------------- | pgd index | pmd index | pte index | offset | ---------------------------------------------- | 38 34 | 33 25 | 24 16 | 15 0 | ---------------------------------------------- Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-32.h | 2 +- arch/riscv/include/asm/pgtable-64.h | 16 ++++++++-------- arch/riscv/include/asm/pgtable.h | 19 +++++++++++++++++++ 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/pgtable-32.h index 2959ab72f926..e0c5c62f88d9 100644 --- a/arch/riscv/include/asm/pgtable-32.h +++ b/arch/riscv/include/asm/pgtable-32.h @@ -11,7 +11,7 @@ #include /* Size of region mapped by a page global directory */ -#define PGDIR_SHIFT 22 +#define PGDIR_SHIFT (10 + PAGE_SHIFT) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE - 1)) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 2649cc90b14e..26c13484e721 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -13,9 +13,9 @@ extern bool pgtable_l4_enabled; extern bool pgtable_l5_enabled; -#define PGDIR_SHIFT_L3 30 -#define PGDIR_SHIFT_L4 39 -#define PGDIR_SHIFT_L5 48 +#define PGDIR_SHIFT_L3 (9 + 9 + PAGE_SHIFT) +#define PGDIR_SHIFT_L4 (9 + PGDIR_SHIFT_L3) +#define PGDIR_SHIFT_L5 (9 + PGDIR_SHIFT_L4) #define PGDIR_SHIFT (pgtable_l5_enabled ? PGDIR_SHIFT_L5 : \ (pgtable_l4_enabled ? PGDIR_SHIFT_L4 : PGDIR_SHIFT_L3)) /* Size of region mapped by a page global directory */ @@ -23,20 +23,20 @@ extern bool pgtable_l5_enabled; #define PGDIR_MASK (~(PGDIR_SIZE - 1)) /* p4d is folded into pgd in case of 4-level page table */ -#define P4D_SHIFT_L3 30 -#define P4D_SHIFT_L4 39 -#define P4D_SHIFT_L5 39 +#define P4D_SHIFT_L3 (9 + 9 + PAGE_SHIFT) +#define P4D_SHIFT_L4 (9 + P4D_SHIFT_L3) +#define P4D_SHIFT_L5 (9 + P4D_SHIFT_L3) #define P4D_SHIFT (pgtable_l5_enabled ? P4D_SHIFT_L5 : \ (pgtable_l4_enabled ? P4D_SHIFT_L4 : P4D_SHIFT_L3)) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) /* pud is folded into pgd in case of 3-level page table */ -#define PUD_SHIFT 30 +#define PUD_SHIFT (9 + 9 + PAGE_SHIFT) #define PUD_SIZE (_AC(1, UL) << PUD_SHIFT) #define PUD_MASK (~(PUD_SIZE - 1)) -#define PMD_SHIFT 21 +#define PMD_SHIFT (9 + PAGE_SHIFT) /* Size of region mapped by a page middle directory */ #define PMD_SIZE (_AC(1, UL) << PMD_SHIFT) #define PMD_MASK (~(PMD_SIZE - 1)) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9fa16c0c20aa..0fd9bd4e0d13 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -30,12 +30,27 @@ /* Number of entries in the page table */ #define PTRS_PER_PTE (PAGE_SIZE / sizeof(pte_t)) +#ifdef CONFIG_RISCV_USE_SW_PAGE + +/* + * PGDIR_SHIFT grows as PAGE_SIZE grows. To avoid va exceeds limitation, pgd + * index bits should be cut. Thus we use HW_PAGE_SIZE instead. + */ +#define __PTRS_PER_PGD (HW_PAGE_SIZE / sizeof(pgd_t)) +#define pgd_index(a) (((a) >> PGDIR_SHIFT) & (__PTRS_PER_PGD - 1)) + +#define KERN_VIRT_SIZE ((__PTRS_PER_PGD / 2 * PGDIR_SIZE) / 2) + +#else + /* * Half of the kernel address space (1/4 of the entries of the page global * directory) is for the direct mapping. */ #define KERN_VIRT_SIZE ((PTRS_PER_PGD / 2 * PGDIR_SIZE) / 2) +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #define VMALLOC_SIZE (KERN_VIRT_SIZE >> 1) #define VMALLOC_END PAGE_OFFSET #define VMALLOC_START (PAGE_OFFSET - VMALLOC_SIZE) @@ -1304,7 +1319,11 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) * Similarly for SV57, bits 63–57 must be equal to bit 56. */ #ifdef CONFIG_64BIT +#ifdef CONFIG_RISCV_USE_SW_PAGE +#define TASK_SIZE_64 (PGDIR_SIZE * __PTRS_PER_PGD / 2) +#else #define TASK_SIZE_64 (PGDIR_SIZE * PTRS_PER_PGD / 2) +#endif #define TASK_SIZE_MAX LONG_MAX #ifdef CONFIG_COMPAT From patchwork Thu Dec 5 10:37:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 48A22E77173 for ; Thu, 5 Dec 2024 10:45:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/iOTLn209lWzSw6kWGwKKV9AGKjDLkXTTDtWlOWxY6g=; b=x/ct4bI+k+svjl CgOhOK2uke/q+d3jOdxY39cpL/5DCXvOraGCx8f8mID6Ff86QcBJV0+I0SURRQBTNAoF9MZpr7j6L 3j5+Fx7F+Cd3gDOn74kg92vXEA7vhd+bTm9iUtDcE1i6WKVfLvl5sTWK5qlJCpiKWiqPOs6T26aj0 tXMlqPq9qRy9QXFcB1SKU2eVa7GfPrbPsChK6G6D+8U7mPCT1keEYceS98KutBoPomeV6XH1DT/yg qEme+gc7WOeoQlDc8geVa7OXZnMGw7l3p49zcyBLyX7RIXmLrMdBUZniA38veK9E2vu0+BtietLL4 AGRJXECyXlwyMfKuBs5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lv-0000000FaVa-2cyd; Thu, 05 Dec 2024 10:44:59 +0000 Received: from mail-ot1-x333.google.com ([2607:f8b0:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Fk-0000000FYHX-3bp8 for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:37 +0000 Received: by mail-ot1-x333.google.com with SMTP id 46e09a7af769-71d41932d32so398936a34.0 for ; Thu, 05 Dec 2024 02:38:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395116; x=1733999916; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nxSp7kmSqkSnu4KEBfrma7Z74EMZDahAlv9AUZJtR6k=; b=JHZwtCtGVdYldN0D80a2oLO4WnQXaC+rw9DeAxJrxW8AVxCLMaADISFwq9qZC+Vbx2 pFillrXdzGu52segu27uRncVBF/jikxo4D3KTYU4xQybc4eBIkzAzec4GD+VsNQfcdQt 00v0hJwlcwiwNf6/idBhJ294EcyFK8nxfYiUqbzBGo03JL/FPazlWDsZSX1UvmOwdqjI ojD/EKeBApj25FRbQbH2k5T9juwJ32+B7mXJkkIkusiGvW9DBuJABCaIpdl0ySZFaIsl Yn9nopeMoYemuCsqRHiz16Mr74lsbDZ185YnnCpLh8HULmRqxEMsqq28s7GJaiaxBtN/ DX2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395116; x=1733999916; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nxSp7kmSqkSnu4KEBfrma7Z74EMZDahAlv9AUZJtR6k=; b=I3n0xohsIXf4SBP3XknVai5usqwJ69l/+4S2nrnBfKM+2Qglu74CQn0x+0jTj8fyAO HadShzynG0rOOhtuUi1rtiJqMZxf5BRLGJvQfu1QxIJCQH0Y78yTqNXIw6PuzRSwQXw5 GC69BagwtHb1EwDPAl+mzJI4qcGjEHRcmZXhEKZvstR0Ib97GM6hSPMtalk01iqVr6jh uR1Dp6vynLTidhpLntTPykLAX2OYQbl/prWHH9sDYcp7ZTiXWbX9djPN4G1DbOCtjcCN 6H+tKOhb7+1DTuPdhaIHLRvV4YKLkdLsnQYHHzagj3/W6o9ngU20sxr6ghH17WXSap8+ 3o+w== X-Forwarded-Encrypted: i=1; AJvYcCU27xdMnAZgii6C/SjiYJTFady9Zs9buGz75qm9YqbgHKGcsWAgMW1i5gzDM8CH06xw9InLQwfpfVw9Bw==@lists.infradead.org X-Gm-Message-State: AOJu0Yyb/mMfE/xrhc9otFuiYwXkSZMIXGaV8aaML2w+GRqZMxTUHm6l rjyGTmGnzNgxiJEjqGAbqO9U2/iWaujt+tH7Tr4fG/xt3B4prkiKPnlFq9k6104= X-Gm-Gg: ASbGncvk8C91Ox1hpXGL1aLNZZFiqTfG8PZy92BwWYDDZvLNqB6eE+iKfvNtubGPzDF y6ENS0K0TQejullNHRX1V1Bp+W9u73ws9NzlA3lcjqgyX/9zhJYxIJ+YceDl6ZGwhxBdEmrpbiP 5W3L30wkmxqhSC+YpAo3KHFxlvd78mPJmiWrC0MPIos4upwsAqoAbe0U8yMd4wMbm4Mj9v5dcBH EbcZB8UE8Yq0ghF19MLp4toT7p1CMze/l5Cr16/uTdluVQoW6oETGdCDpZv6WbldReyHcjtaouT Y3GIXH/LVFJZJhSn18nP4/6QvEEHhTN9 X-Google-Smtp-Source: AGHT+IGAja+xIu0LL2xTFT4mUl3KDqN8KlPjYMDJCXFuXRD/rOHFiqKg6itfbSNH2M80yhGpilt+dg== X-Received: by 2002:a05:6830:4882:b0:71d:6272:862f with SMTP id 46e09a7af769-71dad6d401bmr10590081a34.22.1733395115810; Thu, 05 Dec 2024 02:38:35 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:35 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 14/21] riscv: mm: Only apply svnapot region bigger than software page Date: Thu, 5 Dec 2024 18:37:22 +0800 Message-Id: <20241205103729.14798-15-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023836_896480_3D610472 X-CRM114-Status: GOOD ( 14.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Usually, when it comes to napot pte order, we refer to the order of software page number. Thus, this commit updates the napot order calculation method. Also, we only apply svnapot pte as huge pte when its napot size is bigger than software page. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable-64.h | 21 +++++++++--- arch/riscv/include/asm/pgtable.h | 50 +++++++++++++++++++++++------ arch/riscv/mm/hugetlbpage.c | 7 ++-- 3 files changed, 61 insertions(+), 17 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 26c13484e721..fbdaad9a98dd 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -124,12 +124,23 @@ enum napot_cont_order { NAPOT_ORDER_MAX, }; +#define NAPOT_PAGE_ORDER_BASE \ + ((NAPOT_CONT_ORDER_BASE >= (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_CONT_ORDER_BASE - (PAGE_SHIFT - HW_PAGE_SHIFT)) : 1) +#define NAPOT_PAGE_ORDER_MAX \ + ((NAPOT_ORDER_MAX > (PAGE_SHIFT - HW_PAGE_SHIFT)) ? \ + (NAPOT_ORDER_MAX - (PAGE_SHIFT - HW_PAGE_SHIFT)) : \ + NAPOT_PAGE_ORDER_BASE) + #define for_each_napot_order(order) \ - for (order = NAPOT_CONT_ORDER_BASE; order < NAPOT_ORDER_MAX; order++) + for (order = NAPOT_PAGE_ORDER_BASE; \ + order < NAPOT_PAGE_ORDER_MAX; order++) #define for_each_napot_order_rev(order) \ - for (order = NAPOT_ORDER_MAX - 1; \ - order >= NAPOT_CONT_ORDER_BASE; order--) -#define napot_cont_order(val) (__builtin_ctzl((pte_val(val) >> _PAGE_PFN_SHIFT) << 1)) + for (order = NAPOT_PAGE_ORDER_MAX - 1; \ + order >= NAPOT_PAGE_ORDER_BASE; order--) +#define napot_cont_order(val) \ + (__builtin_ctzl((pte_val(val) >> _PAGE_HWPFN_SHIFT) << 1) - \ + (PAGE_SHIFT - HW_PAGE_SHIFT)) #define napot_cont_shift(order) ((order) + PAGE_SHIFT) #define napot_cont_size(order) BIT(napot_cont_shift(order)) @@ -137,7 +148,7 @@ enum napot_cont_order { #define napot_pte_num(order) BIT(order) #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) +#define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_PAGE_ORDER_BASE)) #else #define HUGE_MAX_HSTATE 2 #endif diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 0fd9bd4e0d13..07d557bc8b39 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -130,7 +130,7 @@ #include #define __page_val_to_hwpfn(_val) (((_val) & _PAGE_HW_PFN_MASK) >> _PAGE_HWPFN_SHIFT) -#define __page_val_to_pfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT) +static inline unsigned long __page_val_to_pfn(unsigned long val); #ifdef CONFIG_64BIT #include @@ -470,15 +470,42 @@ static inline unsigned long pte_napot(pte_t pte) return __pte_napot(pte_val(pte)); } -static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +static inline unsigned long __pte_mknapot(unsigned long pteval, + unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; unsigned long napot_bit = BIT(pos); - unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + unsigned long napot_mask = ~GENMASK(pos, _PAGE_HWPFN_SHIFT); + + BUG_ON(__pte_napot(pteval)); + pteval = (pteval & napot_mask) | napot_bit | _PAGE_NAPOT; - return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); + return pteval; } +#ifdef CONFIG_RISCV_USE_SW_PAGE +static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +{ + unsigned long pteval = pte_val(pte); + unsigned int i; + + pteval = __pte_mknapot(pteval, order); + for (i = 0; i < HW_PAGES_PER_PAGE; i++) + pte.ptes[i] = pteval; + + return pte; +} +#else +static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +{ + unsigned long pteval = pte_val(pte); + + pte_val(pte) = __pte_mknapot(pteval, order); + + return pte; +} +#endif /* CONFIG_RISCV_USE_SW_PAGE */ + #else static __always_inline bool has_svnapot(void) { return false; } @@ -495,15 +522,20 @@ static inline unsigned long pte_napot(pte_t pte) #endif /* CONFIG_RISCV_ISA_SVNAPOT */ -/* Yields the page frame number (PFN) of a page table entry */ -static inline unsigned long pte_pfn(pte_t pte) +static inline unsigned long __page_val_to_pfn(unsigned long pteval) { - unsigned long res = __page_val_to_pfn(pte_val(pte)); + unsigned long res = __page_val_to_hwpfn(pteval); - if (has_svnapot() && pte_napot(pte)) + if (has_svnapot() && __pte_napot(pteval)) res = res & (res - 1UL); - return res; + return hwpfn_to_pfn(res); +} + +/* Yields the page frame number (PFN) of a page table entry */ +static inline unsigned long pte_pfn(pte_t pte) +{ + return __page_val_to_pfn(pte_val(pte)); } #define pte_page(x) pfn_to_page(pte_pfn(x)) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 8896c28ec881..4286c7dea68d 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -212,7 +212,7 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) break; } } - if (order == NAPOT_ORDER_MAX) + if (order == NAPOT_PAGE_ORDER_MAX) entry = pte_mkhuge(entry); return entry; @@ -405,7 +405,8 @@ static __init int napot_hugetlbpages_init(void) unsigned long order; for_each_napot_order(order) - hugetlb_add_hstate(order); + if (napot_cont_shift(order) > PAGE_SHIFT) + hugetlb_add_hstate(order); } return 0; } @@ -426,7 +427,7 @@ static bool __hugetlb_valid_size(unsigned long size) return true; else if (IS_ENABLED(CONFIG_64BIT) && size == PUD_SIZE) return true; - else if (is_napot_size(size)) + else if (is_napot_size(size) && size > PAGE_SIZE) return true; else return false; From patchwork Thu Dec 5 10:37:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 17472E77179 for ; Thu, 5 Dec 2024 10:45:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uT1B+2crqGJQpHXcUvS+aZRdkURk+zTdLberkzyCYlI=; b=0hsrlB7VxccOt1 kpl+x2v5YuM0hpWjNoA3o+oIELGYhDtnvAJ/rBCGdK3CH+Z3mGvcTT07b6bHLudrUOfpbV5BhmzVu uZwCLtv9cCpo8Advs2c/Sx8rs7IExRcOae/JOuQVJ0gGCfPJfDECRktHVAdB7MgpsPkdp0ZgR77jo Ieeyx6CayCOdFWEWHPj9Xdm/g7z70mspvaV4FobZrsZZdScZ/dj1/Msbq3BornCwRX3IX6TWymvPO ebuW6GDP+92AZWUKpsXs3i6H9paj4y+G+VmS6TdH6Bj3njYcXzKfdYviMc4Cew92nbnZy70klnENK hL31dzSLRzh9BAHIxdWg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Lx-0000000FaY8-1UNZ; Thu, 05 Dec 2024 10:45:01 +0000 Received: from mail-il1-x12f.google.com ([2607:f8b0:4864:20::12f]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Fo-0000000FYJo-41uu for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:42 +0000 Received: by mail-il1-x12f.google.com with SMTP id e9e14a558f8ab-3a74d9aeb38so2661175ab.1 for ; Thu, 05 Dec 2024 02:38:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395120; x=1733999920; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uQ/0Scy9E/wZ7/MnA5BYlbNUXDy/8A/TpwTZAMtapEc=; b=FH0A+dteoegbLPMy9M5NQkRlc/gp/62vrdVkaXWlXOine5hccCut39ZJ7pFIRw87MQ d6X3Lg3Fo0H6fBYN3eJ+oVDcJKmbp8MjeI606rrdaJfT2bXVAZtUbx2C/3uTJgGpLIuf JHcYXxcz6RZxelb04VEgo6d/osS3F70v0a96xTkPShyfeXOH0zLUmp0aXag8Z/h3BVV7 clORTddaMMxHQyCVVC4cNttq0VUZ9/OnutOVq0NnQ9p9SHEI/TS2jExXgcBd6py2xzky h5fqqL8WBIx4uzQOBOcPVEyL+86IVCHYoSMyD0yUEr0/HbVXQPpZAc+lKjHdkGt05/yO pugQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395120; x=1733999920; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uQ/0Scy9E/wZ7/MnA5BYlbNUXDy/8A/TpwTZAMtapEc=; b=OhR+OaiuaMfp1QW5ghR2Q3SoTEpnAO3Xmj0gX4wc4kxzWSQxgb1UenR/2FDocf0IU6 Fmxqpibfhh6mHjb0bqKfvCHk8PXDju8LfbXnyqf4utbvrnfxo1K8J62Aoo+XdF/t7sKZ XILbeWXWN0x2yOIxtgIpblFbpVtyUrhBa5Uejo8mdSypd53VbBkn1iqVzCU57wpNYOHc jDtGBUzCbXmQnitqovDs0HCYTzN1msH0tAmiSZ6UxRxms1dsjDJOaTwnfLW62+8gkSxs 9NZVxn33YQFmgwZxgNc3yLJVUAJo2KcFRZRbSR2llcIZI1PAET+zguucp7wr5dvgxtJa dmVg== X-Forwarded-Encrypted: i=1; AJvYcCVJwGtEahUWOCrOVWhzE15TwOstmtsrL7W+1g+4RryexIcZgFhhlcD/bTIeZZtqJd1NIKhGTU+LedKdMQ==@lists.infradead.org X-Gm-Message-State: AOJu0YxSJFNkaaAVsFiyH8QgHpURg598ZQlZpPt5dR+ZbB41GzpO1vWB 0EAVfT/IYKT9dPvMZXpRaInn/CSLfDaE0zJ1PN48vffV1365R7KRUQLw6wOGI74= X-Gm-Gg: ASbGnct7O0r5kit/X0QJlWh8OOC7q5zmZljzx5H+U6jDW4V4/YyyNqFaNJo15Nk/1G0 rRdy1f/oPHpac4gUpE1LPhG2blZ7Xcitk5ZEUu0TI3j0KKV8vt6sY3ZQzcigT6+OCmJ8IhP480V 5E3d7BQbwAdiaT7XfjfGgYYKaHhL7R/JpSZyuJgtjEqKLdcjZb5JiYjFgPnu4Y5KmXIZqsD0hcW ivNQSMwcmKfttXv7JEyxNs9B/VM/42UU9ux7xD9Zm/u3SJM3FeLPbMQkXfldqpA04mDcb3G8QoD f5piwiG8upNIfQfo0f2fLiOmeOToadPe X-Google-Smtp-Source: AGHT+IE8l6yfJ87eJad5FAeGJqDd+iC1OcZn198mK7ZYZDH6TLk8OTxSkqSKrMniNwvDITNae2Qz7w== X-Received: by 2002:a05:6e02:144c:b0:3a7:d792:d6ad with SMTP id e9e14a558f8ab-3a7f9ab0ff1mr115272635ab.22.1733395119846; Thu, 05 Dec 2024 02:38:39 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:39 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 15/21] riscv: mm: Adjust FIX_BTMAPS_SLOTS for variable PAGE_SIZE Date: Thu, 5 Dec 2024 18:37:23 +0800 Message-Id: <20241205103729.14798-16-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023841_004646_9F558127 X-CRM114-Status: UNSURE ( 6.88 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Signed-off-by: Xu Lu --- arch/riscv/include/asm/fixmap.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/fixmap.h b/arch/riscv/include/asm/fixmap.h index 0a55099bb734..17bf31334bd5 100644 --- a/arch/riscv/include/asm/fixmap.h +++ b/arch/riscv/include/asm/fixmap.h @@ -44,7 +44,8 @@ enum fixed_addresses { * before ioremap() is functional. */ #define NR_FIX_BTMAPS (SZ_256K / PAGE_SIZE) -#define FIX_BTMAPS_SLOTS 7 +#define FIX_BTMAPS_SIZE (FIXADDR_SIZE - ((FIX_BTMAP_END + 1) << PAGE_SHIFT)) +#define FIX_BTMAPS_SLOTS (FIX_BTMAPS_SIZE / SZ_256K) #define TOTAL_FIX_BTMAPS (NR_FIX_BTMAPS * FIX_BTMAPS_SLOTS) FIX_BTMAP_END = __end_of_permanent_fixed_addresses, From patchwork Thu Dec 5 10:37:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF03EE7716C for ; Thu, 5 Dec 2024 10:45:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=j6e4MgXq860DImd3JMeoSHkRp/k3eOuQzm8bsN1dUKM=; b=X5SMrBZS4FKq5h nv3S2VB19SHae0LbUsPfzNnJSDKDBWeZOcwSDf/ZWSC+I76Z1/LfRYa9Aw38TJaqZm2cEFSbINCkb IhDhr18vHTPGZvsdcesXzLdpDJbUhaAwfyOQU6HKBE+xKYzODLSeJ1xLgy+xu1laxV5Kx+v1AgTsI Pp9OJVOZgYdv/mvw4m08k1p8w+3Z5QQFUe2VYOBhTG7EXgXh9qZbps76TjzHdsYiw0pzwvNpzMlLc rgqvbZjnKS0ptw7J71pC4Uxt/w2te+jZaSAb3rgWS1JsWu8ZDHqJij//9EtBv7htaKdqfDDBTwN/1 +fa564NrBuXn8kRDp9XA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Ly-0000000Faau-3dWj; Thu, 05 Dec 2024 10:45:02 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Fs-0000000FYM9-3oDe for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:46 +0000 Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-7fc8f0598cdso1485877a12.1 for ; Thu, 05 Dec 2024 02:38:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395124; x=1733999924; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wAxAI+FpEZPqeOtAYXtXd/fFddvtrgZvKhcRJUP9sTk=; b=Q7CU+EjjulTR4PWLbglsOVEvq1/05ZnqwzTHqvvP8vzBIndc8NFGUtkNtIS2oDhNOS F/uqhzCFmwcxvXERm+ZpJD+bXHiCVGmiDrpovxemAIAksrefXAhYHiL/PREGHek14+zv uAWVpC9PfVRyyH0heYB96mM76BDCzA8GhTeEMfPQrLfoeUHBWScG1Gisce+X8hyvNIRQ 5p5QPlrOe/TpcIZeKtidrWKsaRAmYgNA+5U9h90b/shn1wA3CLW3ctkkzoCmToDjZ8rn gHN2JBPQTvlBtIB/WyBFR4JmKaUDrhqo756wFOESi8lAk4BQZ3H/vlNarTcHEZOpvA4U nlBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395124; x=1733999924; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wAxAI+FpEZPqeOtAYXtXd/fFddvtrgZvKhcRJUP9sTk=; b=p1OuUINbp6Urqf5SBPvfWOUb4WkCp2XPunT+rVu7yDHHcmMT+lCdCqEeItLNEmmyl9 Wmz3nyEUlwAs0LB6d0DRyU1uNgMG4eW9vJMipFfBIIWVf7phu8G5BsC71dIS5ekS2k/N 5pOTKBHATdGmXXo4GgQkHtrvMtnwvXOvYCPdcuhO3t5wveYG9F6PXKbTaPRW1ymG8w7o KLogoIxt3FlLJWAGgBVZ8Bd7aa25UVMlzbONzgUL/cWKrCbsgDfgF3uHYLrxpti/Jk3O pvhFFRVfz1pu2vs4lGQg9Rq45z4rD+igOxDGkF6hmIohzF6Q5GQFPypZoXW1Zu7nt492 qrZw== X-Forwarded-Encrypted: i=1; AJvYcCUkhYA/rrn+nbYqZkniw89xcKGIiN8GgUVuUR0jHtHdd11Hn1Fdt/1o951DSP3JNsGzjQ+p15Ks+OWh0w==@lists.infradead.org X-Gm-Message-State: AOJu0YzAbCCGjHfHZHQHO22xGd9XkWOr3aPkz4b63lyTXI1ovoMdsEjJ NivQYEAtSXT8y7qh6TqsHrOTiC922LRbmbor/WRIGTPCWqq9WbVhKaKEWvFiMzo= X-Gm-Gg: ASbGnctx0Edea7TlVZNqog12zdymkkuChmuCjyeLA6/rV7jtOPdkq2hig2l3he3fl0L Eo+vAiE+51Jl7c9c6fmqpo4x6Ud3n4gOAg4eNkw9X1JnPvANSAe2rVFwTNlzQPwMj94fwq13tOt Xtpxej8xeA/la7/R+H19zA1FcUDmrZ48+GfNOHiO1Dl1bqSA8y5unpEgbw4P36oQK27J5WQcRbX b22rdA2fHRlF47rxEYUdCyX10lNCangZBLhd5AXyz48E6+IAaGFbo6v6M/m5QErZNVGF16e/qB0 o+wqDvl39I8tVNCraQWDFCDSBpGvgKDN X-Google-Smtp-Source: AGHT+IE8309GuNeyZxITDQ1e0WzOIa0uJDO0sUbGSJz89pLUWjkP1vS9Smp96TCW7bxn3tkgM30nOA== X-Received: by 2002:a05:6a20:a128:b0:1e0:ea70:7494 with SMTP id adf61e73a8af0-1e17d38c18dmr4437406637.9.1733395123986; Thu, 05 Dec 2024 02:38:43 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:43 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 16/21] riscv: mm: Adjust FIX_FDT_SIZE for variable PMD_SIZE Date: Thu, 5 Dec 2024 18:37:24 +0800 Message-Id: <20241205103729.14798-17-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023844_946558_0A67DA6B X-CRM114-Status: UNSURE ( 8.07 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 07d557bc8b39..5b2ca92ad833 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -108,10 +108,10 @@ #define PCI_IO_END VMEMMAP_START #define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) -#define FIXADDR_TOP PCI_IO_START +#define FIXADDR_TOP (PCI_IO_START & PMD_MASK) #ifdef CONFIG_64BIT #define MAX_FDT_SIZE PMD_SIZE -#define FIX_FDT_SIZE (MAX_FDT_SIZE + SZ_2M) +#define FIX_FDT_SIZE (MAX_FDT_SIZE + PMD_SIZE) #define FIXADDR_SIZE (PMD_SIZE + FIX_FDT_SIZE) #else #define MAX_FDT_SIZE PGDIR_SIZE From patchwork Thu Dec 5 10:37:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C6D75E77173 for ; Thu, 5 Dec 2024 10:45:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HMzK+1kg6raLPixBMFACVQTKQH6cG8dccCsAUDTtfpQ=; b=h8Y8p8kaL0f5er lrRWVJn2X34FC85xVe4cdSbqrHBLSRnNJIJzvaK66xTi+/Ese8l4BS+2T2SWqzcDqh4YnBj7x3vcz 9D/DWtn2WYT90l7rmIKPAblf8hBqhPK9xxE8Cz33mdGV/yh/T9EZ829pHvSWkflfcdV1vAPNTIeVe V929O698m6gKrHAGDoBliU0QPEnpkcg4rhfH6LaMzCW6QzixEM4hE8qYtBG1R9KhYPOuxuqOQbNRD X9nAN5t08rbaHJ6zEwiwp5rBsX9n8Svt9+nH7fsCaDa03ULBxynfFH35uVIi93OpyC7aDh8NwkZaw 34T0WrVAPJuXmOE9HSkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9M0-0000000Fadu-1dS0; Thu, 05 Dec 2024 10:45:04 +0000 Received: from mail-oi1-x22c.google.com ([2607:f8b0:4864:20::22c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9Fx-0000000FYOL-0vaE for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:50 +0000 Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3ea411ef5a9so479350b6e.0 for ; Thu, 05 Dec 2024 02:38:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395128; x=1733999928; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x95dczg6H/IuNQfp1VlRQgQwszamKCGOevCsmbhARnA=; b=j6+8x9Qmw+ZXtap2yXGivdLBlQMPrtTi/yNA3I2mwPEMs/9Z4RO0CGZa9v8Jmzl4ra LtJtbRxj35v72m+TMCnFCCyDhjupi6M7HtY4qRACp39brv1Orvhh/8nw6VGOSG1A6I2j VEXdoYlQMS/NCKKPWH2FUxIfZu2vKAwQOD7pzOr2sWnLUofshU7GFsq6YcPpbSqIr5Kp lhQZtYXidPSUfC5BNfZ9ThertS3dCMKwXXzW5qeojZist8CoC2QMi8JO0c/FduiD2tuw TPWrwq5NEwokIjwkLJsI/O5o60DZ2/amn+h/ObibZZcfhGijuWx2Ta9kMlzdt8karhJn 2AQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395128; x=1733999928; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x95dczg6H/IuNQfp1VlRQgQwszamKCGOevCsmbhARnA=; b=Pfg9HeimaaDYF/UwgkKHLZ7j0NqffxREc6D2r+A2IQVsZ84+xIpebBzpRZ2eOSFYpo 7xAwvizPTUIxm9UNASIYWPXsqkJVH97eO3QE21qClQGo/GGRciNHanPEdcXI2AyI/Vui rmKW5oB38L/rHh6N9YXDHB1171RrBHtTWQVKYaB1m67m37F3G8c2uO6pBtHpAGhVtE/2 PgYRjOZAQTXOFwRxSk6xzrb0wa/8ovLXiQ9H7s9enOg1c6wUbSz4UY9D55Hr4x/wXMW4 FL/muT3QlS2FD3QaZhORnFsQF8Ki/i7bHWraaXhtRJajrlDc8Swv0Z05RMrkD+8zxM9+ yglw== X-Forwarded-Encrypted: i=1; AJvYcCWlLEoj3UfSmjWDhBWuFu74Pn+l43L9kCjK5f/xVyzf2IVuEI+aqxkRf9gLt29kVrJh7tle94ZzPsmrzg==@lists.infradead.org X-Gm-Message-State: AOJu0Yw20Dhw4HEJchnnUNFjrWiejAwtYrEzBmFcrEzKROIMAjETgx+/ rWx44pgiaIkVeejl0KaommwUGxEUCKb7CUD+2s5QpMy2FjgpeAvtqMxyTm72H+Y= X-Gm-Gg: ASbGnctRsXKGhjCF82BC2X+Cmik8k+7eYiQm3TSiyodgqW1aRY1MxmvN5sdNTS3pOZn mtpSuYhttACyGz0Ta8Ha66h8BAGwoudUaQciI3gvg3UfPUDEgRbM+KV7G9ZnmNdNms1L6riVeEr xDU128N+hJIXkKsf8C7nX1ebdDkjc3/HDnp8Kt8tXf3FMvLLDrrcB3Zc1oRUXLlcwiQlejIsHm3 xG8Bt6+Y00DtnfFnvM5/LNNMCl+PL6TCDQZvq8UBDmPCSndTJp23VyJSYht9dJBYUNE+THXnNW8 c21oT6bb2glsX/3CeR8u7JNMFeiZQvVE X-Google-Smtp-Source: AGHT+IFzuODYxPZb1b0Hsra5aByVGtGgW30pbNXl6TXKG3vo0iIQP7jLbPRNzxf3m6EfJtUU2ofomQ== X-Received: by 2002:a05:6808:bcd:b0:3ea:49a1:cba7 with SMTP id 5614622812f47-3eae50c5186mr8718922b6e.41.1733395128040; Thu, 05 Dec 2024 02:38:48 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:47 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 17/21] riscv: mm: Apply Svnapot for base page mapping if possible Date: Thu, 5 Dec 2024 18:37:25 +0800 Message-Id: <20241205103729.14798-18-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023849_256421_81DD89A3 X-CRM114-Status: GOOD ( 12.65 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org All hardware pages in the same software page point to the same contiguous memory region (the region size is equal to the software page size) and has same prots. Thus this commit uses Svnapot extension to optimize the mapping to software page to reduce tlb pressure. Signed-off-by: Xu Lu --- arch/riscv/include/asm/pgtable.h | 16 +++++++++++++++- arch/riscv/mm/pgtable.c | 6 ++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 5b2ca92ad833..9f347e5eefeb 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -483,13 +483,27 @@ static inline unsigned long __pte_mknapot(unsigned long pteval, return pteval; } +static inline unsigned long __pte_denapot(unsigned long pteval) +{ + unsigned long prot_mask = ~(_PAGE_HW_PFN_MASK | _PAGE_NAPOT); + unsigned long res; + + if (!__pte_napot(pteval)) + return pteval; + res = __page_val_to_hwpfn(pteval); + res = res & (res - 1UL); + pteval = (res << _PAGE_HWPFN_SHIFT) | (pteval & prot_mask); + + return pteval; +} + #ifdef CONFIG_RISCV_USE_SW_PAGE static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { unsigned long pteval = pte_val(pte); unsigned int i; - pteval = __pte_mknapot(pteval, order); + pteval = __pte_denapot(pteval); for (i = 0; i < HW_PAGES_PER_PAGE; i++) pte.ptes[i] = pteval; diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 150aea8e2d7a..0bcaffe798d5 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -11,6 +11,12 @@ pte_t __pte(unsigned long pteval) { pte_t pte; unsigned int i; + unsigned int order; + + if (has_svnapot() && __pte_present(pteval) && !__pte_napot(pteval)) + for_each_napot_order(order) + if (napot_cont_shift(order) == PAGE_SHIFT) + pteval = __pte_mknapot(pteval, order); for (i = 0; i < HW_PAGES_PER_PAGE; i++) { pte.ptes[i] = pteval; From patchwork Thu Dec 5 10:37:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E7A0E77170 for ; Thu, 5 Dec 2024 10:45:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q5EaP7/K1wkDb6/G48sRR+TuuxTYmjZzHE3x3TeAPK4=; b=EfG9VpDwC4++7a I2iWg7gGiLtHfHFzLG+TNG1KOUovgl7sxhqagopJbw3PQbqBwq3s5FFd07aDb4w02uQWTLSduCpMG WZ9i6c/uqwNVHA1qgnYabjpsM6f0wZ+h46WVgfn+Ncw4GWt+FLnjmBcyQk4lnQmdlCsJyCdHFrLM8 9fexQRERJh13nQsV08HO823FDC4gavz5a53zLIJ+4EY4WQSHU/fnz5CUw18tExb0THb+v1PdFSmYm ZfV/8tbo0ycAaftm6dUKH0KdcbSPKmf2ferOzj+S5AhOW3AnwPbojlU0bZ6Gn6m97HifOYuBaw5PF Buy7SiFuVRbmJk/t2p8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9M1-0000000Fagq-3n8g; Thu, 05 Dec 2024 10:45:05 +0000 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9G0-0000000FYPq-3tU9 for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:54 +0000 Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-7fcc00285a9so834647a12.2 for ; Thu, 05 Dec 2024 02:38:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395132; x=1733999932; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6fWC9OQ8fDv8IwsAqd/wbk191tCVh4tSTxqplFbmaYk=; b=CKN0SMZ7Kx78PrJw1bMMomfDIPq0CqjvdYbzF289ETyDLd2I8NFUHjYlGztX2X1o7R SUSDxm4VVkbvswhuR9TtRH+DVU9Vo2B0ZGqFzNNZX3VSbo+7ijhaIWfcbThFxmRztbLd Wyw09zJ6BaHGItBgIhF1iwL9rvxQfXmYpS5o2rUWBc/IcfiLZxwqYX14pl1CNEkBs9kU 2RypXiCuiy/taRZFS4Kr1C5OobGuoIevgppG5uhA9C/AICF6STNGy4JNBD0M9VKmBYF4 g6Xlsmf9jUTjcvwkTUcatQEAfDE6b6+TfZNu5b3Gd0x/5Fpg28dfmt2NUw5L/RWXFAHN UlKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395132; x=1733999932; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6fWC9OQ8fDv8IwsAqd/wbk191tCVh4tSTxqplFbmaYk=; b=IRWq62m1ue5Xl1dv35IGcIAdvnHexZENi17wcbEHckidOwwJeYG/l5sQxsa7k4HRek 7yLxiIOjkd5/sBeK8qsMT5vlX+NOYbd+676SKXXpwEkydTefMJ7HvXmCfmYWsbY1gpxM Thq0KkmGr4/RuuJwhp2GmBDTyTHKOW7K23BGelIYCLYy9Qg9DnlW+8PiDVHqp9AZsSd8 V1fXCNXNZCNIh3WQ+j01LVuaWm6jfFjWeHGWN6dXjwsWDX33Qzq88yoNfxKMFRr2pb5v +uHTUH+wbE+7uLooVRz9becO1qBza4xtSZBWX8B2y9bhSBguwy10J+6lNLndAfPh7HJh GcqA== X-Forwarded-Encrypted: i=1; AJvYcCWLZgxdXhPFja0OUF72JqrsXScN2DJyhPcOTAVJrbvzl2RhXHYh24QXIYOx1HxBPfAHS1T4ZP+QHEJBwQ==@lists.infradead.org X-Gm-Message-State: AOJu0YwH/nGDA0U+Xqil8gy0NED/ngWxtg+D0O/Dz9JFCs7VdvaDIQJL GFawmvdxj3jfP7//aJnLCogqsMGy1arFYcTpcFraWnt8HuITWyCn2hddrFg+dic= X-Gm-Gg: ASbGncviorIeTn13XuAkJw7IiKKNnB2srOz50V1ahjuwnqWKZpYGwiFrkrb2wiwiY5w PxgTLuhohyr5/qI2ai3MhsPtUpV5GErQ0gy1bVmV1eP2QODfmiIdlcWHZFccCDs4s6RI+9egX3x +wPVuWkKQxyDOC2vxyrw+eOu52ZFOnse/+7sKHHwfOm/6ziKg3h7FZO/VCXbikatQ729HK008Jl PbO1NB6Q+qJrLaU0RdH0sm3v54RymTAe0fYqyQP+yJ1qT9L0E/JhtYMk6XdYP78JVke/XSTqWq8 hTnALF3yegFOjRLcQ5tj2vP4Vs7/nD0g X-Google-Smtp-Source: AGHT+IEXD1d9VHmdEfHzDn4iIzGpKroyoIRBm9V8GpOCRUVEQI3tEWlF/wbQq82AYL+CFEewTt8KjA== X-Received: by 2002:a05:6a20:7350:b0:1db:ff9d:1560 with SMTP id adf61e73a8af0-1e1653bb992mr16055868637.18.1733395132079; Thu, 05 Dec 2024 02:38:52 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:51 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 18/21] riscv: Kconfig: Introduce 64K page size Date: Thu, 5 Dec 2024 18:37:26 +0800 Message-Id: <20241205103729.14798-19-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023852_965711_3B0B365A X-CRM114-Status: UNSURE ( 8.10 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch introduces new config to control whether enabling the 64K base page feature on RISC-V. The 64K config will set software page size as 64K and automatically use svnapot to accelerate basic page mapping. Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 2c0cb175a92a..592eb5766029 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -167,7 +167,6 @@ config RISCV select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD select HAVE_MOVE_PMD select HAVE_MOVE_PUD - select HAVE_PAGE_SIZE_4KB select HAVE_PCI select HAVE_PERF_EVENTS select HAVE_PERF_REGS @@ -885,6 +884,28 @@ config RISCV_BOOT_SPINWAIT If unsure what to do here, say N. +choice + prompt "Page size" + default RISCV_4K_PAGES + help + Page size (translation granule) configuration. + +config RISCV_4K_PAGES + bool "4KB" + select HAVE_PAGE_SIZE_4KB + help + This feature enables 4KB pages support. + +config RISCV_64K_PAGES + bool "64KB" + depends on RISCV_ISA_SVNAPOT && 64BIT + select HAVE_PAGE_SIZE_64KB + select RISCV_USE_SW_PAGE + help + This feature enables 64KB pages support. + +endchoice + config ARCH_SUPPORTS_KEXEC def_bool y From patchwork Thu Dec 5 10:37:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D968DE77175 for ; Thu, 5 Dec 2024 10:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4Y9/fLh37lv3hXSep5yvmnOkTbtgJpAL5Gh/BQtb95I=; b=cXq9zYgEz06dNi jbQ9UdZx4rjw8dN0f42OC7HdpEGRcr6UcW0akORMwP+gFI34QcHkFqZ31q0sRCIAlaKO0RLY8tKzZ AgiT51IpN7dfSFnwr5AZlILxcx4ygr8vDVLkqKKkL29fCzvxQi6h2OC2J+bmXkmk7+udqAbwQ5GM7 UeZWPDKtXYUEWpbkIe27+RQAQRN85XVGLl5ElefWVDshaXnlXwPXg8U3DwPJ89+Xml0NE5SzFvwut KviQY/I5GOKDKEdf0x9UpVNGyc4lUxs+mvfK5lgxJsbN5PTpXWnE+ILPf07Y/C1CqNxzFTn4tSOXS wJISiaW2mZs4325tAcVw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9M2-0000000Faiu-3jRl; Thu, 05 Dec 2024 10:45:06 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9G4-0000000FYRJ-2hgj for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:38:57 +0000 Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-723f37dd76cso684749b3a.0 for ; Thu, 05 Dec 2024 02:38:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395136; x=1733999936; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VfDjQEoTxdpd9iDlK1ps7bFltPFhkJicFLVtDzgiM9U=; b=bOmv9/Re517JUb59ziGDBgPZV15rgzsI1DhD/pj3CDqaJi2s5jPyQltyUfcJSva+VN gj6wT/SQsrR9hld2dHAC4ltpnS7wm/dZQw7a4hZn+jnLtr8Y03y5+4TNcPzy+M0pHWDa iVOGFsEK4jrO5/JvpSrBz0v6XlH10UfoUKxQ3OWfGg+kIBi4jOZr4SNdCT3LffizwizI 3cXxVpQnY5SNfCc81qiL0gHXtfdmHiEXzmE5rQSsu2d5HWyGvyxFlFx7pexM9i71BFYb tG23ExU2+D9ElPjmAjEAR7+0hp0UBF6X11OE1y/6Xqy0UIEmsA8/a2U7RNlVYsrmVLCW vZRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395136; x=1733999936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VfDjQEoTxdpd9iDlK1ps7bFltPFhkJicFLVtDzgiM9U=; b=jYs4mwsdIWelEw5w7/uPywlrrjXyTVowzr1zlrpSkyggaP0shBz3O7fi4fTIW6ojhE uPGZ4Pn29IZCM8WK/5+OdlL2ioW3+SDz88FlSdMnVYULJOrm346RIQRiWU91jiCUiISQ nwBrC9nykPFshBoXIIGAk6C0oZjccMGN0JcrjTX9AgstgV7m1ZAQ6V8eKC4WGvEYsLEz Kj0Own+KOSGmJi4X6B6xfTu45pjorl6lNEq9utSdMbwAGXTNEPuYvl7Wkq4P0PNTrId8 Q5hyoS9Apk4Xh5OUp5qMP7k9+PlF3KZwQSXx9CF47250eDBXYERr2N/T+C8HvCtMCCpE JOMw== X-Forwarded-Encrypted: i=1; AJvYcCWnj+Zv3h1VArtCAfVx0wI7okQeEygk6270Lhw/3oM7ofDClW/PthIiNhnSjxJWsPpPAYiXWVCX2G1/3g==@lists.infradead.org X-Gm-Message-State: AOJu0Yx97DIKnghKoh8qO2UUhGb8yTxrfL/2jgESfB37Grc0OckW6ikE /g+0y2ZlV45fo8FNMMyrXzih6mkRDGWVSaBq4kXijmCneBrkZFgH+bPk/FXgUx8= X-Gm-Gg: ASbGncvbN6Qqrlo1EoghAaPHRnnpV66i47OSHHhzd6tjm/5wFEiefQMyMzYS1RV+Rzy PwfwUCG7PhMZNAhCxiC0C1+L6I9/mQ6PhBIwX3SOGkl4gDLQfx010I0YUsr7a37Vo8QXyn6QhqX Comy8Ae6i5oJrEQrSeP+JhyZRBH6pOOGyS4SamWbKyQATKY4ejjnCdSFriQQjQvvaq9bwajme7Z 0ZVbMHmjYZI4oLvx1ZOFnLJnV2hvN02bigKcTRVPo1Ku+EDQzccm5yqUaGigXWlpz7Uj5CQ6v72 o3xHgdzhoHueelNnOd1BFO/AgwlBehjQ X-Google-Smtp-Source: AGHT+IHmLj1p53xhXYUvBEpMrjDW3Wg1vBwEcbP7EM0jf/sz5snayXz0eteUjpLhTi1nk0wWuAYigw== X-Received: by 2002:a05:6a00:1703:b0:725:3bd4:9b52 with SMTP id d2e1a72fcca58-7257fa5a308mr13643770b3a.3.1733395136132; Thu, 05 Dec 2024 02:38:56 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:55 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 19/21] riscv: Kconfig: Adjust mmap rnd bits for 64K Page Date: Thu, 5 Dec 2024 18:37:27 +0800 Message-Id: <20241205103729.14798-20-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023856_683373_557841C5 X-CRM114-Status: UNSURE ( 5.80 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Signed-off-by: Xu Lu --- arch/riscv/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 592eb5766029..9dfe95a12890 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -253,6 +253,7 @@ config ARCH_MMAP_RND_COMPAT_BITS_MIN # max bits determined by the following formula: # VA_BITS - PAGE_SHIFT - 3 config ARCH_MMAP_RND_BITS_MAX + default 20 if 64BIT && RISCV_64K_PAGES # SV39 based default 24 if 64BIT # SV39 based default 17 From patchwork Thu Dec 5 10:37:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2D9C4E7716C for ; Thu, 5 Dec 2024 10:45:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=k6gwxxFJCUUFcBwPmDGlyt9urE02ylYcHGUrtfabC3o=; b=m1z974OSvtGFZh ANku8t9DdM57Hp/BrfoJLEljepMISTGzLWPGOo704IxGGuV7TmNYHsUl4Ki93oDoDvs9ylCLr5NGY M+P1Vp8OvQg3roF8jyNkrJSNBoiP9UoTJwqTSOG2Uw8jzDxAyPBWI8nvJDLorOX5WWQBGdkd6BMkv iKvsdcCXd/ZASaz7TEBe0zvw1DJRSW6+dDRzhupnXf9px6AKaXgntM3AMzmvLItB8Ux0s4G+GoxBK CwasgTxsboSnx8ao/zCSUQ2oObZx6FT2XLNHfmyIWPOMpxc+7zGHaSzvNjkCsbIhtFkis5mIW1LWB u68Cmey4JnVrcEYKaqhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9M3-0000000FakS-3NDI; Thu, 05 Dec 2024 10:45:07 +0000 Received: from mail-pg1-x532.google.com ([2607:f8b0:4864:20::532]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9G8-0000000FYT9-393W for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:39:01 +0000 Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-7ee51d9ae30so516912a12.1 for ; Thu, 05 Dec 2024 02:39:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395140; x=1733999940; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0HKGaUSlXgW0yAKeFS5JtDgO8P73Aga3Eke367yRUOM=; b=h7wMj4ZprgLG5Y2UyryzuHrT8uVQe6PY2wgeYifmwozN8175dSHsRLFo6jsQW6uIR0 f1XcC9sy167mdTd/yRytdNNcjz11lX7qQEH0BznCkpB+rO8jj4OPWqI2P4EuhVO+Figb TM/4xgKTWVYSk/EL8v9vax+xMhq57r5Lu+YqheP1rVYVVp82DNTlK7Lv1cF2zrcdtOZy OpMFvg6WmWErPWEUzRp0vcMpFD1hExYgXEF1mDT1pfKQCIJ84866AyQ1nOWC1SeAET5H fSJRoqHKhnv3ckJs3WEbbfa38Q27hlkvxTZcqTKYIUMs89nip808fmLMaldeCXy36yuF znvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395140; x=1733999940; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0HKGaUSlXgW0yAKeFS5JtDgO8P73Aga3Eke367yRUOM=; b=rHCn0+DviaHm4abesaVjTKjkVmrIojKsAbqDKUg3Lj2c36ZYR7rFX73+o6ED0pj95t +T/wd0k3MZiyD+33LZlMwO5Lr/jgw+EqR3nZv6c5Y8FlqI5suvOsFUkYxchBe0qAy2tq y9tJldTmAFN38KQ2fyr5najcWVz0gf5EEyukUh19N649tbMVShVbRPq6BKgmyOgQUEOY vnI1akYGnXdvmDDnqdw2yJXTSb1WPjiyIiEYgGGkhcJTRjuEFKdCviqtJ/0KRbGdUaxf qSIeEcjkzfib78N5g8O4lMlZiuJvIfA/WLikcKgO9vZmm3Svd1ZI4GSqktMavNQA+6uZ Xhig== X-Forwarded-Encrypted: i=1; AJvYcCXkKGkvwowlSz05/VIJ76kkO1RczBnNzBJ2NaMrgnnC6Ot/C0RBBc9lGAkgrAqG5VwpeX0DMaL/4zyPAw==@lists.infradead.org X-Gm-Message-State: AOJu0Yx5gDRk6yW3a1UQ7FU0jJc1DwLvfOH9YVkxT1QoNIT07BZFuyn9 IRT2Rk+j774Pos4het2HU6+FljPKBmnQbGuuY6XvFZ730uARERnBRo5/nAfgkPY= X-Gm-Gg: ASbGnctTv07pC23n9eXs3m+21SUpvWDjRabYm4fB0xwe7IvYF53w2iTByIwfh6UYBy6 E1qYihMRnZDu7037QYQqCdZMZlzbIJB4RSX9GhVftbYDdsleOp1M/nPQ3qk9bba9JUTYYuOnR/O iy8uUm5fyWtYzb+K0nq+iyHL5a7ZLfmS/GHrNtdFsYMeFy6QuNdo8HmjM7clgJ5MyOaB98jKCEb iFFh3IiuHj18NXpsAQ5inXecK/gg26SdWAS8cYpB8Y1ftg9wqxAJExlxINyt/y34FDhHE5n/Kx2 DmQCVsduB5T69A6EQkywVmcKoeFSn5PQ X-Google-Smtp-Source: AGHT+IENYN2wIy2LzaSrsQXEce8aC4H6FioafWLUrw0utp3p27SZ8r2YKo2I/DuxTOOKoa8IMLVh9Q== X-Received: by 2002:a05:6a21:78a6:b0:1e0:d60c:3f49 with SMTP id adf61e73a8af0-1e165413484mr13667504637.43.1733395140137; Thu, 05 Dec 2024 02:39:00 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.38.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:38:59 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 20/21] riscv: mm: Adjust address space layout and init page table for 64K Page Date: Thu, 5 Dec 2024 18:37:28 +0800 Message-Id: <20241205103729.14798-21-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023900_786006_98EABFCF X-CRM114-Status: UNSURE ( 9.57 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Signed-off-by: Xu Lu --- arch/riscv/include/asm/page.h | 6 +++++- arch/riscv/include/asm/pgtable.h | 12 +++++++++++ arch/riscv/mm/init.c | 36 +++++++++++++++++++++++--------- 3 files changed, 43 insertions(+), 11 deletions(-) diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 9bc908d94c7a..236b0106a1c9 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -40,8 +40,12 @@ * By default, CONFIG_PAGE_OFFSET value corresponds to SV57 address space so * define the PAGE_OFFSET value for SV48 and SV39. */ +#ifdef CONFIG_RISCV_64K_PAGES +#define PAGE_OFFSET_L4 _AC(0xffffa80000000000, UL) +#else #define PAGE_OFFSET_L4 _AC(0xffffaf8000000000, UL) -#define PAGE_OFFSET_L3 _AC(0xffffffd600000000, UL) +#endif /* CONFIG_RISCV_64K_PAGES */ +#define PAGE_OFFSET_L3 _AC(0xffffffd800000000, UL) #else #define PAGE_OFFSET _AC(CONFIG_PAGE_OFFSET, UL) #endif /* CONFIG_64BIT */ diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9f347e5eefeb..fbc397c4e1c8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -147,6 +147,18 @@ static inline unsigned long __page_val_to_pfn(unsigned long val); #include #endif /* CONFIG_64BIT */ +#define __PMD_SHIFT (PMD_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __PMD_SIZE (_AC(1, UL) << __PMD_SHIFT) + +#define __PUD_SHIFT (PUD_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __PUD_SIZE (_AC(1, UL) << __PUD_SHIFT) + +#define __P4D_SHIFT (P4D_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __P4D_SIZE (_AC(1, UL) << __P4D_SHIFT) + +#define __PGD_SHIFT (PGD_SHIFT - (PAGE_SHIFT - HW_PAGE_SHIFT)) +#define __PGD_SIZE (_AC(1, UL) << __PGD_SHIFT) + #include #ifdef CONFIG_XIP_KERNEL diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 678b892b4ed8..2c6b7ea33009 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -695,15 +695,15 @@ static uintptr_t __meminit best_map_size(phys_addr_t pa, uintptr_t va, phys_addr return PAGE_SIZE; if (pgtable_l5_enabled && - !(pa & (P4D_SIZE - 1)) && !(va & (P4D_SIZE - 1)) && size >= P4D_SIZE) + !(pa & (__P4D_SIZE - 1)) && !(va & (P4D_SIZE - 1)) && size >= P4D_SIZE) return P4D_SIZE; if (pgtable_l4_enabled && - !(pa & (PUD_SIZE - 1)) && !(va & (PUD_SIZE - 1)) && size >= PUD_SIZE) + !(pa & (__PUD_SIZE - 1)) && !(va & (PUD_SIZE - 1)) && size >= PUD_SIZE) return PUD_SIZE; if (IS_ENABLED(CONFIG_64BIT) && - !(pa & (PMD_SIZE - 1)) && !(va & (PMD_SIZE - 1)) && size >= PMD_SIZE) + !(pa & (__PMD_SIZE - 1)) && !(va & (PMD_SIZE - 1)) && size >= PMD_SIZE) return PMD_SIZE; return PAGE_SIZE; @@ -937,17 +937,33 @@ static void __init create_kernel_page_table(pgd_t *pgdir, PMD_SIZE, PAGE_KERNEL); } #else + +#ifdef CONFIG_RISCV_64K_PAGES +/* TODO: better implementation */ +#define KERNEL_MAP_STEP PAGE_SIZE +#else +#define KERNEL_MAP_STEP PMD_SIZE +#endif + static void __init create_kernel_page_table(pgd_t *pgdir, bool early) { uintptr_t va, end_va; end_va = kernel_map.virt_addr + kernel_map.size; - for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) - create_pgd_mapping(pgdir, va, - kernel_map.phys_addr + (va - kernel_map.virt_addr), - PMD_SIZE, - early ? - PAGE_KERNEL_EXEC : pgprot_from_va(va)); + if (early) + for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) + create_pgd_mapping(pgdir, va, + kernel_map.phys_addr + (va - kernel_map.virt_addr), + PMD_SIZE, + early ? + PAGE_KERNEL_EXEC : pgprot_from_va(va)); + else + for (va = kernel_map.virt_addr; va < end_va; va += KERNEL_MAP_STEP) + create_pgd_mapping(pgdir, va, + kernel_map.phys_addr + (va - kernel_map.virt_addr), + KERNEL_MAP_STEP, + early ? + PAGE_KERNEL_EXEC : pgprot_from_va(va)); } #endif @@ -1138,7 +1154,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) /* Sanity check alignment and size */ BUG_ON((PAGE_OFFSET % PGDIR_SIZE) != 0); - BUG_ON((kernel_map.phys_addr % PMD_SIZE) != 0); + BUG_ON((kernel_map.phys_addr % __PMD_SIZE) != 0); #ifdef CONFIG_64BIT /* From patchwork Thu Dec 5 10:37:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13894998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 46312E77173 for ; Thu, 5 Dec 2024 10:45:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xSCrIVHIkIicTGt5aIuWR1aSRV1M6PPm9zVXzGwPA3E=; b=XXc4BuY+bsf3ew PHFJrZVGbKXRcdJA2tpSrVf4e/+KO6pW7+wbZWDyHvoxD7GYGgL7xTp/t4gpF4+6UrTTE83HNi8yK w5bFSE8xW1uqD/wK+r6TubY4Sw+SYNGEVLYXsilvbwMnuMTuEpSgNJ/ONhllg+JRUNDSlLwdOrjcu aTme1z6WE4hVHX04wIBVewamSFBUXUVasknEHWc1+I810ZOjmxbcSnJqWxFeFEUeDc2bqOb087rK2 mwGa6tF2Exe+MB41brq83/3QMY1+z2mBd0KB3uKwSJfYUVvqy5kCtTd0z4HcJaon5j7a4E66NmxoZ rF1TKbC7YMoyJeHIyesg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9M5-0000000Fanp-37kP; Thu, 05 Dec 2024 10:45:09 +0000 Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJ9GD-0000000FYUr-0ATV for linux-riscv@lists.infradead.org; Thu, 05 Dec 2024 10:39:06 +0000 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-7fcf59a41ddso474622a12.3 for ; Thu, 05 Dec 2024 02:39:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733395144; x=1733999944; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iDBjWSESYKBIcxm/6NEJ0fQX3zIek6Dl7/vbsKDZ5GU=; b=e4cbb3//rN8pfSGJdbp6Z7lLbB27L+mq7YblVtFcbS5O2uvPbQ1DV+YRZmyhc++7hf xb5XxW1y7kd8soqzY4sD7otOx8HfXH9OkwQOkFSATSleGY8DSt6k2nPkU0uAhOMHuf2i iSfufSXL2mIdTkk2W+cb1byaYkJVS2oha8MvylpxOp6bWgkJ/j5eDcceXyOr6gy4D6xm fC2cR3BgW+K/zJymyXqPw8C5NHkAMGFnSExrc3d2tepDJXTSIXfFhZXXWrYU7yut0C4O o9/b9kZMKuJjia/mQw9KeR7hjKBwF+t59Vi5YaIX3WTn20chv+itOPJ9G4dElifqlr1b BdIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733395144; x=1733999944; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iDBjWSESYKBIcxm/6NEJ0fQX3zIek6Dl7/vbsKDZ5GU=; b=F1fFAnabQY0D3s/UoB5RYXXokZsynPS/FlcNdOABR/BeYho1eT8NmJOvelfJviohnu TOfk2RPENjgk9sIb3n+xdGMum7y32q/1O3JNWONiwhk4P7OfxxGHvyx5iT1Zo4dIQ6xS SFfegIuaJHQ4HyRZsqqoSUiGHOY9LlEicZDMAqD9s/2eqRbf742fRJFpOhZU8L2U8yIb wPCwMVdtEuqxcv5GEfpsUbACQCWToI7wAj72EcqsHpl8tgkJCsmXg5+VmLNV4e09PZkg bupZI0td+2Kv/1NxW59YwdWHnjGzmKUtb+kUc2hoFB48aafW45vnK/8pvXjdz75VsL5g dafg== X-Forwarded-Encrypted: i=1; AJvYcCW8ImFTOlyTMhCGKEOQweAtzuEZ5vCx1FLi9CsKdiUSRQInxxVVUkvLuibuhSko/DQLjIt40yigE4Riig==@lists.infradead.org X-Gm-Message-State: AOJu0YykCcWaZDCVNwucqzpLlpbi9ECOaNophll4H61/Sk7REM/2wVAD +bBc+ZkAqaX5fednTnVaX/JIBo0BB6V1caOJJJN76ejiLQBVJM9Lo7SKuwux7SE= X-Gm-Gg: ASbGncvcUHrVEVyXdMyqWVbOyTb7OTK4ftmX1/6jZdgOV8N5D8AOz5KG2xWpID9j+QE UUr0W8/UbXZWZkjduO7Jy4iB+BWffWZiZ9UMPjptsGolQdqBERSghkzBoTx451gmfCE6k/kqxGu FD3ynH1MlMwdWsOlpLUHL/IUcT1bqaqdhrH1Qg85odCtLgBI3bpZRUQJ+y6gM5Jd6woLqR0owez mnexOS5psNNdwLUZ3IFNVHSRytB9dDZoB10h1UWnghtvvL/J0PMxID1L8O03pF2eP21viLWFY54 OloLaj5lg8PuoCCNr0ugofsb3CSowFhi X-Google-Smtp-Source: AGHT+IHKuqD7zayvSRrPEZJa2a5jeQV4cpoc2Wz2fLw8d8qFILc/CQ/vzSdEu/3MiaJEA8awaTzYNQ== X-Received: by 2002:a05:6a20:3d87:b0:1db:d8a5:7c82 with SMTP id adf61e73a8af0-1e16539f381mr15601577637.5.1733395144181; Thu, 05 Dec 2024 02:39:04 -0800 (PST) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.56]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd156f048csm886826a12.39.2024.12.05.02.39.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 05 Dec 2024 02:39:03 -0800 (PST) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org Cc: xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Xu Lu Subject: [RFC PATCH v2 21/21] riscv: mm: Update EXEC_PAGESIZE for 64K Page Date: Thu, 5 Dec 2024 18:37:29 +0800 Message-Id: <20241205103729.14798-22-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241205103729.14798-1-luxu.kernel@bytedance.com> References: <20241205103729.14798-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241205_023905_077484_DD2BAA5B X-CRM114-Status: GOOD ( 12.19 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Signed-off-by: Xu Lu --- arch/riscv/include/uapi/asm/param.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 arch/riscv/include/uapi/asm/param.h diff --git a/arch/riscv/include/uapi/asm/param.h b/arch/riscv/include/uapi/asm/param.h new file mode 100644 index 000000000000..1221e570a077 --- /dev/null +++ b/arch/riscv/include/uapi/asm/param.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright (C) 2024 RISCV Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ +#ifndef __ASM_PARAM_H +#define __ASM_PARAM_H + +#define EXEC_PAGESIZE 65536 + +#include + +#endif