From patchwork Fri Dec 6 16:30:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= X-Patchwork-Id: 13897452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E5C2E77173 for ; Fri, 6 Dec 2024 16:41:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Rih2z+q1Taz5W9gbG28kyjqjg/d4/wnIMKspBHeQrDw=; b=a36wCwagh1ZQg3LuQ/dpCA0lGB kdS9VZn7pdYECSzta4svrP9/2VRvuIOmw2sHnDKgkBdX2rHgFo3BLx2KF15HnmjiPo8zzJnVcrVV3 kKoW81MWgPJe5Eqg6Jm+q3utIjSIXKZQO/fObyrenQcDHlIbGd1ptiLcPruaHd6Zst+uBGYzQ6miq RCUk8XY2BZ+p9tODyWdp3OA1ImmwNEDGzNdNt0k2ZHhCcOus+26lRxiiLgLZXEclINSO0Z6LHJsFc +CPgkQOAThXnGukt8lJPukGxS24aWHDgFhtzBs7lioXCNs+px8UWedMiG04fBpFyPcsEV4mjp+DCi 2mcP6vYA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJbOR-00000002CvV-2LEM; Fri, 06 Dec 2024 16:41:27 +0000 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJbEV-00000002AJ8-2f3j for linux-arm-kernel@lists.infradead.org; Fri, 06 Dec 2024 16:31:13 +0000 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-4349e4e252dso23643715e9.0 for ; Fri, 06 Dec 2024 08:31:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1733502670; x=1734107470; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Rih2z+q1Taz5W9gbG28kyjqjg/d4/wnIMKspBHeQrDw=; b=Vwfq/vCxL3GP+gxRj2MjDxASPYPxR0utpfve16xptAetsKp8kp5BmcssysbHJt4J1C 7OzgJFzFJiKSIaEqWU40WMN9KOUoLM6/8HSxFsnoPrBcHoXNhfuZNI2Io6/1xDwiVHsp H7NOlx5RPupOT3lc8CJXNHIFQWHsaULEsfVxz7FnScvaRHdjt9KaBTbVPvdlP1vbC0MI wouDC2JTD88oJazZ9vEXPG5atM30aoh9dWG33W44lOqZgL5ctC0JdXvdZYLVxb/5w/9e R5RngJRNhQ8URXk7+XyHH/R7G0IrmTMqIN1/yN9gMmOPYvQNDDjjTSYZ/1+PC3ZAni5P oDwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733502670; x=1734107470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rih2z+q1Taz5W9gbG28kyjqjg/d4/wnIMKspBHeQrDw=; b=Ncd081kJ+z9guJsm51rDHYrZ5cPsd7d/s4ulIIHp+/5l/k5h2K1rmf5XRZ1pioPqA3 VbUYRQecE3Ub3JNlWEcipFNQnKEqXHELQDunqlhoAE45s8perH+tYstkbLf7o5MXHxDL mFy4JjFKKE7bJBULB4BAfbipjzcLaa4GttQXO07XsRx38rs5ae6uIes1Exf9V18OijQK 9LIcq1cZbhykpV1gqIwoQoBmdnQgsFPcSHSUFqPJBOQ6kZz6PEkFFq50OK+THE+wj76m PFwwq3KjAKCrZ2OBm2sptsHcW/2GeOtxdj+dHb4BYNeA09MOHIT+WujiPQPaqAe720f1 phdA== X-Forwarded-Encrypted: i=1; AJvYcCXxe9GGpoeaVhUsIJULDcOQrzSj0cQ/ByKiuTxdIzqUEZ/4BAO0uXzlIGJvlk680g5dg29ZxolG0KeeC3gnxpN2@lists.infradead.org X-Gm-Message-State: AOJu0YxAoxNXiT3c7uRLKdB0SpLJjXqWl8wOwrrd1UZDwXPHOLDvM1Eg Z3+mxmi6OFZrXa+24eAwTbcXOtYJkOjmlJ/0RLvh0XLr71BqYi0aypk8lHOwKheBg0ElsOb+P/f 1pmg= X-Gm-Gg: ASbGnctFXv4tO6vYSvMM1yfbYAgeVlrCUlqFlLsHBKX0EF2D3O1NbmFho3CQf7rzWxv Jqy9inLsGTWe8dZbAnFeh6Vi/PIW/YYN6GBnYHfWxAqN8NVHD1bybr7k2GHlPm9iVXJiF/3C7QM G1ybEGpjy8/al0AkvyOkjTQftKe9F2uPyfUgApnFGjD/UTkbEGXYueiPKBlrRL3g89xn1iVkcpK 0LAewqYNpdFTg/9i2lDb0H1gPsMNulieXj7TwPuR4ZwLX7rPfY= X-Google-Smtp-Source: AGHT+IFZNS/vLYTFlhcpAiQaF2L19OGIDhrILZwk3DfUtSNiJLmw56KxCqGdAEFEOkAw7CQrELObyw== X-Received: by 2002:a5d:64c3:0:b0:385:f7d2:7e29 with SMTP id ffacd0b85a97d-3862b35d19fmr3174954f8f.15.1733502669549; Fri, 06 Dec 2024 08:31:09 -0800 (PST) Received: from carbon-x1.. ([2a01:e0a:e17:9700:16d2:7456:6634:9626]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3862f02f5c3sm1151942f8f.65.2024.12.06.08.31.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 08:31:08 -0800 (PST) From: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= To: Paul Walmsley , Palmer Dabbelt , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Himanshu Chauhan , Anup Patel , Xu Lu , Atish Patra Subject: [PATCH v3 1/4] riscv: add SBI SSE extension definitions Date: Fri, 6 Dec 2024 17:30:57 +0100 Message-ID: <20241206163102.843505-2-cleger@rivosinc.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241206163102.843505-1-cleger@rivosinc.com> References: <20241206163102.843505-1-cleger@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241206_083111_671928_8B39E987 X-CRM114-Status: GOOD ( 10.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add needed definitions for SBI Supervisor Software Events extension [1]. This extension enables the SBI to inject events into supervisor software much like ARM SDEI. [1] https://lists.riscv.org/g/tech-prs/message/515 Signed-off-by: Clément Léger --- arch/riscv/include/asm/sbi.h | 62 ++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 6c82318065cf..032dde350d40 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -35,6 +35,7 @@ enum sbi_ext_id { SBI_EXT_DBCN = 0x4442434E, SBI_EXT_STA = 0x535441, SBI_EXT_NACL = 0x4E41434C, + SBI_EXT_SSE = 0x535345, /* Experimentals extensions must lie within this range */ SBI_EXT_EXPERIMENTAL_START = 0x08000000, @@ -401,6 +402,63 @@ enum sbi_ext_nacl_feature { #define SBI_NACL_SHMEM_SRET_X(__i) ((__riscv_xlen / 8) * (__i)) #define SBI_NACL_SHMEM_SRET_X_LAST 31 +enum sbi_ext_sse_fid { + SBI_SSE_EVENT_ATTR_READ = 0, + SBI_SSE_EVENT_ATTR_WRITE, + SBI_SSE_EVENT_REGISTER, + SBI_SSE_EVENT_UNREGISTER, + SBI_SSE_EVENT_ENABLE, + SBI_SSE_EVENT_DISABLE, + SBI_SSE_EVENT_COMPLETE, + SBI_SSE_EVENT_SIGNAL, + SBI_SSE_EVENT_HART_UNMASK, + SBI_SSE_EVENT_HART_MASK, +}; + +enum sbi_sse_state { + SBI_SSE_STATE_UNUSED = 0, + SBI_SSE_STATE_REGISTERED = 1, + SBI_SSE_STATE_ENABLED = 2, + SBI_SSE_STATE_RUNNING = 3, +}; + +/* SBI SSE Event Attributes. */ +enum sbi_sse_attr_id { + SBI_SSE_ATTR_STATUS = 0x00000000, + SBI_SSE_ATTR_PRIO = 0x00000001, + SBI_SSE_ATTR_CONFIG = 0x00000002, + SBI_SSE_ATTR_PREFERRED_HART = 0x00000003, + SBI_SSE_ATTR_ENTRY_PC = 0x00000004, + SBI_SSE_ATTR_ENTRY_ARG = 0x00000005, + SBI_SSE_ATTR_INTERRUPTED_SEPC = 0x00000006, + SBI_SSE_ATTR_INTERRUPTED_FLAGS = 0x00000007, + SBI_SSE_ATTR_INTERRUPTED_A6 = 0x00000008, + SBI_SSE_ATTR_INTERRUPTED_A7 = 0x00000009, + + SBI_SSE_ATTR_MAX = 0x0000000A +}; + +#define SBI_SSE_ATTR_STATUS_STATE_OFFSET 0 +#define SBI_SSE_ATTR_STATUS_STATE_MASK 0x3 +#define SBI_SSE_ATTR_STATUS_PENDING_OFFSET 2 +#define SBI_SSE_ATTR_STATUS_INJECT_OFFSET 3 + +#define SBI_SSE_ATTR_CONFIG_ONESHOT (1 << 0) + +#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPP (1 << 0) +#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_SSTATUS_SPIE (1 << 1) +#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPV (1 << 2) +#define SBI_SSE_ATTR_INTERRUPTED_FLAGS_HSTATUS_SPVP (1 << 3) + +#define SBI_SSE_EVENT_LOCAL_RAS 0x00000000 +#define SBI_SSE_EVENT_GLOBAL_RAS 0x00008000 +#define SBI_SSE_EVENT_LOCAL_PMU 0x00010000 +#define SBI_SSE_EVENT_LOCAL_SOFTWARE 0xffff0000 +#define SBI_SSE_EVENT_GLOBAL_SOFTWARE 0xffff8000 + +#define SBI_SSE_EVENT_PLATFORM (1 << 14) +#define SBI_SSE_EVENT_GLOBAL (1 << 15) + /* SBI spec version fields */ #define SBI_SPEC_VERSION_DEFAULT 0x1 #define SBI_SPEC_VERSION_MAJOR_SHIFT 24 @@ -418,6 +476,8 @@ enum sbi_ext_nacl_feature { #define SBI_ERR_ALREADY_STARTED -7 #define SBI_ERR_ALREADY_STOPPED -8 #define SBI_ERR_NO_SHMEM -9 +#define SBI_ERR_INVALID_STATE -10 +#define SBI_ERR_BAD_RANGE -11 extern unsigned long sbi_spec_version; struct sbiret { @@ -504,6 +564,8 @@ static inline int sbi_err_map_linux_errno(int err) case SBI_ERR_DENIED: return -EPERM; case SBI_ERR_INVALID_PARAM: + case SBI_ERR_BAD_RANGE: + case SBI_ERR_INVALID_STATE: return -EINVAL; case SBI_ERR_INVALID_ADDRESS: return -EFAULT; From patchwork Fri Dec 6 16:30:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= X-Patchwork-Id: 13897453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3941AE77173 for ; Fri, 6 Dec 2024 16:42:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pZs2436mAWrHKZxo6XHcSc+Z7MR9g3Sqc/8PR7lb/OY=; b=kiBnCRfnLX0be1EXq2rCLT+yWI dPcMXe1Ix8V559AA85L+FT5eI13u80FLpAaTniG6KoNKscJn5s85ni7BANLz2XM2BKQuYZs1aFBvk TpvG/LrcYZQ8G1ThhQVZYEZr1WMarXoLkGswPyo4l2kQCR8R+mAEl0zKKQX5w9//6SHilJTwaTL17 tja+loKOJQfKsGselgCeWS/3A6ACMT/d8v4+sWVApUGrIXIgYGQ/FDcTvDmtPCfyUu22eXRe/9Kyv M6/juqetTAxaBn6bVCFcu44q6OYMDjk9JoxwxVOu8ywW8GA6J5tSzoyy2QC8lgLumtrSjCxOkCTdD o8Q4+YSg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJbPR-00000002D9x-3PLu; Fri, 06 Dec 2024 16:42:29 +0000 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJbEX-00000002AKJ-0wIv for linux-arm-kernel@lists.infradead.org; Fri, 06 Dec 2024 16:31:14 +0000 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-385e25c5d75so1263484f8f.1 for ; Fri, 06 Dec 2024 08:31:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1733502671; x=1734107471; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pZs2436mAWrHKZxo6XHcSc+Z7MR9g3Sqc/8PR7lb/OY=; b=NWQECNgyNfMgAImmZ01Qaw/837ga/0zS1UxO536VdiPxRPYdcIhO9IFTMi37csUv8q GP6e115YGJeZfwtfvgeR+9EAYr+ZgjkSgsYwdoF+evTlP+EAx4wyMjXdbFb8BSK5Wfpw WMcX/3T58Wj5YQ0dCS1H0i2GL6yihIrrhMfnK3UANizI06/GJHrBehndW2HH7j5SN+s1 1YCJjQ6L0FI5W7R7F/FMJM7QaDp9mM7JCczTJeoYP8JjzGvf+/zSZtG1PZTGbW0ARq2L 21NFYFVPO+tjhqdLdj8k1s48GBCHf7eG2tna19H0ohAiwlGZJf/9PeuKFz/7ykxFkaLW g2Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733502671; x=1734107471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pZs2436mAWrHKZxo6XHcSc+Z7MR9g3Sqc/8PR7lb/OY=; b=UbBs0WWsGWx9xVDXabzmp7oqlztdE2JZGNVKDD5MEuXIVBWjVVbDhup73QI0FsQHxq T19IBaayA2RxTK03FAzh5Xo8NKg8DKVytIFSHzFBowcVHmSoX2tWzA+7ehVve8QSr8V8 C2xLavTsCX+wEF81t5gQq1TjL7+gCGwSnCkKcNbODmx0b8cbnL1q6HDXx40KxfZe/EGN 6A5GTO6iWWThN36w/YR9NOCDmE6y4smQy/QUw3umwiwYiOjV9AcsqvwJgZgIkKUWndWg asUp2nrwTcds09vn273NhKnUEjz5e8nNTj/nOWBsN705iugKXzjHpjsigsKNFMWWZMD+ VdnA== X-Forwarded-Encrypted: i=1; AJvYcCWH3mcvIJvb/A8l5iAtLv5tant8cgsTmPfMfsgRFTiF/1WsuF4pe9K+5qMCeSZIGTgfledtrkYARWGJmccwmMRl@lists.infradead.org X-Gm-Message-State: AOJu0YyL+n4OFkCFQKDolDPBf2+XqdbHyNzTRhuuGiq2lnejd3nOlqQv wz+4LACVX4AbOk4NhLa4l8VSsFHehryl6sZEfyoHjMbHYbHQ9RVkrtmzpYYPrPA= X-Gm-Gg: ASbGncvqj5V2Tjezz0N3pbVQukonz1Kb0jfa7NlVrOix6oOR5+IDOd71/r68+eTa7db A1BZp7V7XVLrGUHjllLhttHTeQNr1Qqhx5NXGorl6qYz6hzPN6K3GA4bBA1gSc6NuqjsilvbWlG CDrx2JalCq/ZRVoGK24NBPcq5FAM6/eX1YLQnmJzQFbvCiQr2shz3fH/+llF6NYsIHlOzXQFeu1 ITL5ckbyyTq1EL34bf9skD4JWF9yzd7t2zxplLzxsaxRM3h9Rg= X-Google-Smtp-Source: AGHT+IFHTqdtI6/G+fllUvPdGXlMfmZ7zOUkMH4p5p0imSiny8q9+zdmJJro+C/BeLlZrxw2h4Qlqg== X-Received: by 2002:a05:6000:1548:b0:385:dffb:4d70 with SMTP id ffacd0b85a97d-3862b3e818dmr2258522f8f.54.1733502671081; Fri, 06 Dec 2024 08:31:11 -0800 (PST) Received: from carbon-x1.. ([2a01:e0a:e17:9700:16d2:7456:6634:9626]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3862f02f5c3sm1151942f8f.65.2024.12.06.08.31.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 08:31:10 -0800 (PST) From: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= To: Paul Walmsley , Palmer Dabbelt , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Himanshu Chauhan , Anup Patel , Xu Lu , Atish Patra Subject: [PATCH v3 2/4] riscv: add support for SBI Supervisor Software Events extension Date: Fri, 6 Dec 2024 17:30:58 +0100 Message-ID: <20241206163102.843505-3-cleger@rivosinc.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241206163102.843505-1-cleger@rivosinc.com> References: <20241206163102.843505-1-cleger@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241206_083113_275896_C352FCEC X-CRM114-Status: GOOD ( 33.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The SBI SSE extension allows the supervisor software to be notified by the SBI of specific events that are not maskable. The context switch is handled partially by the firmware which will save registers a6 and a7. When entering kernel we can rely on these 2 registers to setup the stack and save all the registers. Since SSE events can be delivered at any time to the kernel (including during exception handling, we need a way to locate the current_task for context tracking. On RISC-V, it is sotred in scratch when in user space or tp when in kernel space (in which case SSCRATCH is zero). But at a at the beginning of exception handling, SSCRATCH is used to swap tp and check the origin of the exception. If interrupted at that point, then, there is no way to reliably know were is located the current task_struct. Even checking the interruption location won't work as SSE event can be nested on top of each other so the original interruption site might be lost at some point. In order to retrieve it reliably, store the current task in an additionnal __sse_entry_task per_cpu array. This array is then used to retrieve the current task based on the hart ID that is passed to the SSE event handler in a6. That being said, the way the current task struct is stored should probably be reworked to find a better reliable alternative. Since each events (and each CPU for local events) have their own context and can preempt each other, allocate a stack (and a shadow stack if needed for each of them (and for each cpu for local events). When completing the event, if we were coming from kernel with interrupts disabled, simply return there. If coming from userspace or kernel with interrupts enabled, simulate an interrupt exception by setting IE_SIE in CSR_IP to allow delivery of signals to user task. For instance this can happen, when a RAS event has been generated by a user application and a SIGBUS has been sent to a task. Signed-off-by: Clément Léger --- arch/riscv/include/asm/asm.h | 14 ++- arch/riscv/include/asm/scs.h | 7 ++ arch/riscv/include/asm/sse.h | 38 ++++++ arch/riscv/include/asm/switch_to.h | 14 +++ arch/riscv/include/asm/thread_info.h | 1 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 12 ++ arch/riscv/kernel/sse.c | 134 +++++++++++++++++++++ arch/riscv/kernel/sse_entry.S | 171 +++++++++++++++++++++++++++ 9 files changed, 389 insertions(+), 3 deletions(-) create mode 100644 arch/riscv/include/asm/sse.h create mode 100644 arch/riscv/kernel/sse.c create mode 100644 arch/riscv/kernel/sse_entry.S diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 776354895b81..de8427c58f02 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -89,16 +89,24 @@ #define PER_CPU_OFFSET_SHIFT 3 #endif -.macro asm_per_cpu dst sym tmp - REG_L \tmp, TASK_TI_CPU_NUM(tp) - slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT +.macro asm_per_cpu_with_cpu dst sym tmp cpu + slli \tmp, \cpu, PER_CPU_OFFSET_SHIFT la \dst, __per_cpu_offset add \dst, \dst, \tmp REG_L \tmp, 0(\dst) la \dst, \sym add \dst, \dst, \tmp .endm + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + asm_per_cpu_with_cpu \dst \sym \tmp \tmp +.endm #else /* CONFIG_SMP */ +.macro asm_per_cpu_with_cpu dst sym tmp cpu + la \dst, \sym +.endm + .macro asm_per_cpu dst sym tmp la \dst, \sym .endm diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 0e45db78b24b..62344daad73d 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -18,6 +18,11 @@ load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_sse_stack reg_evt + REG_L gp, SSE_REG_EVT_SHADOW_STACK(\reg_evt) +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -41,6 +46,8 @@ .endm .macro scs_load_irq_stack tmp .endm +.macro scs_load_sse_stack reg_evt +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/include/asm/sse.h b/arch/riscv/include/asm/sse.h new file mode 100644 index 000000000000..431a19d4cd9c --- /dev/null +++ b/arch/riscv/include/asm/sse.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 Rivos Inc. + */ +#ifndef __ASM_SSE_H +#define __ASM_SSE_H + +#ifdef CONFIG_RISCV_SSE + +struct sse_event_interrupted_state { + unsigned long a6; + unsigned long a7; +}; + +struct sse_event_arch_data { + void *stack; + void *shadow_stack; + unsigned long tmp; + struct sse_event_interrupted_state interrupted; + unsigned long interrupted_state_phys; + u32 evt_id; +}; + +struct sse_registered_event; +int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id, + int cpu); +void arch_sse_free_event(struct sse_event_arch_data *arch_evt); +int arch_sse_register_event(struct sse_event_arch_data *arch_evt); + +void sse_handle_event(struct sse_event_arch_data *arch_evt, + struct pt_regs *regs); +asmlinkage void handle_sse(void); +asmlinkage void do_sse(struct sse_event_arch_data *arch_evt, + struct pt_regs *reg); + +#endif + +#endif diff --git a/arch/riscv/include/asm/switch_to.h b/arch/riscv/include/asm/switch_to.h index 94e33216b2d9..e166fabe04ab 100644 --- a/arch/riscv/include/asm/switch_to.h +++ b/arch/riscv/include/asm/switch_to.h @@ -88,6 +88,19 @@ static inline void __switch_to_envcfg(struct task_struct *next) :: "r" (next->thread.envcfg) : "memory"); } +#ifdef CONFIG_RISCV_SSE +DECLARE_PER_CPU(struct task_struct *, __sse_entry_task); + +static inline void __switch_sse_entry_task(struct task_struct *next) +{ + __this_cpu_write(__sse_entry_task, next); +} +#else +static inline void __switch_sse_entry_task(struct task_struct *next) +{ +} +#endif + extern struct task_struct *__switch_to(struct task_struct *, struct task_struct *); @@ -122,6 +135,7 @@ do { \ if (switch_to_should_flush_icache(__next)) \ local_flush_icache_all(); \ __switch_to_envcfg(__next); \ + __switch_sse_entry_task(__next); \ ((last) = __switch_to(__prev, __next)); \ } while (0) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index f5916a70879a..28e9805e61fc 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -36,6 +36,7 @@ #define OVERFLOW_STACK_SIZE SZ_4K #define IRQ_STACK_SIZE THREAD_SIZE +#define SSE_STACK_SIZE THREAD_SIZE #ifndef __ASSEMBLY__ diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 063d1faf5a53..1e8fb83b1162 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -99,6 +99,7 @@ obj-$(CONFIG_DYNAMIC_FTRACE) += mcount-dyn.o obj-$(CONFIG_PERF_EVENTS) += perf_callchain.o obj-$(CONFIG_HAVE_PERF_REGS) += perf_regs.o obj-$(CONFIG_RISCV_SBI) += sbi.o sbi_ecall.o +obj-$(CONFIG_RISCV_SSE) += sse.o sse_entry.o ifeq ($(CONFIG_RISCV_SBI), y) obj-$(CONFIG_SMP) += sbi-ipi.o obj-$(CONFIG_SMP) += cpu_ops_sbi.o diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index e89455a6a0e5..60590a3d9519 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -14,6 +14,8 @@ #include #include #include +#include +#include #include void asm_offsets(void); @@ -511,4 +513,14 @@ void asm_offsets(void) DEFINE(FREGS_A6, offsetof(struct __arch_ftrace_regs, a6)); DEFINE(FREGS_A7, offsetof(struct __arch_ftrace_regs, a7)); #endif + +#ifdef CONFIG_RISCV_SSE + OFFSET(SSE_REG_EVT_STACK, sse_event_arch_data, stack); + OFFSET(SSE_REG_EVT_SHADOW_STACK, sse_event_arch_data, shadow_stack); + OFFSET(SSE_REG_EVT_TMP, sse_event_arch_data, tmp); + + DEFINE(SBI_EXT_SSE, SBI_EXT_SSE); + DEFINE(SBI_SSE_EVENT_COMPLETE, SBI_SSE_EVENT_COMPLETE); + DEFINE(NR_CPUS, NR_CPUS); +#endif } diff --git a/arch/riscv/kernel/sse.c b/arch/riscv/kernel/sse.c new file mode 100644 index 000000000000..b48ae69dad8d --- /dev/null +++ b/arch/riscv/kernel/sse.c @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2024 Rivos Inc. + */ +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +DEFINE_PER_CPU(struct task_struct *, __sse_entry_task); + +void __weak sse_handle_event(struct sse_event_arch_data *arch_evt, struct pt_regs *regs) +{ +} + +void do_sse(struct sse_event_arch_data *arch_evt, struct pt_regs *regs) +{ + nmi_enter(); + + /* Retrieve missing GPRs from SBI */ + sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, arch_evt->evt_id, + SBI_SSE_ATTR_INTERRUPTED_A6, + (SBI_SSE_ATTR_INTERRUPTED_A7 - SBI_SSE_ATTR_INTERRUPTED_A6) + 1, + arch_evt->interrupted_state_phys, 0, 0); + + memcpy(®s->a6, &arch_evt->interrupted, sizeof(arch_evt->interrupted)); + + sse_handle_event(arch_evt, regs); + + /* + * The SSE delivery path does not uses the "standard" exception path and + * thus does not process any pending signal/softirqs. Some drivers might + * enqueue pending work that needs to be handled as soon as possible. + * For that purpose, set the software interrupt pending bit which will + * be serviced once interrupts are reenabled + */ + csr_set(CSR_IP, IE_SIE); + + nmi_exit(); +} + +#ifdef CONFIG_VMAP_STACK +static unsigned long *sse_stack_alloc(unsigned int cpu, unsigned int size) +{ + return arch_alloc_vmap_stack(size, cpu_to_node(cpu)); +} + +static void sse_stack_free(unsigned long *stack) +{ + vfree(stack); +} +#else /* CONFIG_VMAP_STACK */ + +static unsigned long *sse_stack_alloc(unsigned int cpu, unsigned int size) +{ + return kmalloc(size, GFP_KERNEL); +} + +static void sse_stack_free(unsigned long *stack) +{ + kfree(stack); +} + +#endif /* CONFIG_VMAP_STACK */ + +static int sse_init_scs(int cpu, struct sse_event_arch_data *arch_evt) +{ + void *stack; + + if (!scs_is_enabled()) + return 0; + + stack = scs_alloc(cpu_to_node(cpu)); + if (!stack) + return 1; + + arch_evt->shadow_stack = stack; + + return 0; +} + +int arch_sse_init_event(struct sse_event_arch_data *arch_evt, u32 evt_id, int cpu) +{ + void *stack; + + arch_evt->evt_id = evt_id; + stack = sse_stack_alloc(cpu, SSE_STACK_SIZE); + if (!stack) + return -ENOMEM; + + arch_evt->stack = stack + SSE_STACK_SIZE; + + if (sse_init_scs(cpu, arch_evt)) + goto free_stack; + + if (is_kernel_percpu_address((unsigned long)&arch_evt->interrupted)) { + arch_evt->interrupted_state_phys = + per_cpu_ptr_to_phys(&arch_evt->interrupted); + } else { + arch_evt->interrupted_state_phys = + virt_to_phys(&arch_evt->interrupted); + } + + return 0; + +free_stack: + sse_stack_free(arch_evt->stack - SSE_STACK_SIZE); + + return -ENOMEM; +} + +void arch_sse_free_event(struct sse_event_arch_data *arch_evt) +{ + scs_free(arch_evt->shadow_stack); + sse_stack_free(arch_evt->stack - SSE_STACK_SIZE); +} + +int arch_sse_register_event(struct sse_event_arch_data *arch_evt) +{ + struct sbiret sret; + + sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_REGISTER, arch_evt->evt_id, + (unsigned long) handle_sse, (unsigned long) arch_evt, + 0, 0, 0); + + return sbi_err_map_linux_errno(sret.error); +} diff --git a/arch/riscv/kernel/sse_entry.S b/arch/riscv/kernel/sse_entry.S new file mode 100644 index 000000000000..0b2f890edd89 --- /dev/null +++ b/arch/riscv/kernel/sse_entry.S @@ -0,0 +1,171 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 Rivos Inc. + */ + +#include +#include + +#include +#include +#include + +/* When entering handle_sse, the following registers are set: + * a6: contains the hartid + * a6: contains struct sse_registered_event pointer + */ +SYM_CODE_START(handle_sse) + /* Save stack temporarily */ + REG_S sp, SSE_REG_EVT_TMP(a7) + /* Set entry stack */ + REG_L sp, SSE_REG_EVT_STACK(a7) + + addi sp, sp, -(PT_SIZE_ON_STACK) + REG_S ra, PT_RA(sp) + REG_S s0, PT_S0(sp) + REG_S s1, PT_S1(sp) + REG_S s2, PT_S2(sp) + REG_S s3, PT_S3(sp) + REG_S s4, PT_S4(sp) + REG_S s5, PT_S5(sp) + REG_S s6, PT_S6(sp) + REG_S s7, PT_S7(sp) + REG_S s8, PT_S8(sp) + REG_S s9, PT_S9(sp) + REG_S s10, PT_S10(sp) + REG_S s11, PT_S11(sp) + REG_S tp, PT_TP(sp) + REG_S t0, PT_T0(sp) + REG_S t1, PT_T1(sp) + REG_S t2, PT_T2(sp) + REG_S t3, PT_T3(sp) + REG_S t4, PT_T4(sp) + REG_S t5, PT_T5(sp) + REG_S t6, PT_T6(sp) + REG_S gp, PT_GP(sp) + REG_S a0, PT_A0(sp) + REG_S a1, PT_A1(sp) + REG_S a2, PT_A2(sp) + REG_S a3, PT_A3(sp) + REG_S a4, PT_A4(sp) + REG_S a5, PT_A5(sp) + + /* Retrieve entry sp */ + REG_L a4, SSE_REG_EVT_TMP(a7) + /* Save CSRs */ + csrr a0, CSR_EPC + csrr a1, CSR_SSTATUS + csrr a2, CSR_STVAL + csrr a3, CSR_SCAUSE + + REG_S a0, PT_EPC(sp) + REG_S a1, PT_STATUS(sp) + REG_S a2, PT_BADADDR(sp) + REG_S a3, PT_CAUSE(sp) + REG_S a4, PT_SP(sp) + + /* Disable user memory access and floating/vector computing */ + li t0, SR_SUM | SR_FS_VS + csrc CSR_STATUS, t0 + + load_global_pointer + scs_load_sse_stack a7 + + /* Restore current task struct from __sse_entry_task */ + li t1, NR_CPUS + move t3, zero + +#ifdef CONFIG_SMP + /* Find the CPU id associated to the hart id */ + la t0, __cpuid_to_hartid_map +.Lhart_id_loop: + REG_L t2, 0(t0) + beq t2, a6, .Lcpu_id_found + + /* Increment pointer and CPU number */ + addi t3, t3, 1 + addi t0, t0, RISCV_SZPTR + bltu t3, t1, .Lhart_id_loop + + /* + * This should never happen since we expect the hart_id to match one + * of our CPU, but better be safe than sorry + */ + la tp, init_task + la a0, sse_hart_id_panic_string + la t0, panic + jalr t0 + +.Lcpu_id_found: +#endif + asm_per_cpu_with_cpu t2 __sse_entry_task t1 t3 + REG_L tp, 0(t2) + + move a1, sp /* pt_regs on stack */ + /* Kernel was interrupted, create stack frame */ + beqz s1, .Lcall_do_sse + +.Lcall_do_sse: + /* + * Save sscratch for restoration since we might have interrupted the + * kernel in early exception path and thus, we don't know the content of + * sscratch. + */ + csrr s4, CSR_SSCRATCH + /* In-kernel scratch is 0 */ + csrw CSR_SCRATCH, x0 + + move a0, a7 + + call do_sse + + csrw CSR_SSCRATCH, s4 + + REG_L a0, PT_EPC(sp) + REG_L a1, PT_STATUS(sp) + REG_L a2, PT_BADADDR(sp) + REG_L a3, PT_CAUSE(sp) + csrw CSR_EPC, a0 + csrw CSR_SSTATUS, a1 + csrw CSR_STVAL, a2 + csrw CSR_SCAUSE, a3 + + REG_L ra, PT_RA(sp) + REG_L s0, PT_S0(sp) + REG_L s1, PT_S1(sp) + REG_L s2, PT_S2(sp) + REG_L s3, PT_S3(sp) + REG_L s4, PT_S4(sp) + REG_L s5, PT_S5(sp) + REG_L s6, PT_S6(sp) + REG_L s7, PT_S7(sp) + REG_L s8, PT_S8(sp) + REG_L s9, PT_S9(sp) + REG_L s10, PT_S10(sp) + REG_L s11, PT_S11(sp) + REG_L tp, PT_TP(sp) + REG_L t0, PT_T0(sp) + REG_L t1, PT_T1(sp) + REG_L t2, PT_T2(sp) + REG_L t3, PT_T3(sp) + REG_L t4, PT_T4(sp) + REG_L t5, PT_T5(sp) + REG_L t6, PT_T6(sp) + REG_L gp, PT_GP(sp) + REG_L a0, PT_A0(sp) + REG_L a1, PT_A1(sp) + REG_L a2, PT_A2(sp) + REG_L a3, PT_A3(sp) + REG_L a4, PT_A4(sp) + REG_L a5, PT_A5(sp) + + REG_L sp, PT_SP(sp) + + li a7, SBI_EXT_SSE + li a6, SBI_SSE_EVENT_COMPLETE + ecall + +SYM_CODE_END(handle_sse) + +sse_hart_id_panic_string: + .ascii "Unable to match hart_id with cpu\0" From patchwork Fri Dec 6 16:30:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= X-Patchwork-Id: 13897454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7D9E0E77173 for ; Fri, 6 Dec 2024 16:43:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BfT5Kkdmbft1Gxq8r2rJeiri4i23l4sd3A0EAGdLLmI=; b=cImhUuzOntIExIeSdwM8vVwIco 8gFPc5fP9h84Vc6cH7gBvyIC09rDS0MeMvD+X95AUvAT0uM0Qz8S3ZxI2I0PvTggwG6pHyAxEROTW ZgEFee7LVE95J+5zBdc08UXtFTUUxEbvadvHuaAM601Z5FVayxlmh2lmWt5gJLYq8QYqHRWQMreVM HhCLN+6OWN2R2ScaoEMBvAbc6lLJAa2w9Hvf4xcX8qIRMW/1VsrzIiDHnBVJZtJ7dgVhsZlB/+wwO Rblp0ceX+YqQlYScw05s9y/p5dW0tD3uKMglMIeDiaEY3bgzWE3rRSDhHToKIjcUaJqq9pw5qLsPh M2/VhMOw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJbQS-00000002DNh-1jpG; Fri, 06 Dec 2024 16:43:32 +0000 Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJbEY-00000002AKh-0B5Q for linux-arm-kernel@lists.infradead.org; Fri, 06 Dec 2024 16:31:15 +0000 Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-43497839b80so15146295e9.2 for ; Fri, 06 Dec 2024 08:31:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1733502672; x=1734107472; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BfT5Kkdmbft1Gxq8r2rJeiri4i23l4sd3A0EAGdLLmI=; b=hqHTK32MOLbWiPexacSp26n90lFX4fYD3ZluyzfCJo6d9oySye8u5b0msqdkrpFKe7 bHO5xp3aHFMN3cTJuEBbyzOfEyXfCevyngCNSoDJuTbIG+MUmHfx5A24IpzqdRjcnemM aULHKdSZYN+6eET0KoCwWPm3Dml+ECrYezWYBzLUYddPCYtXIkm8bMJCWmLuCt0SdcxF g0kCrVobDSmvWViL0CVKsFlOvOiXqKlAFYveEbgADDkCz73ZGZdTXiQl32VmgdbpE1bp 8wuijrBGuDMLUSkCdC4RSvHzo1QJiufzODdFEJFZb6HaFAAWPpB1P8+tT9dBW/B9sIjq k2rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733502672; x=1734107472; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BfT5Kkdmbft1Gxq8r2rJeiri4i23l4sd3A0EAGdLLmI=; b=TtF4rDgb57S+DxEZqIywYjta6vmCoZIrFl4w6Ps7ak9TNrmQJHHSRYGUdjOuI17UvC aFWff9MU5d3q9mBs+aex2dP1hpT5YqnbFcfd9LO9VDmSNP7iFLLbjxeuPqDVQxTe4GaD L/FCmMVFlo+2Qj/xCpoNWzEtkgXeT+lnm9fD2JuliqHoMkTc8h/ohxDWAYArPZnPSzjt m+zJ2Wl2ge9QLfQpjhj1Tw3vUMovn5SVDTmzkpcYP5LX7baa1DbZX4s+77swRO3roD86 K6np4ouK3poSPVT3V6QJHpwWSpJ11e9AlFFwMxWAhr+GuzddTmV++tVJEDvBu9nzM/LI qL6w== X-Forwarded-Encrypted: i=1; AJvYcCUWaEaXCARKCd+/RXLnkozNidmEZJas7VXQcWRYvsgAJIRWvqkwZo09cX/mtF/vzIZHm8+VUcSZuzGJsnbGGqDP@lists.infradead.org X-Gm-Message-State: AOJu0Yw3QZUN+rkEYZwr7saqlnjXSX0SLaYBSczDVpaEnTvVU7XG4qa8 8qE+lDZXVxoPOlHJQzTKsjb8XZOYMLgTsBgDnVX8JXLpYlJgeh71v+IwUH0c6no= X-Gm-Gg: ASbGnctGYPRfFqm9cyxvuqGqaF7sWsHmvy/0PMMeEs8jrBxSqGM5WRdCSr38sO3T3A1 eDdvmZr1IVCEU+r80ZP9zUiRJUytEwd9vCgn9Fis2Zn+RVFqC6kEmNSskV0Nt/jQphzH0f3E7pO 1weur8oqvLtJAh0Qf4oPimTmFljbhjlViDTFq18wUt5nrexRdjc1Hg2jFwUJcDEl7LGDYdjvTCU dhNXdlo/cOCZ97mm0SPZMCIDwje8bNKZ334dFg1SvbU21iyrbQ= X-Google-Smtp-Source: AGHT+IHy+99vjEXsDzeRxTuszjeNWkcia8EHvdNhsO5qiI0y2ktocNDtRBy0o1xn0ld1+Yk9pm/TMA== X-Received: by 2002:a05:6000:789:b0:385:fb66:faee with SMTP id ffacd0b85a97d-3862b36a455mr2853683f8f.18.1733502672402; Fri, 06 Dec 2024 08:31:12 -0800 (PST) Received: from carbon-x1.. ([2a01:e0a:e17:9700:16d2:7456:6634:9626]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3862f02f5c3sm1151942f8f.65.2024.12.06.08.31.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 08:31:11 -0800 (PST) From: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= To: Paul Walmsley , Palmer Dabbelt , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Himanshu Chauhan , Anup Patel , Xu Lu , Atish Patra Subject: [PATCH v3 3/4] drivers: firmware: add riscv SSE support Date: Fri, 6 Dec 2024 17:30:59 +0100 Message-ID: <20241206163102.843505-4-cleger@rivosinc.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241206163102.843505-1-cleger@rivosinc.com> References: <20241206163102.843505-1-cleger@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241206_083114_151797_26807F6E X-CRM114-Status: GOOD ( 24.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add driver level interface to use RISC-V SSE arch support. This interface allows registering SSE handlers, and receive them. This will be used by PMU and GHES driver. Signed-off-by: Himanshu Chauhan Co-developed-by: Himanshu Chauhan Signed-off-by: Clément Léger --- MAINTAINERS | 14 + drivers/firmware/Kconfig | 1 + drivers/firmware/Makefile | 1 + drivers/firmware/riscv/Kconfig | 15 + drivers/firmware/riscv/Makefile | 3 + drivers/firmware/riscv/riscv_sse.c | 691 +++++++++++++++++++++++++++++ include/linux/riscv_sse.h | 56 +++ 7 files changed, 781 insertions(+) create mode 100644 drivers/firmware/riscv/Kconfig create mode 100644 drivers/firmware/riscv/Makefile create mode 100644 drivers/firmware/riscv/riscv_sse.c create mode 100644 include/linux/riscv_sse.h diff --git a/MAINTAINERS b/MAINTAINERS index 686109008d8e..a3ddde7fe9fb 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -20125,6 +20125,13 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/iommu/linux.git F: Documentation/devicetree/bindings/iommu/riscv,iommu.yaml F: drivers/iommu/riscv/ +RISC-V FIRMWARE DRIVERS +M: Conor Dooley +L: linux-riscv@lists.infradead.org +S: Maintained +T: git git://git.kernel.org/pub/scm/linux/kernel/git/conor/linux.git +F: drivers/firmware/riscv/* + RISC-V MICROCHIP FPGA SUPPORT M: Conor Dooley M: Daire McNamara @@ -20177,6 +20184,13 @@ F: drivers/perf/riscv_pmu.c F: drivers/perf/riscv_pmu_legacy.c F: drivers/perf/riscv_pmu_sbi.c +RISC-V SSE DRIVER +M: Clément Léger +L: linux-riscv@lists.infradead.org +S: Maintained +F: drivers/firmware/riscv/riscv_sse.c +F: include/linux/riscv_sse.h + RISC-V THEAD SoC SUPPORT M: Drew Fustini M: Guo Ren diff --git a/drivers/firmware/Kconfig b/drivers/firmware/Kconfig index 71d8b26c4103..9e996a1fd511 100644 --- a/drivers/firmware/Kconfig +++ b/drivers/firmware/Kconfig @@ -267,6 +267,7 @@ source "drivers/firmware/meson/Kconfig" source "drivers/firmware/microchip/Kconfig" source "drivers/firmware/psci/Kconfig" source "drivers/firmware/qcom/Kconfig" +source "drivers/firmware/riscv/Kconfig" source "drivers/firmware/smccc/Kconfig" source "drivers/firmware/tegra/Kconfig" source "drivers/firmware/xilinx/Kconfig" diff --git a/drivers/firmware/Makefile b/drivers/firmware/Makefile index 7a8d486e718f..c0f5009949a8 100644 --- a/drivers/firmware/Makefile +++ b/drivers/firmware/Makefile @@ -33,6 +33,7 @@ obj-y += efi/ obj-y += imx/ obj-y += psci/ obj-y += qcom/ +obj-y += riscv/ obj-y += smccc/ obj-y += tegra/ obj-y += xilinx/ diff --git a/drivers/firmware/riscv/Kconfig b/drivers/firmware/riscv/Kconfig new file mode 100644 index 000000000000..8056ed3262d9 --- /dev/null +++ b/drivers/firmware/riscv/Kconfig @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: GPL-2.0-only +menu "Risc-V Specific firmware drivers" +depends on RISCV + +config RISCV_SSE + bool "Enable SBI Supervisor Software Events support" + depends on RISCV_SBI + default y + help + The Supervisor Software Events support allow the SBI to deliver + NMI-like notifications to the supervisor mode software. When enable, + this option provides support to register callbacks on specific SSE + events. + +endmenu diff --git a/drivers/firmware/riscv/Makefile b/drivers/firmware/riscv/Makefile new file mode 100644 index 000000000000..4ccfcbbc28ea --- /dev/null +++ b/drivers/firmware/riscv/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_RISCV_SSE) += riscv_sse.o diff --git a/drivers/firmware/riscv/riscv_sse.c b/drivers/firmware/riscv/riscv_sse.c new file mode 100644 index 000000000000..c165e32cc9a5 --- /dev/null +++ b/drivers/firmware/riscv/riscv_sse.c @@ -0,0 +1,691 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 Rivos Inc. + */ + +#define pr_fmt(fmt) "sse: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +struct sse_event { + struct list_head list; + u32 evt; + u32 priority; + sse_event_handler *handler; + void *handler_arg; + bool is_enabled; + /* Only valid for global events */ + unsigned int cpu; + + union { + struct sse_registered_event *global; + struct sse_registered_event __percpu *local; + }; +}; + +static int sse_hp_state; +static bool sse_available; +static DEFINE_SPINLOCK(events_list_lock); +static LIST_HEAD(events); +static DEFINE_MUTEX(sse_mutex); + +struct sse_registered_event { + struct sse_event_arch_data arch; + struct sse_event *evt; + unsigned long attr_buf; +}; + +void sse_handle_event(struct sse_event_arch_data *arch_event, + struct pt_regs *regs) +{ + int ret; + struct sse_registered_event *reg_evt = + container_of(arch_event, struct sse_registered_event, arch); + struct sse_event *evt = reg_evt->evt; + + ret = evt->handler(evt->evt, evt->handler_arg, regs); + if (ret) + pr_warn("event %x handler failed with error %d\n", evt->evt, + ret); +} + +static bool sse_event_is_global(u32 evt) +{ + return !!(evt & SBI_SSE_EVENT_GLOBAL); +} + +static +struct sse_event *sse_event_get(u32 evt) +{ + struct sse_event *sse_evt = NULL, *tmp; + + scoped_guard(spinlock, &events_list_lock) { + list_for_each_entry(tmp, &events, list) { + if (tmp->evt == evt) { + return sse_evt; + } + } + } + + return NULL; +} + +static phys_addr_t sse_event_get_phys(struct sse_registered_event *reg_evt, + void *addr) +{ + phys_addr_t phys; + + if (sse_event_is_global(reg_evt->evt->evt)) + phys = virt_to_phys(addr); + else + phys = per_cpu_ptr_to_phys(addr); + + return phys; +} + +static int sse_sbi_event_func(struct sse_event *event, unsigned long func) +{ + struct sbiret ret; + u32 evt = event->evt; + + ret = sbi_ecall(SBI_EXT_SSE, func, evt, 0, 0, 0, 0, 0); + if (ret.error) + pr_debug("Failed to execute func %lx, event %x, error %ld\n", + func, evt, ret.error); + + return sbi_err_map_linux_errno(ret.error); +} + +static int sse_sbi_disable_event(struct sse_event *event) +{ + return sse_sbi_event_func(event, SBI_SSE_EVENT_DISABLE); +} + +static int sse_sbi_enable_event(struct sse_event *event) +{ + return sse_sbi_event_func(event, SBI_SSE_EVENT_ENABLE); +} + +static int sse_event_attr_get_no_lock(struct sse_registered_event *reg_evt, + unsigned long attr_id, unsigned long *val) +{ + struct sbiret sret; + u32 evt = reg_evt->evt->evt; + unsigned long phys; + + phys = sse_event_get_phys(reg_evt, ®_evt->attr_buf); + + sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_READ, evt, + attr_id, 1, phys, 0, 0); + if (sret.error) { + pr_debug("Failed to get event %x attr %lx, error %ld\n", evt, + attr_id, sret.error); + return sbi_err_map_linux_errno(sret.error); + } + + *val = reg_evt->attr_buf; + + return 0; +} + +static int sse_event_attr_set_nolock(struct sse_registered_event *reg_evt, + unsigned long attr_id, unsigned long val) +{ + struct sbiret sret; + u32 evt = reg_evt->evt->evt; + unsigned long phys; + + reg_evt->attr_buf = val; + phys = sse_event_get_phys(reg_evt, ®_evt->attr_buf); + + sret = sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_ATTR_WRITE, evt, + attr_id, 1, phys, 0, 0); + if (sret.error && sret.error != SBI_ERR_INVALID_STATE) { + pr_debug("Failed to set event %x attr %lx, error %ld\n", evt, + attr_id, sret.error); + return sbi_err_map_linux_errno(sret.error); + } + + return 0; +} + +static int sse_event_set_target_cpu_nolock(struct sse_event *event, + unsigned int cpu) +{ + unsigned int hart_id = cpuid_to_hartid_map(cpu); + struct sse_registered_event *reg_evt = event->global; + u32 evt = event->evt; + bool was_enabled; + int ret; + + if (!sse_event_is_global(evt)) + return -EINVAL; + + was_enabled = event->is_enabled; + if (was_enabled) + sse_sbi_disable_event(event); + do { + ret = sse_event_attr_set_nolock(reg_evt, + SBI_SSE_ATTR_PREFERRED_HART, + hart_id); + } while (ret == -EINVAL); + + if (ret == 0) + event->cpu = cpu; + + if (was_enabled) + sse_sbi_enable_event(event); + + return 0; +} + +int sse_event_set_target_cpu(struct sse_event *event, unsigned int cpu) +{ + int ret; + + scoped_guard(mutex, &sse_mutex) { + cpus_read_lock(); + + if (!cpu_online(cpu)) + return -EINVAL; + + ret = sse_event_set_target_cpu_nolock(event, cpu); + + cpus_read_unlock(); + } + + return ret; +} + +static int sse_event_init_registered(unsigned int cpu, + struct sse_registered_event *reg_evt, + struct sse_event *event) +{ + reg_evt->evt = event; + arch_sse_init_event(®_evt->arch, event->evt, cpu); + + return 0; +} + +static void sse_event_free_registered(struct sse_registered_event *reg_evt) +{ + arch_sse_free_event(®_evt->arch); +} + +static int sse_event_alloc_global(struct sse_event *event) +{ + int err; + struct sse_registered_event *reg_evt; + + reg_evt = kzalloc(sizeof(*reg_evt), GFP_KERNEL); + if (!reg_evt) + return -ENOMEM; + + event->global = reg_evt; + err = sse_event_init_registered(smp_processor_id(), reg_evt, + event); + if (err) + kfree(reg_evt); + + return err; +} + +static int sse_event_alloc_local(struct sse_event *event) +{ + int err; + unsigned int cpu, err_cpu; + struct sse_registered_event *reg_evt; + struct sse_registered_event __percpu *reg_evts; + + reg_evts = alloc_percpu(struct sse_registered_event); + if (!reg_evts) + return -ENOMEM; + + event->local = reg_evts; + + for_each_possible_cpu(cpu) { + reg_evt = per_cpu_ptr(reg_evts, cpu); + err = sse_event_init_registered(cpu, reg_evt, event); + if (err) { + err_cpu = cpu; + goto err_free_per_cpu; + } + } + + return 0; + +err_free_per_cpu: + for_each_possible_cpu(cpu) { + if (cpu == err_cpu) + break; + reg_evt = per_cpu_ptr(reg_evts, cpu); + sse_event_free_registered(reg_evt); + } + + free_percpu(reg_evts); + + return err; +} + +static struct sse_event *sse_event_alloc(u32 evt, + u32 priority, + sse_event_handler *handler, void *arg) +{ + int err; + struct sse_event *event; + + event = kzalloc(sizeof(*event), GFP_KERNEL); + if (!event) + return ERR_PTR(-ENOMEM); + + event->evt = evt; + event->priority = priority; + event->handler_arg = arg; + event->handler = handler; + + if (sse_event_is_global(evt)) { + err = sse_event_alloc_global(event); + if (err) + goto err_alloc_reg_evt; + } else { + err = sse_event_alloc_local(event); + if (err) + goto err_alloc_reg_evt; + } + + return event; + +err_alloc_reg_evt: + kfree(event); + + return ERR_PTR(err); +} + +static int sse_sbi_register_event(struct sse_event *event, + struct sse_registered_event *reg_evt) +{ + int ret; + + ret = sse_event_attr_set_nolock(reg_evt, SBI_SSE_ATTR_PRIO, + event->priority); + if (ret) + return ret; + + return arch_sse_register_event(®_evt->arch); +} + +static int sse_event_register_local(struct sse_event *event) +{ + int ret; + struct sse_registered_event *reg_evt = per_cpu_ptr(event->local, + smp_processor_id()); + + ret = sse_sbi_register_event(event, reg_evt); + if (ret) + pr_debug("Failed to register event %x: err %d\n", event->evt, + ret); + + return ret; +} + + +static int sse_sbi_unregister_event(struct sse_event *event) +{ + return sse_sbi_event_func(event, SBI_SSE_EVENT_UNREGISTER); +} + +struct sse_per_cpu_evt { + struct sse_event *event; + unsigned long func; + atomic_t error; +}; + +static void sse_event_per_cpu_func(void *info) +{ + int ret; + struct sse_per_cpu_evt *cpu_evt = info; + + if (cpu_evt->func == SBI_SSE_EVENT_REGISTER) + ret = sse_event_register_local(cpu_evt->event); + else + ret = sse_sbi_event_func(cpu_evt->event, cpu_evt->func); + + if (ret) + atomic_set(&cpu_evt->error, ret); +} + +static void sse_event_free(struct sse_event *event) +{ + unsigned int cpu; + struct sse_registered_event *reg_evt; + + if (sse_event_is_global(event->evt)) { + sse_event_free_registered(event->global); + kfree(event->global); + } else { + for_each_possible_cpu(cpu) { + reg_evt = per_cpu_ptr(event->local, cpu); + sse_event_free_registered(reg_evt); + } + free_percpu(event->local); + } + + kfree(event); +} + +int sse_event_enable(struct sse_event *event) +{ + int ret = 0; + struct sse_per_cpu_evt cpu_evt; + + scoped_guard(mutex, &sse_mutex) { + cpus_read_lock(); + if (sse_event_is_global(event->evt)) { + ret = sse_sbi_enable_event(event); + } else { + cpu_evt.event = event; + atomic_set(&cpu_evt.error, 0); + cpu_evt.func = SBI_SSE_EVENT_ENABLE; + on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1); + ret = atomic_read(&cpu_evt.error); + if (ret) { + cpu_evt.func = SBI_SSE_EVENT_DISABLE; + on_each_cpu(sse_event_per_cpu_func, &cpu_evt, + 1); + } + } + cpus_read_unlock(); + + if (ret == 0) + event->is_enabled = true; + } + + return ret; +} + +static void sse_events_mask(void) +{ + sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_MASK, 0, 0, 0, 0, 0, 0); +} + +static void sse_events_unmask(void) +{ + sbi_ecall(SBI_EXT_SSE, SBI_SSE_EVENT_HART_UNMASK, 0, 0, 0, 0, 0, 0); +} + +static void sse_event_disable_nolock(struct sse_event *event) +{ + struct sse_per_cpu_evt cpu_evt; + + if (sse_event_is_global(event->evt)) { + sse_sbi_disable_event(event); + } else { + cpu_evt.event = event; + cpu_evt.func = SBI_SSE_EVENT_DISABLE; + on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1); + } +} + +void sse_event_disable(struct sse_event *event) +{ + scoped_guard(mutex, &sse_mutex) { + cpus_read_lock(); + sse_event_disable_nolock(event); + event->is_enabled = false; + cpus_read_unlock(); + } +} + +struct sse_event *sse_event_register(u32 evt, u32 priority, + sse_event_handler *handler, void *arg) +{ + struct sse_per_cpu_evt cpu_evt; + struct sse_event *event; + int ret = 0; + + if (!sse_available) + return ERR_PTR(-EOPNOTSUPP); + + mutex_lock(&sse_mutex); + if (sse_event_get(evt)) { + pr_debug("Event %x already registered\n", evt); + ret = -EEXIST; + goto out_unlock; + } + + event = sse_event_alloc(evt, priority, handler, arg); + if (IS_ERR(event)) { + ret = PTR_ERR(event); + goto out_unlock; + } + + cpus_read_lock(); + if (sse_event_is_global(evt)) { + unsigned long preferred_hart; + + ret = sse_event_attr_get_no_lock(event->global, + SBI_SSE_ATTR_PREFERRED_HART, + &preferred_hart); + if (ret) + goto err_event_free; + event->cpu = riscv_hartid_to_cpuid(preferred_hart); + + ret = sse_sbi_register_event(event, event->global); + if (ret) + goto err_event_free; + + } else { + cpu_evt.event = event; + atomic_set(&cpu_evt.error, 0); + cpu_evt.func = SBI_SSE_EVENT_REGISTER; + on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1); + ret = atomic_read(&cpu_evt.error); + if (ret) { + cpu_evt.func = SBI_SSE_EVENT_UNREGISTER; + on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1); + goto err_event_free; + } + } + cpus_read_unlock(); + + scoped_guard(spinlock, &events_list_lock) + list_add(&event->list, &events); + + mutex_unlock(&sse_mutex); + + return event; + +err_event_free: + cpus_read_unlock(); + sse_event_free(event); +out_unlock: + mutex_unlock(&sse_mutex); + + return ERR_PTR(ret); +} + +static void sse_event_unregister_nolock(struct sse_event *event) +{ + struct sse_per_cpu_evt cpu_evt; + + if (sse_event_is_global(event->evt)) { + sse_sbi_unregister_event(event); + } else { + cpu_evt.event = event; + cpu_evt.func = SBI_SSE_EVENT_UNREGISTER; + on_each_cpu(sse_event_per_cpu_func, &cpu_evt, 1); + } +} + +void sse_event_unregister(struct sse_event *event) +{ + scoped_guard(mutex, &sse_mutex) { + cpus_read_lock(); + sse_event_unregister_nolock(event); + cpus_read_unlock(); + + scoped_guard(spinlock, &events_list_lock) + list_del(&event->list); + + sse_event_free(event); + } +} + +static int sse_cpu_online(unsigned int cpu) +{ + struct sse_event *sse_evt; + + scoped_guard(spinlock, &events_list_lock) { + list_for_each_entry(sse_evt, &events, list) { + if (sse_event_is_global(sse_evt->evt)) + continue; + + sse_event_register_local(sse_evt); + if (sse_evt->is_enabled) + sse_sbi_enable_event(sse_evt); + } + } + + /* Ready to handle events. Unmask SSE. */ + sse_events_unmask(); + + return 0; +} + +static int sse_cpu_teardown(unsigned int cpu) +{ + unsigned int next_cpu; + struct sse_event *sse_evt; + + /* Mask the sse events */ + sse_events_mask(); + + scoped_guard(spinlock, &events_list_lock) { + list_for_each_entry(sse_evt, &events, list) { + if (!sse_event_is_global(sse_evt->evt)) { + + if (sse_evt->is_enabled) + sse_sbi_disable_event(sse_evt); + + sse_sbi_unregister_event(sse_evt); + continue; + } + + if (sse_evt->cpu != smp_processor_id()) + continue; + + /* Update destination hart for global event */ + next_cpu = cpumask_any_but(cpu_online_mask, cpu); + sse_event_set_target_cpu_nolock(sse_evt, next_cpu); + } + } + + return 0; +} + +static void sse_reset(void) +{ + struct sse_event *event = NULL; + + list_for_each_entry(event, &events, list) { + sse_event_disable_nolock(event); + sse_event_unregister_nolock(event); + } +} + +static int sse_pm_notifier(struct notifier_block *nb, unsigned long action, + void *data) +{ + WARN_ON_ONCE(preemptible()); + + switch (action) { + case CPU_PM_ENTER: + sse_events_mask(); + break; + case CPU_PM_EXIT: + case CPU_PM_ENTER_FAILED: + sse_events_unmask(); + break; + default: + return NOTIFY_DONE; + } + + return NOTIFY_OK; +} + +static struct notifier_block sse_pm_nb = { + .notifier_call = sse_pm_notifier, +}; + +/* + * Mask all CPUs and unregister all events on panic, reboot or kexec. + */ +static int sse_reboot_notifier(struct notifier_block *nb, unsigned long action, + void *data) +{ + cpuhp_remove_state(sse_hp_state); + + sse_reset(); + + return NOTIFY_OK; +} + +static struct notifier_block sse_reboot_nb = { + .notifier_call = sse_reboot_notifier, +}; + +static int __init sse_init(void) +{ + int cpu, ret; + + if (sbi_probe_extension(SBI_EXT_SSE) <= 0) { + pr_err("Missing SBI SSE extension\n"); + return -EOPNOTSUPP; + } + pr_info("SBI SSE extension detected\n"); + + for_each_possible_cpu(cpu) + INIT_LIST_HEAD(&events); + + ret = cpu_pm_register_notifier(&sse_pm_nb); + if (ret) { + pr_warn("Failed to register CPU PM notifier...\n"); + return ret; + } + + ret = register_reboot_notifier(&sse_reboot_nb); + if (ret) { + pr_warn("Failed to register reboot notifier...\n"); + goto remove_cpupm; + } + + ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "riscv/sse:online", + sse_cpu_online, sse_cpu_teardown); + if (ret < 0) + goto remove_reboot; + + sse_hp_state = ret; + sse_available = true; + + return 0; + +remove_reboot: + unregister_reboot_notifier(&sse_reboot_nb); + +remove_cpupm: + cpu_pm_unregister_notifier(&sse_pm_nb); + + return ret; +} +arch_initcall(sse_init); diff --git a/include/linux/riscv_sse.h b/include/linux/riscv_sse.h new file mode 100644 index 000000000000..c73184074b8c --- /dev/null +++ b/include/linux/riscv_sse.h @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2024 Rivos Inc. + */ + +#ifndef __LINUX_RISCV_SSE_H +#define __LINUX_RISCV_SSE_H + +#include +#include + +struct sse_event; +struct pt_regs; + +typedef int (sse_event_handler)(u32 event_num, void *arg, struct pt_regs *regs); + +#ifdef CONFIG_RISCV_SSE + +struct sse_event *sse_event_register(u32 event_num, u32 priority, + sse_event_handler *handler, void *arg); + +void sse_event_unregister(struct sse_event *evt); + +int sse_event_set_target_cpu(struct sse_event *sse_evt, unsigned int cpu); + +int sse_event_enable(struct sse_event *sse_evt); + +void sse_event_disable(struct sse_event *sse_evt); + +#else +static inline struct sse_event *sse_event_register(u32 event_num, u32 priority, + sse_event_handler *handler, + void *arg) +{ + return ERR_PTR(-EOPNOTSUPP); +} + +static inline void sse_event_unregister(struct sse_event *evt) {} + +static inline int sse_event_set_target_cpu(struct sse_event *sse_evt, + unsigned int cpu) +{ + return -EOPNOTSUPP; +} + +static inline int sse_event_enable(struct sse_event *sse_evt) +{ + return -EOPNOTSUPP; +} + +static inline void sse_event_disable(struct sse_event *sse_evt) {} + + +#endif + +#endif /* __LINUX_RISCV_SSE_H */ From patchwork Fri Dec 6 16:31:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= X-Patchwork-Id: 13897464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BA4B9E77179 for ; Fri, 6 Dec 2024 16:44:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=6pTgQSov7vYTWNucdQE0t9hyfOwyYPp0snQQ1hygLdA=; b=LGSFx7bRQrMIhvYAob7E/EboPu fKgOBt+T8lu2EkgHujfzxAF2vWJesDDYhv85648uvFeAjPIXvOQ3c/dqJYlkL4qdU8xzRmvyRavwy euP/qHE3Bza7M2gV9IOS4150lzPtKfcRJRa9eJ5DbwKSeP3kiAUK82Yi1i9t33FdxzWTmUzkDec2I NPUSzzyBMqA6bJDtJxOqWfkbBJeh0cIw+ROw/fWlCxAcar+ZTVD+d+9yvRxW5+TdJZwyN8l19Ou1Q otUwuSoVbh49jod+44PmwnOkkTAJxQogCpBank9yTkNOMSXHYG8iC4ODEnlXQ2w7T9VqquOSB0kpt Rm4l8Y5A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tJbRS-00000002DpO-0zjd; Fri, 06 Dec 2024 16:44:34 +0000 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tJbEZ-00000002AM6-2mHG for linux-arm-kernel@lists.infradead.org; Fri, 06 Dec 2024 16:31:16 +0000 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-434a766b475so22249335e9.1 for ; Fri, 06 Dec 2024 08:31:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1733502673; x=1734107473; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6pTgQSov7vYTWNucdQE0t9hyfOwyYPp0snQQ1hygLdA=; b=vPs+nmqP5EhBxPfOxdtt4W6CeUWNr/LKELP4Am8MDe1EAi4E08bI/6o+8M5F0JgK8g XTrJDVbuX++23Q3C2H66slqDF+wiEzaQlRjKXwchCGimsxE63uoErpB1/rcWps2Sx+Nk 86mYPc1StFzIaKSRA4mBo7idztungb9FPn/96NpmqaouxEuecTImSev8/zQy/egz5YOj D+J2z8M1CyNqSyNLFXrYoBIsbaJLcRr3xaDz5G1YUGFZ6X1+8OBZVKt/zw2HcR6KiH5D nb+zwWrxxtBxYu0TjBFO4NrJux14hdZiAFDaIROgq6Ht0MFIUO1pX6JIxdoJlOD30B+3 PK2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733502673; x=1734107473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6pTgQSov7vYTWNucdQE0t9hyfOwyYPp0snQQ1hygLdA=; b=wQO3wNWt+LqmKqWxqPjTQkXfivyakf45PKbi+3yg6fSXCvVkw2dqiKkhX+6pGiolRr QVQzPOiblVWVJuken1Jeof6tqpPGEJxNAWPlTRdAUU5/MTqpvo++fClfQc/YA9WBzSHH yxHqNvMkGkEbdBGf4YjqepCUGt1wu9A/iJ/KDpLMuaXCqkZjjhENgq5pLBepIRV0qU3X 2+9KjUE4GxBBz7C376GpKLV/Ld4EcD6VywpRu/z+wuRRQ06uu1Fy7D6BHynd8JnKA8EB aJ+e9DP/5BDBJTtQpApUlvP21yxVSzQ+xY22MMnkBDo8YRe8R9o0Q62NyArnDLbz5M4C Q88A== X-Forwarded-Encrypted: i=1; AJvYcCW6I8bB1cKWH7I2nOcibYv/yywHg19L4Vjmj7xF0jsbTDofEX3nfvE8Vz14F2Tzg5SVH5xPIQdeLbNAjlScdLXe@lists.infradead.org X-Gm-Message-State: AOJu0Yx6DzzDhzPyqi1XhekwuA7PX6KcDK5bgH3CZBKJ1W5vfIXs+MmS 6ddvMT8T6Ty9vp3BDuT8CBQ6EgZN+/3MG+55GYBrM4+R8upuhwz6qbzu8CBWnHw= X-Gm-Gg: ASbGncvaUTGTyx4EzWQ3UW1Ki70XxvoU3LasO8TcOJOJAp/I1ZPewVRl2hbYnDekmkg AmDnfOwRIUOB0yx7bPo577dtSE660E0uMQYEEvkE/Ztet7rr2nYuSbdH8BQxBXlF5IYVly9hV16 tMNwCgGLo7QesxPXHSJ5/5ioCN4EMGm59SsZ777DR/GdxbieOrbEA6gDIZj8aWltDdW+6Q6FCyG 9vfPMj+8RsZ2pvsezjStSYc2I8bGlSJYcLLquvNYRPUr+/nCHM= X-Google-Smtp-Source: AGHT+IEXqcNtnOQtrn9HCLkcY6FnKIIl9CkgV9xxYEovPWiiPgpTNBlpOFS9Y9bIrLbUO0lLfarFWQ== X-Received: by 2002:a05:6000:1846:b0:385:fb2c:6034 with SMTP id ffacd0b85a97d-3862b3e6fddmr2723661f8f.47.1733502673558; Fri, 06 Dec 2024 08:31:13 -0800 (PST) Received: from carbon-x1.. ([2a01:e0a:e17:9700:16d2:7456:6634:9626]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3862f02f5c3sm1151942f8f.65.2024.12.06.08.31.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Dec 2024 08:31:12 -0800 (PST) From: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= To: Paul Walmsley , Palmer Dabbelt , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Himanshu Chauhan , Anup Patel , Xu Lu , Atish Patra Subject: [PATCH v3 4/4] perf: RISC-V: add support for SSE event Date: Fri, 6 Dec 2024 17:31:00 +0100 Message-ID: <20241206163102.843505-5-cleger@rivosinc.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241206163102.843505-1-cleger@rivosinc.com> References: <20241206163102.843505-1-cleger@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241206_083115_694209_B6973B57 X-CRM114-Status: GOOD ( 18.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to use SSE within PMU drivers, register a SSE handler for the local PMU event. Reuse the existing overflow IRQ handler and pass appropriate pt_regs. Signed-off-by: Clément Léger --- drivers/perf/riscv_pmu_sbi.c | 51 +++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 10 deletions(-) diff --git a/drivers/perf/riscv_pmu_sbi.c b/drivers/perf/riscv_pmu_sbi.c index 1aa303f76cc7..bd7ab15483db 100644 --- a/drivers/perf/riscv_pmu_sbi.c +++ b/drivers/perf/riscv_pmu_sbi.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -946,10 +947,10 @@ static void pmu_sbi_start_overflow_mask(struct riscv_pmu *pmu, pmu_sbi_start_ovf_ctrs_sbi(cpu_hw_evt, ctr_ovf_mask); } -static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) +static irqreturn_t pmu_sbi_ovf_handler(struct cpu_hw_events *cpu_hw_evt, + struct pt_regs *regs, bool from_sse) { struct perf_sample_data data; - struct pt_regs *regs; struct hw_perf_event *hw_evt; union sbi_pmu_ctr_info *info; int lidx, hidx, fidx; @@ -957,7 +958,6 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) struct perf_event *event; u64 overflow; u64 overflowed_ctrs = 0; - struct cpu_hw_events *cpu_hw_evt = dev; u64 start_clock = sched_clock(); struct riscv_pmu_snapshot_data *sdata = cpu_hw_evt->snapshot_addr; @@ -967,13 +967,15 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) /* Firmware counter don't support overflow yet */ fidx = find_first_bit(cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS); if (fidx == RISCV_MAX_COUNTERS) { - csr_clear(CSR_SIP, BIT(riscv_pmu_irq_num)); + if (!from_sse) + csr_clear(CSR_SIP, BIT(riscv_pmu_irq_num)); return IRQ_NONE; } event = cpu_hw_evt->events[fidx]; if (!event) { - ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); + if (!from_sse) + ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); return IRQ_NONE; } @@ -988,16 +990,16 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) /* * Overflow interrupt pending bit should only be cleared after stopping - * all the counters to avoid any race condition. + * all the counters to avoid any race condition. When using SSE, + * interrupt is cleared when stopping counters. */ - ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); + if (!from_sse) + ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); /* No overflow bit is set */ if (!overflow) return IRQ_NONE; - regs = get_irq_regs(); - for_each_set_bit(lidx, cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS) { struct perf_event *event = cpu_hw_evt->events[lidx]; @@ -1053,6 +1055,22 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) return IRQ_HANDLED; } +static irqreturn_t pmu_sbi_ovf_irq_handler(int irq, void *dev) +{ + return pmu_sbi_ovf_handler(dev, get_irq_regs(), false); +} + +static int pmu_sbi_ovf_sse_handler(uint32_t evt, void *arg, + struct pt_regs *regs) +{ + struct cpu_hw_events __percpu *hw_events = arg; + struct cpu_hw_events *hw_event = raw_cpu_ptr(hw_events); + + pmu_sbi_ovf_handler(hw_event, regs, true); + + return 0; +} + static int pmu_sbi_starting_cpu(unsigned int cpu, struct hlist_node *node) { struct riscv_pmu *pmu = hlist_entry_safe(node, struct riscv_pmu, node); @@ -1100,9 +1118,22 @@ static int pmu_sbi_dying_cpu(unsigned int cpu, struct hlist_node *node) static int pmu_sbi_setup_irqs(struct riscv_pmu *pmu, struct platform_device *pdev) { int ret; + struct sse_event *evt; struct cpu_hw_events __percpu *hw_events = pmu->hw_events; struct irq_domain *domain = NULL; + evt = sse_event_register(SBI_SSE_EVENT_LOCAL_PMU, 0, + pmu_sbi_ovf_sse_handler, hw_events); + if (!IS_ERR(evt)) { + ret = sse_event_enable(evt); + if (!ret) { + pr_info("using SSE for PMU event delivery\n"); + return 0; + } + + sse_event_unregister(evt); + } + if (riscv_isa_extension_available(NULL, SSCOFPMF)) { riscv_pmu_irq_num = RV_IRQ_PMU; riscv_pmu_use_irq = true; @@ -1137,7 +1168,7 @@ static int pmu_sbi_setup_irqs(struct riscv_pmu *pmu, struct platform_device *pde return -ENODEV; } - ret = request_percpu_irq(riscv_pmu_irq, pmu_sbi_ovf_handler, "riscv-pmu", hw_events); + ret = request_percpu_irq(riscv_pmu_irq, pmu_sbi_ovf_irq_handler, "riscv-pmu", hw_events); if (ret) { pr_err("registering percpu irq failed [%d]\n", ret); return ret;