From patchwork Tue Mar 2 14:59:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Quentin Perret X-Patchwork-Id: 12113767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60483C433E6 for ; Wed, 3 Mar 2021 16:13:45 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0627E64F07 for ; Wed, 3 Mar 2021 16:13:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0627E64F07 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=C93ohdCIj6V6o4O/5y8pR9c4bwR2VoyVKhL4ZwIZqC0=; b=L/YIptt6zGYv/4 fFhx+2g/PJmjjf7yfQp5KoLjqsXkZgwh+GPB/LoUtFZ6OZT6ISfOQUX+55/3nPpF+6G2/63mweU8A fZVrGoDQ8eFG76WFdoqU9TRk1m4gz7xIkgkhh+1nTXCnytY5nAdCtp22aXzTWNm2YGSrikjbInZNf 6eODawLUuD7vMvg7wa8m3FPZkw1ElZCkxJNu7rqKnneXC1EK09Aofb7amv+cDWgw9sw+0NJGfZ4AL eDb6yR39k0N0EzTPeC4JAA5cRDBSYY3UpDjnAk/BacB2kFWLof3KfGomtTfDlUkNuICPkS/R/pBnX yvPTRkVpnOluLG8aioxA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHU68-005Y83-Ot; Wed, 03 Mar 2021 16:11:41 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHSp1-005Dmh-62 for linux-arm-kernel@desiato.infradead.org; Wed, 03 Mar 2021 14:49:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=c79EyduIP7MoVuBONgvo5g7CIT pzZhjzJgCbkFjYl748/7y5RNyhx6HS+/OsPbLoIxOI0D/m3st1H132NfZglbYfJ0M/DuQLrwmk6h5 37VzZs8jRxGUH0hODBpN9rZU6R3efklsNME7Tk+kiJYUFvwlqwor+3tSN/dzhpQlBPj/ost+qdWY/ ig4amP4u6LIt0cENnWv2huQyZ6QBvQbBSVZfG5rxg3IVEyw65/9ILVs4p4CViK+DFHnCxq9jG64mS 9xaBa4k99rZtDcTpF/4xgOiLGdeI9kOYYSr9HdIoeEEU3W/y/BZa0ZI5G3E/OYJJd5ZcaJPqeiLQF VCIdcgjw==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lH6Wi-00HJ16-Si for linux-arm-kernel@lists.infradead.org; Tue, 02 Mar 2021 15:01:36 +0000 Received: by mail-wm1-x349.google.com with SMTP id w10so789449wmk.1 for ; Tue, 02 Mar 2021 07:01:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=IWWlsrxoA4/VxGyh8bUKvZgzoCrsAILxDoeJNc3FK3cKMETE655OoJhqjhur1cilCo LYMhoMHTidPCmYEdxDntem+etgADvVUzHzOW5PAiTOjO1sxQz4TD7miwsbP4bqNvgZRY GlZRhT8A/bOsfAFAa73IPeR9tNI/bBCdkf7nnRVUDgwoAz6Myk3gOAqETYRQmdfeuwDY 8lBXQN3iPad2h8Ls6GQLyVj6IEr29AuM9kgnxAvmC7NESy44kzshGuXQ3dD7/Q0a9qA3 OFFXEc6wzEJ7OPcIwYWrU4XqEJ8dFiy2sRgH0Sm3Ojhm/myIj/5hcEc+9CMBFz1e24oI 1/Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nEicUMjcQyXP0pA9FIMjfTzv1YgzRd5C/+30qtrzjc8=; b=rWOIxYxuT5W+kh1aKlDaUIVI9lA5G/rEvBh0LGQjFRqwiGrM8j7nDadN3PkWbNLq+v vfb9rW9XOnhGGXbj6ANDa88X4sBl1YiXm6ONkUsSOcvVTy7ALfp8mFlhDjg8uXvjzHke pL0lCkMirBFwTsaeKmdz9d7Tdv2NVKAWqWdmvtAqy/0nsQaxTj10Cje/4yi5j0X+WdKG nuXyqvxieQs9V7lxFIn079QWw8MZrMH4mhBErM5KTZ1jp/twvYdTrwB/J1rC1o3Sn8sN sUOWDWroEQQ9F8Tz/ow81HQoidmcmnpUPwPqUCX6Djma8ASyJB9g8cyeAr9bkAn4IVTB O/xA== X-Gm-Message-State: AOAM531u3KVH0cU/KxQT6KQWWN/sXaPl7EMEH0EouKLGnK0R72djBAMf z8qfcAHf9KpS3YdL1gupZZLyHBh8Bc/2 X-Google-Smtp-Source: ABdhPJyhRguqkBiAL8rawxftQTL4cQe8tkMBmRvxwBQhjvtLIZuZAUV1pK/Un8+K/eUrAfebq5+z4KnW7CHU X-Received: from r2d2-qp.c.googlers.com ([fda3:e722:ac3:10:28:9cb1:c0a8:1652]) (user=qperret job=sendgmr) by 2002:a1c:5f89:: with SMTP id t131mr4235263wmb.173.1614697212949; Tue, 02 Mar 2021 07:00:12 -0800 (PST) Date: Tue, 2 Mar 2021 14:59:33 +0000 In-Reply-To: <20210302150002.3685113-1-qperret@google.com> Message-Id: <20210302150002.3685113-4-qperret@google.com> Mime-Version: 1.0 References: <20210302150002.3685113-1-qperret@google.com> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog Subject: [PATCH v3 03/32] arm64: kvm: Add standalone ticket spinlock implementation for use at hyp From: Quentin Perret To: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com Cc: android-kvm@google.com, linux-kernel@vger.kernel.org, kernel-team@android.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, tabba@google.com, mark.rutland@arm.com, dbrazdil@google.com, mate.toth-pal@arm.com, seanjc@google.com, qperret@google.com, robh+dt@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210302_150136_056085_C5902E5C X-CRM114-Status: GOOD ( 14.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Will Deacon We will soon need to synchronise multiple CPUs in the hyp text at EL2. The qspinlock-based locking used by the host is overkill for this purpose and relies on the kernel's "percpu" implementation for the MCS nodes. Implement a simple ticket locking scheme based heavily on the code removed by commit c11090474d70 ("arm64: locking: Replace ticket lock implementation with qspinlock"). Signed-off-by: Will Deacon Signed-off-by: Quentin Perret --- arch/arm64/kvm/hyp/include/nvhe/spinlock.h | 92 ++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/spinlock.h diff --git a/arch/arm64/kvm/hyp/include/nvhe/spinlock.h b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h new file mode 100644 index 000000000000..76b537f8d1c6 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/spinlock.h @@ -0,0 +1,92 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * A stand-alone ticket spinlock implementation for use by the non-VHE + * KVM hypervisor code running at EL2. + * + * Copyright (C) 2020 Google LLC + * Author: Will Deacon + * + * Heavily based on the implementation removed by c11090474d70 which was: + * Copyright (C) 2012 ARM Ltd. + */ + +#ifndef __ARM64_KVM_NVHE_SPINLOCK_H__ +#define __ARM64_KVM_NVHE_SPINLOCK_H__ + +#include +#include + +typedef union hyp_spinlock { + u32 __val; + struct { +#ifdef __AARCH64EB__ + u16 next, owner; +#else + u16 owner, next; +#endif + }; +} hyp_spinlock_t; + +#define hyp_spin_lock_init(l) \ +do { \ + *(l) = (hyp_spinlock_t){ .__val = 0 }; \ +} while (0) + +static inline void hyp_spin_lock(hyp_spinlock_t *lock) +{ + u32 tmp; + hyp_spinlock_t lockval, newval; + + asm volatile( + /* Atomically increment the next ticket. */ + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ +" prfm pstl1strm, %3\n" +"1: ldaxr %w0, %3\n" +" add %w1, %w0, #(1 << 16)\n" +" stxr %w2, %w1, %3\n" +" cbnz %w2, 1b\n", + /* LSE atomics */ +" mov %w2, #(1 << 16)\n" +" ldadda %w2, %w0, %3\n" + __nops(3)) + + /* Did we get the lock? */ +" eor %w1, %w0, %w0, ror #16\n" +" cbz %w1, 3f\n" + /* + * No: spin on the owner. Send a local event to avoid missing an + * unlock before the exclusive load. + */ +" sevl\n" +"2: wfe\n" +" ldaxrh %w2, %4\n" +" eor %w1, %w2, %w0, lsr #16\n" +" cbnz %w1, 2b\n" + /* We got the lock. Critical section starts here. */ +"3:" + : "=&r" (lockval), "=&r" (newval), "=&r" (tmp), "+Q" (*lock) + : "Q" (lock->owner) + : "memory"); +} + +static inline void hyp_spin_unlock(hyp_spinlock_t *lock) +{ + u64 tmp; + + asm volatile( + ARM64_LSE_ATOMIC_INSN( + /* LL/SC */ + " ldrh %w1, %0\n" + " add %w1, %w1, #1\n" + " stlrh %w1, %0", + /* LSE atomics */ + " mov %w1, #1\n" + " staddlh %w1, %0\n" + __nops(1)) + : "=Q" (lock->owner), "=&r" (tmp) + : + : "memory"); +} + +#endif /* __ARM64_KVM_NVHE_SPINLOCK_H__ */