From patchwork Fri Apr 2 00:56:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12180159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95208C433B4 for ; Fri, 2 Apr 2021 01:01:10 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 00AC161106 for ; Fri, 2 Apr 2021 01:01:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00AC161106 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:Reply-To:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:From:Subject:References:Mime-Version: Message-Id:In-Reply-To:Date:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BhU8UBAluXpaHXzMHMsAWXFZebEZ9H5QN9NFs/ENdnc=; b=cwfnVLLFvjEE7tYZHRBJy/nxb vOIr7rx9PbEo5AGk0tuOF2VAXZ7A8QgSqBzdhRZ3CumrMzktj8PTZY2+dgM1KKIef6Ch8B8Dstw+Z NVQJIm0tSVWmaCyAlLKeta6a1/2BoHXi/3Lrs3fYPCcjxXRX+7U3kkA66nAy1y7JvbSAo1qOK/H1u HqpC5DsMFAnapBfpf8ADyax1mHxVVMXms5y6zL7s9+IAwo0F/dJzmnRu6GTAOWloWC69hhDwbe1zP ZLwkb1mZM9GxJTeVEWa1Ngq+DkHfCop7h04YC1Hoq8JK62xt0wam27+SVOfBB36w5LxQDq4cVL/5O 3eHBtoRvQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lS89V-00BZQa-0Y; Fri, 02 Apr 2021 00:59:09 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lS87k-00BYrK-Kq for linux-arm-kernel@lists.infradead.org; Fri, 02 Apr 2021 00:57:22 +0000 Received: by mail-yb1-xb49.google.com with SMTP id i6so7669316ybk.2 for ; Thu, 01 Apr 2021 17:57:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=0/ecd8NmYTmeaiIYnkr6nuUXDKg3hRMT4+OQ0nyX4DE=; b=oYFGYSyo5B35UKJnfDl/ptvTjCVW8KNCbB5+juLbSKFFyhdI1o15HSF2gl9NbqxKmR WTeg8loXzFmw7sYPBcDz9SnE1ep/TQ8R11Jo0v6/+BXHFnVXD5GetugNTwD0b5Bqt1L/ mx+Wv2fDdgWFKAi4FQThRwTMjWA7vs3uA/I3KI9C+xT2UVJEymjeiJxY9WdDCMtNY1Gh GqU4Bg7UboHpTTQPVmttgw2eCvKNRYVbBfciANMUqXXzeQ8x+WKQOPRt9i/fgo0c9Wxl r3IbL0WnEuE8LcmbAeXvX6nCRCfN/9RtrNuT+vA3q5p7oxMFeV4Y6S1fLZYoQC3OglWL 5a1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=0/ecd8NmYTmeaiIYnkr6nuUXDKg3hRMT4+OQ0nyX4DE=; b=TDqr2gV9/BZiIXfw2jBKdZYdYK2an0CwVUMurQOD2NC2FSheP1uwptCFv/kAcA3YFF 1+gSoRCwkbNekKbN+ImjEksOF45/sqj+unSUG0jhB/naBsofKk0pgrc02ob4RW9Hikjo uDuOre7iCOvjh5Cqx2+cfyAVcRhIggkSz5OgPDxzWWgK7PqpSC0ucIZysWOL0nGLGGJd pfsTVoYLZTIR8GgkqPHEtjfGR39LbOuUEC1Bqvl6QFkWg72Ccax0CLpVnFlTTDto0aU4 g/+tnkU86ONVphK60WXJbe0GjrVMYb+jneNvkFqCazunAHmEGTx7OxA6pseVQRivenoT gv8g== X-Gm-Message-State: AOAM531FV/qDl72OC4YFZPJg79CGUXb/ArLdxNh461aDq4FRAt8shjyi wLn3YRYaV7Q2cfsh34Mt2UkKxvaoTWQ= X-Google-Smtp-Source: ABdhPJzt4o9P3PY2HrRiNrmz7rdm15sQ6vGfWjEexZR+eCJ9LN9XguP8ZdUR1MsXg+JiybkoEf+aAjrbxig= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:c0b4:8b8:bb34:6a56]) (user=seanjc job=sendgmr) by 2002:a25:74d2:: with SMTP id p201mr16375842ybc.406.1617325038741; Thu, 01 Apr 2021 17:57:18 -0700 (PDT) Date: Thu, 1 Apr 2021 17:56:56 -0700 In-Reply-To: <20210402005658.3024832-1-seanjc@google.com> Message-Id: <20210402005658.3024832-9-seanjc@google.com> Mime-Version: 1.0 References: <20210402005658.3024832-1-seanjc@google.com> X-Mailer: git-send-email 2.31.0.208.g409f899ff0-goog Subject: [PATCH v2 08/10] KVM: Take mmu_lock when handling MMU notifier iff the hva hits a memslot From: Sean Christopherson To: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Paolo Bonzini Cc: James Morse , Julien Thierry , Suzuki K Poulose , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, linux-kernel@vger.kernel.org, Ben Gardon X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210402_015720_912012_C0D12AA2 X-CRM114-Status: GOOD ( 15.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Sean Christopherson Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Defer acquiring mmu_lock in the MMU notifier paths until a "hit" has been detected in the memslots, i.e. don't take the lock for notifications that don't affect the guest. For small VMs, spurious locking is a minor annoyance. And for "volatile" setups where the majority of notifications _are_ relevant, this barely qualifies as an optimization. But, for large VMs (hundreds of threads) with static setups, e.g. no page migration, no swapping, etc..., the vast majority of MMU notifier callbacks will be unrelated to the guest, e.g. will often be in response to the userspace VMM adjusting its own virtual address space. In such large VMs, acquiring mmu_lock can be painful as it blocks vCPUs from handling page faults. In some scenarios it can even be "fatal" in the sense that it causes unacceptable brownouts, e.g. when rebuilding huge pages after live migration, a significant percentage of vCPUs will be attempting to handle page faults. x86's TDP MMU implementation is especially susceptible to spurious locking due it taking mmu_lock for read when handling page faults. Because rwlock is fair, a single writer will stall future readers, while the writer is itself stalled waiting for in-progress readers to complete. This is exacerbated by the MMU notifiers often firing multiple times in quick succession, e.g. moving a page will (always?) invoke three separate notifiers: .invalidate_range_start(), invalidate_range_end(), and .change_pte(). Unnecessarily taking mmu_lock each time means even a single spurious sequence can be problematic. Note, this optimizes only the unpaired callbacks. Optimizing the .invalidate_range_{start,end}() pairs is more complex and will be done in a future patch. Suggested-by: Ben Gardon Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 25ecb5235e17..f6697ad741ed 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -482,10 +482,10 @@ static void kvm_null_fn(void) static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, const struct kvm_hva_range *range) { + bool ret = false, locked = false; struct kvm_gfn_range gfn_range; struct kvm_memory_slot *slot; struct kvm_memslots *slots; - bool ret = false; int i, idx; /* A null handler is allowed if and only if on_lock() is provided. */ @@ -493,11 +493,13 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler))) return 0; - KVM_MMU_LOCK(kvm); - idx = srcu_read_lock(&kvm->srcu); + /* The on_lock() path does not yet support lock elision. */ if (!IS_KVM_NULL_FN(range->on_lock)) { + locked = true; + KVM_MMU_LOCK(kvm); + range->on_lock(kvm, range->start, range->end); if (IS_KVM_NULL_FN(range->handler)) @@ -532,6 +534,10 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + if (!locked) { + locked = true; + KVM_MMU_LOCK(kvm); + } ret |= range->handler(kvm, &gfn_range); } } @@ -540,7 +546,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); out_unlock: - KVM_MMU_UNLOCK(kvm); + if (locked) + KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx);