From patchwork Fri Aug 9 19:43:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759215 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0692E15F404 for ; Fri, 9 Aug 2024 19:43:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232623; cv=none; b=fv5WcoiWCdaOh9IJ0XW9v/00M++hjT8jaWnNA23x3sF7Nqyn/Bv5G9sWQsP+aGIw1NMuuwBdmPUlbHI2Q8KqH6iV6lXrvQvM8+6+9vugydbGGAypZ89iiWatjIMpsyaM/wTMshjJ7QHPqzZF7tHW07MZBIvrKz52K+6MpoHta50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232623; c=relaxed/simple; bh=UxnvgXHSiR2kQsYdw03waRMGPjo2zH4fYnT3IGJL6c0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eYV2UnWhyePaFlMet3CjGO9TybyukeMMb9fsEkCIZNnGW9FbInnWV32jUZqK9yI2s7SNMI9uOAwy3jouF4HZd6OT2mYe4SNm3OdETdx16IMLZgQ+YRO3WmKnoj05ZdN1ap284gJ/nzTcr8G19Z2CPW/L61UIpiO42QdQB1gek40= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LCaId1Mh; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LCaId1Mh" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1fc5acc1b96so25144885ad.3 for ; Fri, 09 Aug 2024 12:43:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232621; x=1723837421; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=7EQAoYr/rc8nw+x53hvnHC7R0vSXW8+m/U14Qwxr4YU=; b=LCaId1Mh/cfaZ1A3flIXbMN83yBaKr3eYyJEIzeWrB9ivmSxDZc+VvMaDztmkpgr+q IdtIZrzYDAjomWisZ/GQgXsnJ2RxFlGQp400ic94KFn3V1GkU8cMXITkZFSi1CYETvNO eslSjnmvX3Gj8rLGgXN7ugHWhNACs36OiQo2pTzCcg095YFxQQClYPbo9USpdyVBHTmS L9b+E2cFQ6RB4w5N+v2/Jln6IXLEsoKEinkMSazH2Kn2OQWqYfiKyLAVrGkpivH1bZQ9 dIBIQGc3YW+kRLmcI2UXk01/cFwcNdb4N5S+kbP6KKvJsGuWn0OnUhnlunOXCMJ/n7ZR uuWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232621; x=1723837421; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=7EQAoYr/rc8nw+x53hvnHC7R0vSXW8+m/U14Qwxr4YU=; b=wPXXCzeY0ZiirKnDdEC019d5eDMDQJ0jUb5YzMJuri5TCSDB/rOdfxHo8rrov9Ko0b EocOT7uhuM0gznZGmzs3M4ttCtAMv4uVXNZdW4rUDs5Uu0IVmuS830IanAE6GxzkStuU zpeakCFviTrwNg+xYrodI3GbpW1IZNjhXa9j8VDuP+Wq9Z3s1kXliQ273TLgUleyhuLL jNtMttKeSd4NskS1EqrNUj2bNsyzdiwcI3P0VuLfB6soK4FWlsosqJ2qotKqkW5XYWAN N62JJOPH6HE1GLzdBoYAmGGDGjQ81pgY01SoqCOOsCBoqYUSXTiTkAgyxM6CM565pt0Z t7ZQ== X-Gm-Message-State: AOJu0YxcoRoaqa/G7j9AWYjKss8IgNoRrrIAJli7u3A1UZQHSRUyrenz r9YnlBKxT/xufr0+YlnvhNmzC6fGOGPg+DWv6YYsKaepWM4MtPt4SZ8OwmGUhSrRKg3YZvdOVZr rcg== X-Google-Smtp-Source: AGHT+IFZbM8cukPN8xrEPWCmEvmIVa0hspDzfqR9/uQbMDXcIueCj/k5JgcFqB326LDruTDoIrGeiyLQyKs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:2342:b0:1fb:56c8:f273 with SMTP id d9443c01a7336-200ae4d640bmr1732425ad.1.1723232621226; Fri, 09 Aug 2024 12:43:41 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:13 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-2-seanjc@google.com> Subject: [PATCH 01/22] KVM: selftests: Check for a potential unhandled exception iff KVM_RUN succeeded From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Don't check for an unhandled exception if KVM_RUN failed, e.g. if it returned errno=EFAULT, as reporting unhandled exceptions is done via a ucall, i.e. requires KVM_RUN to exit cleanly. Theoretically, checking for a ucall on a failed KVM_RUN could get a false positive, e.g. if there were stale data in vcpu->run from a previous exit. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/lib/kvm_util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 56b170b725b3..0e25011d9b51 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -1719,7 +1719,8 @@ int _vcpu_run(struct kvm_vcpu *vcpu) rc = __vcpu_run(vcpu); } while (rc == -1 && errno == EINTR); - assert_on_unhandled_exception(vcpu); + if (!rc) + assert_on_unhandled_exception(vcpu); return rc; } From patchwork Fri Aug 9 19:43:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759216 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15A851662FD for ; Fri, 9 Aug 2024 19:43:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232625; cv=none; b=p1A7RzGzYSxObY0B/sBooEtDKc1a+AWl0Qz4nJzEUA/n8ByRmlqGtlYKvtNTshTc6qw6zlNGo27DzQuyGMKTu8PNrsq1Piy9gRm0WCIBH/T7k8ua9idqbE9INKY4VjJQYteq+BhA7HWOxncXAPjxzSp2lPsac5om33kXq7FVBCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232625; c=relaxed/simple; bh=ht4r0fbuo5n7i8loKFQk+Ce1717qM4Ak8JQynsJUkR0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g7zRfYaaJL2mPkMu5f8BeE9GvR29kmOiEUbJa6c4lXX+wOyXqdE5LjaYiHOMRLaHUJOnIJ5T0NJJTecE8VEoYAnCuZb8s5awyNKFMP4uRViUwF6SW0qTaMkIf7Dn2qrZA722zxr3eczSzBSfWAjIqNspBczM3Om9qzOK0sxbRTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=voYt1Ial; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="voYt1Ial" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e05e9e4dfbeso4337679276.1 for ; Fri, 09 Aug 2024 12:43:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232623; x=1723837423; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=D+L1OtzzgTK4mUsnsbZ07IxjRBCo4OaeWvC+DT5ZIuI=; b=voYt1Ialpznoltp+hT0r4u1JwjHZY1y6xSPgveyofeQ5rOrSTAn9PM/SZUcxvf3mXa OwXnZBTpRwL9oVbaed1gH8oLH6gs1OmqFrgbu3PXhfYmedF5wsmXxZxRivSHNJwhg4mu 4aFkpsfbQwmVBD3B1orVO6Bfu83R8oe3vGrxeggaz58ewESOpmAObo9dL1Sh5bpw1Rv0 aYN0SBcOHO/oyROXa9T4r95ujEyIV69d3UeGRvvnx0kLqaaBh5626FSYDX8YTv7UGFCX 0XWFUOAOTycrBthgxv/mHQJgwWfvi9b/u9Daw7Cp3hvxj+3EU4wZpcNaWd6HnHxWfjSY T7Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232623; x=1723837423; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=D+L1OtzzgTK4mUsnsbZ07IxjRBCo4OaeWvC+DT5ZIuI=; b=O8DCAanvOA9/atJKYG98IU4j7KLKSlR8Wytv5JAKY3/poeh3K7cB9PsDf5mFAz/f79 gjtHOTirixF7Uop8PK384KS0ZIUOB5sFHxehsBuWYaDNmKzxDsJ4UfWkFBpTIf5t7ybr AnzYLn6VG/gEaGxCZoqTwqllHDlh5k5PDPmDmDmdY/2sE3V5egdsDYk8dhiBQm3LKXo6 nhZ47J+2ifVhvy7s7OEeXcY5BICD7SpVwXI8KeeRyR7I8FISozZdu6rApFVPs09LB+rf 3BAKtDOaipd+noMJb18zyFqMtW84JWtt+7prN/MYsnRl4w4K9G3WAZoczU8wWCg86LJw +ONA== X-Gm-Message-State: AOJu0YwpKA/q+V62a93y6/O/07P4z50zUEADJ0nyDImHXlx8huSchQae J2U/qUk0pIhPs5U7iY5AnMhw7ZFBxiwegfXKp7iPWXwukJMa5XQVFx6qZCPvYAus9zZZW1vrLp9 m3A== X-Google-Smtp-Source: AGHT+IEKeKIaPW8YLP4xWw2FBfo2QHuCebzeo0dHE5fX5nmyiW5fDNnzekf775+ZUqeRP6mPt7GOOOJZQOw= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:f0a:b0:e08:6c33:7334 with SMTP id 3f1490d57ef6-e0eb9a10ac7mr19255276.8.1723232622954; Fri, 09 Aug 2024 12:43:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:14 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-3-seanjc@google.com> Subject: [PATCH 02/22] KVM: selftests: Rename max_guest_memory_test to mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Rename max_guest_memory_test to mmu_stress_test so that the name isn't horribly misleading when future changes extend the test to verify things like mprotect() interactions, and because the test is useful even when its configured to populate far less than the maximum amount of guest memory. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/Makefile | 2 +- .../kvm/{max_guest_memory_test.c => mmu_stress_test.c} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename tools/testing/selftests/kvm/{max_guest_memory_test.c => mmu_stress_test.c} (100%) diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile index b084ba2262a0..cc25f19b62da 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -138,7 +138,7 @@ TEST_GEN_PROGS_x86_64 += guest_print_test TEST_GEN_PROGS_x86_64 += hardware_disable_test TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus TEST_GEN_PROGS_x86_64 += kvm_page_table_test -TEST_GEN_PROGS_x86_64 += max_guest_memory_test +TEST_GEN_PROGS_x86_64 += mmu_stress_test TEST_GEN_PROGS_x86_64 += memslot_modification_stress_test TEST_GEN_PROGS_x86_64 += memslot_perf_test TEST_GEN_PROGS_x86_64 += rseq_test diff --git a/tools/testing/selftests/kvm/max_guest_memory_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c similarity index 100% rename from tools/testing/selftests/kvm/max_guest_memory_test.c rename to tools/testing/selftests/kvm/mmu_stress_test.c From patchwork Fri Aug 9 19:43:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759217 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF0F51684A7 for ; Fri, 9 Aug 2024 19:43:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232627; cv=none; b=RhOpziUGvWrqYiB+zZvtos66naGMJe6oX4dA9uRSQdMfHbKiWsaP85sSyoATOCb9QyEdiNBp0OcJz43p69CtOjGT7BNBUYG5OodaQS/QxEY8KUkmRqG5aCl9I7evZ0UUY6r+egAsrYh0yVXKzhMMuNu+4l7ysaYBSBsV00cnLIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232627; c=relaxed/simple; bh=i8/Z9P+b877HKYf3foOn3WZsd81L9lIIe5UnyEoK280=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VQf0laUYwgYsdjABYtFiPzSZ1+GffaathB+U2nhHpcScY9UTpvuFac9CMmNIBryZOMZxl17+dbogTldvhEXEuoEWAIXxu82CC4T2GJ1SAPKQ7xMVaZisprnP/H43ryULUYldHMHE6XixX1nl153aNPTnBwR45voSaSfFF0MSn1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2+mwTNZ+; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2+mwTNZ+" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7c3da34af55so130130a12.0 for ; Fri, 09 Aug 2024 12:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232625; x=1723837425; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ZhbtdXG7GxIZJlUwxv6/PwIOZfF88Cf7tN+DbOKZy1o=; b=2+mwTNZ+PaQ22lVNLKw4Jx9yQ9nbPlrzD0GvqMLNbAJH+jPnGWDPgdyTRr3GNDIHUt vjAmVRB3ZE9KMFwZ81pKpHzIodwGcckOLyja9jacLw+Mq94dED4USn1ZNHz1uY8DpOZo 79Gm5nrinALn9DsjNSu+G+2/5y2jfixyD2z+ACQzrJwWL1VpDEFdhVRFjTUlVhDv9Jdi FCFDX4jVuPhDDwtZWG36dgr6WV9tDkbZTKczflJBH3QVFlvXbRQFkHW9P7a5Jjq5Kp08 mJoihL6PEwLRpNWFKMI6QfKqkyz8JjrYtc3typ3EN1hW5HrIWQ8u2y0wCi84JuItigKh G1XA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232625; x=1723837425; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZhbtdXG7GxIZJlUwxv6/PwIOZfF88Cf7tN+DbOKZy1o=; b=JR0fYG5gYBKBKgliFDBgh0kxUW802xXme2gYJgMMxeZWcaS9QqewsSNUHnQEeYxG5/ bLTV/S/L2FA17uMBbF9EGDM6Crbg+ZjnBKeUf4lTW2bRKPObHCrTYZjJKqIiHEp+hutI qySNFrWoQ6Du9WPUedvvCcoczRUBAtgEp19etT+uWU+Ei0Ri9xC4BmAqWeg7Gm5M2gE9 wpDN6Nr0o5bTp/7tdsMOa4hvvY8Qt7VBzMB9413nJBr/eGo5wiWC4ka6EFpV7mgioCnc XmR//XNA5cnqPJj9FmGLFoKgjYRrrApzJI97IsOqrTF1UjT/8z68C+z98HQ547rp+LtQ BOHQ== X-Gm-Message-State: AOJu0YzmcT2lMCECjj6VVqLlotNlG8zFQy4/r5Ef7O3E0pjtdXvzEg4a XMsI9zRcMrpZnfW6fiNVXqPNRHHPkTjIg+oIMt+A1XHaIOKnoDRdpqAZixH03ZBm6MUpiq1ZvLe 5vQ== X-Google-Smtp-Source: AGHT+IEJSGntFoEairVQfIondbRS/wPjqzJ9gWC9vEesMP0+8PhPR1ut4xtIPVXbgE0J+qJ9SIwm32dkTPA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90b:f81:b0:2c9:81a2:e8e4 with SMTP id 98e67ed59e1d1-2d1e80605e9mr5545a91.5.1723232625089; Fri, 09 Aug 2024 12:43:45 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:15 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-4-seanjc@google.com> Subject: [PATCH 03/22] KVM: selftests: Only muck with SREGS on x86 in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Try to get/set SREGS in mmu_stress_test only when running on x86, as the ioctls are supported only by x86 and PPC, and the latter doesn't yet support KVM selftests. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 0b9678858b6d..847da23ec1b1 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -59,10 +59,10 @@ static void run_vcpu(struct kvm_vcpu *vcpu) static void *vcpu_worker(void *data) { + struct kvm_sregs __maybe_unused sregs; struct vcpu_info *info = data; struct kvm_vcpu *vcpu = info->vcpu; struct kvm_vm *vm = vcpu->vm; - struct kvm_sregs sregs; vcpu_args_set(vcpu, 3, info->start_gpa, info->end_gpa, vm->page_size); @@ -70,12 +70,12 @@ static void *vcpu_worker(void *data) run_vcpu(vcpu); rendezvous_with_boss(); +#ifdef __x86_64__ vcpu_sregs_get(vcpu, &sregs); -#ifdef __x86_64__ /* Toggle CR0.WP to trigger a MMU context reset. */ sregs.cr0 ^= X86_CR0_WP; -#endif vcpu_sregs_set(vcpu, &sregs); +#endif rendezvous_with_boss(); run_vcpu(vcpu); From patchwork Fri Aug 9 19:43:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759218 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A607B16B398 for ; Fri, 9 Aug 2024 19:43:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232629; cv=none; b=LDax1hwd9k8mAK9tjX9IbK36pEgGn7vNPwo0NKoMIpSDe7PM1gfhFHZGy6DL8eLNIxPduVUMo0ypDDFolVVTH9ohRLI3iuj3YSsHfuiMda0ftx03M2r/GoTRQixMGT1VZzNlu+Q+Vqrnj8x8Y0RGRhM1rGZZsqeBCp+pjegPtEQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232629; c=relaxed/simple; bh=2ubMLB5DYODq+qnXHnLMuf0I7PP+FeHp7xjE64E2dt8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FcYj/fVs/P19pDBQAq/r8znKf+Fzp1ZuWuqNAB3M2+lHIg/hCwTS5B8wdH8LiNtqGVWB/a8sQS+TGVwG+OD9AUUv3zc7BmIkVQQP6MmLlQygqah0HoVO42yUTeF1YKXN5f9rTsNabudXjyWkM860iH/O1MosiUh3Eek8y4evhwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uho2lfO3; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uho2lfO3" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-710ca162162so2146693b3a.0 for ; Fri, 09 Aug 2024 12:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232627; x=1723837427; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=RH8NStSlBVyK9g+UsmyXuI40nq9wzYPRU0gcFL0VkzM=; b=uho2lfO3JEKa3BYwurP6bC28fXllFmtwsLtKqkYwXKuVhG9iGLmF2aaPR5Odju1Ebl Z9NS24VN5EgBnwC8jBTgeHCFw2AsVyYNmazjHtqug5NluFgb9zdsyV3lRidSK7IbdXzO SCBUzgu7UERbgaLMluS8YD20m2GpAIJeCkYKVxXWoViB3+m1ymd9pJ3TY9AQlWsLi3mG ukBmfSIkISSc7wZvKIuD2A98IXIKxXMdV0Wo3kFxs+Vue5h2yajrz24/2pvUgYvrWhhP qXu6rPM4PbVbmOA+Q0UCHXc5EYZRTr0eYAybkGBmS0e7gFq+rVkpau1v/mDziY9M4tpz B86g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232627; x=1723837427; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RH8NStSlBVyK9g+UsmyXuI40nq9wzYPRU0gcFL0VkzM=; b=FRvFh97/FMwv2EzyHVflU8bMjeSPj+0+Qec047W1DeRP86sJmlYQu1Sd6qv0gWQSpR /hREmjJy+6obKbd6x2pVh1ACq64Fk3XZVSPXOQYQ1OryuKe8+vb/XinuEEG7D0jLNjZ1 dUkYD/mvty4Z7kB8C5fsEnexU8aRZxnKbA6KyKMSfQbZ2sm+nSi74AKYKT136/29Eqb5 vh0QrfE0bzwNoOUgT0FuUOLptjTkSIZD/A3vUhh108d96Utlerp2uaPtwP1qjTcJoGmx 5ICh8jYNJuUqYU7LwaajsiawdF3uWgm1As0FZY6CttGs9OwGB7fZWYdRXMoOAvKNanLe bHHA== X-Gm-Message-State: AOJu0YxPpXuws2yNaAX/KWkVkKDH41WhJ+zPZKYl9uWx/lEvf3/WD2sr TNyxJJXnyQG6KESYv7rI6mEzulZ0wMoevDiSPFs0QNwUa64wVzIeJaC9PWnGO11lopd8ZF/XfYq jnQ== X-Google-Smtp-Source: AGHT+IFF/ikXFYmI7zWUXbZoAyAwoVgAHirCcT99AraR5wR32CSYiy6TQQF1iEJF/4Gcjt4kmzWk21e9a/Y= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:6f12:b0:70d:30a8:abaa with SMTP id d2e1a72fcca58-710dcb31a13mr28290b3a.5.1723232626683; Fri, 09 Aug 2024 12:43:46 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:16 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-5-seanjc@google.com> Subject: [PATCH 04/22] KVM: selftests: Compute number of extra pages needed in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Create mmu_stress_tests's VM with the correct number of extra pages needed to map all of memory in the guest. The bug hasn't been noticed before as the test currently runs only on x86, which maps guest memory with 1GiB pages, i.e. doesn't need much memory in the guest for page tables. Signed-off-by: Sean Christopherson Reviewed-by: James Houghton --- tools/testing/selftests/kvm/mmu_stress_test.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 847da23ec1b1..5467b12f5903 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -209,7 +209,13 @@ int main(int argc, char *argv[]) vcpus = malloc(nr_vcpus * sizeof(*vcpus)); TEST_ASSERT(vcpus, "Failed to allocate vCPU array"); - vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus); + vm = __vm_create_with_vcpus(VM_SHAPE_DEFAULT, nr_vcpus, +#ifdef __x86_64__ + max_mem / SZ_1G, +#else + max_mem / vm_guest_mode_params[VM_MODE_DEFAULT].page_size, +#endif + guest_code, vcpus); max_gpa = vm->max_gfn << vm->page_shift; TEST_ASSERT(max_gpa > (4 * slot_size), "MAXPHYADDR <4gb "); From patchwork Fri Aug 9 19:43:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759219 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8610A16D4F5 for ; Fri, 9 Aug 2024 19:43:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232630; cv=none; b=r5SZdmUbbWpRsI5TFK6jPKye5WOva8vFZvh2sJVPR29wzVbf46UQIpZrbDhvHT4NHc0UB2YrvZ0jbilO2B26U+7/VJfRSQRANXLqnSZPBH9txfEs5IrjmtIKi+zI2lrc0LhDRK4Qva6COdLH4gnhhjvphAfDb2DXPYs9kMUdMIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232630; c=relaxed/simple; bh=LVFq/UF8yA/HGcn66G7X9qPCu7tUDBlRE4IGgs3dGtA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=abkEfyapnOwTL2UZ9ecf2p51u+US9Vsxq8iBvkq0EElKc2mpfIOY22pXTPcyPFWJymsEwXBEs53+AxHKfWQciKGJcLnljkWprfcV+PHoc3ujt4jzvQCL2gz17oJMti2B7pvUxD6hs7xDAkwMzq84k/CPnofLd4XI0BUMNd4/cx0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4NU4x8JE; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4NU4x8JE" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1fd5fe96cfeso24956335ad.0 for ; Fri, 09 Aug 2024 12:43:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232629; x=1723837429; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=SWkGT60Gokzaw7JUaOakLjtXVj6aQ6BddU74vUMlOq4=; b=4NU4x8JERO+C70hLXuiNZ493SWVQUvaPJSy0P/sZKZUWmW+DkZeJWFA/nX7Axu2bxF F6KJs7VmmEJUAwxuJavd4YOQii+X7qWhRew3ld0Xewk/0NUlXi/8snlu1UI2SgbVneOz 6ezwWRyJjGuDw4Tn7wR+q9FgaHAiOtP5UxW72NWDIfr//rLW6QA4Q+NjOUbmq9NQrgC6 dHSePuhHidAyoeNkAxw1wybz/fA/Q+zJoUGQeqNX/nYkgGm+acNKooqQAegVc4e1zTy4 LOH60kpWare2laFB7anvkw++H3Y1AX+sYhIVUA68QGxPOpoWGdTOKYQ5q/KgDtv7nr0d 0M2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232629; x=1723837429; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SWkGT60Gokzaw7JUaOakLjtXVj6aQ6BddU74vUMlOq4=; b=Te7BZn9DNq9oiLFU3issrgtvU+VpSrNMctZRAdM3kkJCF3A/YeD+DseLOBfxPz9NqN CANiZX2RwStX2F/rHJbUp+LgCXEPaREeYhllvWBik4h6aGdG0AY71KeqlDBb1XQytsPP 0bnbe9q2KIlY4mcdW4FJ2aaOLHpoOA+3fjmssHWe74KaGgSerea0Y7uPgVhZP+9DLD++ FREYG2mdxjDraKSd1TuVBRG105cGa7PK0SJYEl4/MhJd389jduXXZi6v4SAt2AhI9xj5 xy/A6q87wwYQM5wRm3W93OolF9aNHx2UbQlD8/9QXRsjrP4qcVErDp4ahXHuTe/X//d6 9hWA== X-Gm-Message-State: AOJu0Yz782JV8TcLJwYLkrImp195qFwu9r/yQuGZQADpV5YoBE5LBB9z Q0qUfL9780yHKFbcjCL+ChfgH+a1Dg+zptao91rkbOfmUd9uZ22N8d/kx5TkjyocjRaMO8uKJ4I COQ== X-Google-Smtp-Source: AGHT+IFmZCXNvXRh9QF9VYGRkhlcjb88/DI8HKkyVC2+5y45yUvR2nURAEut79pR9l0FdvfAx89nEJfuzPI= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:d482:b0:200:62bb:e22c with SMTP id d9443c01a7336-200ae5d8874mr822625ad.12.1723232628645; Fri, 09 Aug 2024 12:43:48 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:17 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-6-seanjc@google.com> Subject: [PATCH 05/22] KVM: selftests: Enable mmu_stress_test on arm64 From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Enable the mmu_stress_test on arm64. The intent was to enable the test across all architectures when it was first added, but a few goofs made it unrunnable on !x86. Now that those goofs are fixed, at least for arm64, enable the test. Cc: Oliver Upton Cc: Marc Zyngier Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile index cc25f19b62da..4198df29503f 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -174,6 +174,7 @@ TEST_GEN_PROGS_aarch64 += kvm_create_max_vcpus TEST_GEN_PROGS_aarch64 += kvm_page_table_test TEST_GEN_PROGS_aarch64 += memslot_modification_stress_test TEST_GEN_PROGS_aarch64 += memslot_perf_test +TEST_GEN_PROGS_aarch64 += mmu_stress_test TEST_GEN_PROGS_aarch64 += rseq_test TEST_GEN_PROGS_aarch64 += set_memory_region_test TEST_GEN_PROGS_aarch64 += steal_time From patchwork Fri Aug 9 19:43:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759220 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3535616D9DE for ; Fri, 9 Aug 2024 19:43:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232632; cv=none; b=ZsjsEVGDM78oknZ/Tn7K9YRTbDmTz+ddJQ1whfqG46UgQSE4irSHK/S2+tXRaZ599QhlAWlbpa3XNgFtLe0CG22ufYMwduyXIYsd5lWpn4yzB5+nj42bL3INKq07KLL03g/2fajf7uo/cT3dsJl0j9Zo8r3np93sySfRDtkCB9Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232632; c=relaxed/simple; bh=WEKb1xlIGs0k4iOZ5Yx5UQ9S4hkU1zI6zoLoK8JwUP8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j4SEkU2EXqiIURcNgxbx8OXoaKI4IjW/ZIo9P65b/wc0ttbFowRIorbZF81K+kF1Nxv0cEPf0QW0sgwxCWKjZqCYyaIy9C02PJO0yl5kD63RZ27+DyqE4GUCHWdKIO1ug8FyG0j2To8IwNg8wYkOwFC+qFRyyXGycILcpBNcG8Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4cGYP35V; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4cGYP35V" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7a12bb066aaso1830235a12.3 for ; Fri, 09 Aug 2024 12:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232630; x=1723837430; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=qOWEA+gf18ibEg+1IgBuUDOlia9YtX+EopmbGWf1wCE=; b=4cGYP35V9J9edTKAVY6ReehrY14PvgOdHA6GXtdUdpGI4d4RmtIWAj6+uHnGIZnNl4 7WlH1DZd57Ia6pMgOrgJsgmvjsDijx6MwJR33tcIT98ZwOz9W0qkao+m2DuwYoAQdMwu WgpJe9kKQMm9Kmcz2gfjfXPRXJsgZ1Gk7K5LWurw9AQHRck7waZVmWxw4/JuQQWVYhec XdLoc/hIYfcPNQkrHjks3Rrgsp1suuE2ez8Lhxvd2bddXJumBvVSeFS8FzQtFb5S3LaP P6kPeeZ2eGN0Ten0EpFhF/PqrPiubwOg73pyOPANOwCkuh6otJXQOUoKzhu/QMWm+nh4 oW4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232630; x=1723837430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=qOWEA+gf18ibEg+1IgBuUDOlia9YtX+EopmbGWf1wCE=; b=ABiuEiiLXsmiNB13Zvmo7u9t9gSAMoklRuEQkRtW3xNqxdrB7edb9Hjb9PvORIEC6n tncnb+4Jc85rYBCxBKSrQan5ahGEP0Np7GJufdU9skH8qFtmjESZVYUc1gMhFG0S4EFp EiesS9mSevCJ8ImOwTCH1GyHhhcZXEVmP/mOb3bOPM7/7+ew3NGE7O0T+lSkCavQLzb7 OphAuGHUF3zPSZzUPsWJA3pH3A/ns1jJpgKAjr13aHSaJKJrjBrVVsFD3oZD+nvYS6Qh LiOjOIe/IPY+VBCmrE51L6WpsJ0mdH4VlEOZ1OpfPPa7v5absvJLzSbIq7bSU6ijf1wO dFLw== X-Gm-Message-State: AOJu0Yxw9lWOpn+UekLh/F5c/5ndZpD37Hp3tAWXphSY2OfO6MKravST MzkcOOfjA4lDBYyxoh0SrPr6qDzc4IDusbUEG1Pp8nooVnXCyyFMLj1OTdboRljha7uaQsK4ZjF iZA== X-Google-Smtp-Source: AGHT+IEN/4gRYyT6fFtEw5/PSlX3VvEJMGy4wvRZ57TDQ5LWVk26M6iloQiU8f0vV2+Ypg0y5mNrr4U+8Bc= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:6f49:0:b0:6e6:a607:19d4 with SMTP id 41be03b00d2f7-7c3d2c7fa08mr4698a12.9.1723232630436; Fri, 09 Aug 2024 12:43:50 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:18 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-7-seanjc@google.com> Subject: [PATCH 06/22] KVM: selftests: Use vcpu_arch_put_guest() in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Use vcpu_arch_put_guest() to write memory from the guest in mmu_stress_test as an easy way to provide a bit of extra coverage. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 5467b12f5903..80863e8290db 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -22,7 +22,7 @@ static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stride) for (;;) { for (gpa = start_gpa; gpa < end_gpa; gpa += stride) - *((volatile uint64_t *)gpa) = gpa; + vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); GUEST_SYNC(0); } } From patchwork Fri Aug 9 19:43:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759221 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32A6216DEDC for ; Fri, 9 Aug 2024 19:43:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232634; cv=none; b=r5xMZTWPYPLkFIo67fM6lsBNAyowoYUGlvkemv6d/rhW451KUQju/Y4srr+i/q7QGHC88FLjK6pMhCgyMEZylP5mkmbHfUZ+ePkNYVpwynUZHqp9Klz7IIfoFi3fT1a/+b/tmuJmANz7FsUQenEf79QM3FyZeYnUCelznjIYuiQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232634; c=relaxed/simple; bh=oqLsHduvmYrRBvhT430Njd0wP4bVpgHOh7c0T70VHPc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kHOlmGERieW0UnYTwDO6IGChEbJY1a1TgYKPEDjc9aGGkQNbbKCNFtD7CvUJ/xQ225hCnRrXCIfqpt31xKJ4F0Lgsy3Gl1qAmVxvtsOrZDJn0ZQGGHeSTGMcI5H0ioP+ODyKKJfmeTuOiaxQQqLW6s8PM63R1MDMJKXySHgvqPU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3G3qEOen; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3G3qEOen" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7ad78c1a019so2151102a12.2 for ; Fri, 09 Aug 2024 12:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232632; x=1723837432; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Cvey1ZPxH+FTyjcmf9WeApEHMp7j68Ugrd4yPwlOS4s=; b=3G3qEOen1ah2pii+2eNgPqGvrKcUEnJVE0wQ8m3axxVsmLNGatC5zxOV4DBk5fmf7z POhd0EXVYHSm7I+1KvHCGagkHkJVBRHvv8e6bWDdfGbeJEq3KnYmLbLyZLJhPwkmG73P fILFf06DLCT0ZpGVfv7bRjclIpMwhKGFx5hE5jRx6PSBkA5mccvyVz7FjV85qNrbVO35 woRAHl/MFwTCmjDlMDRB0r/sXUMYoVrR0ie6/CQzZjJt8lj/TA81RFsB+8LSDNd2Sfdu +hGF4odoebHk8n6hCDpJB+zZQOUZuZbiw2iQAAi3PChVP/mTmy6gGmDTY/PHiXZKkCnd rJuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232632; x=1723837432; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Cvey1ZPxH+FTyjcmf9WeApEHMp7j68Ugrd4yPwlOS4s=; b=lAVPUm4mxo/4oXALp1gSmXtzSestxK2SQ2Or3kckYOgNt8uJvkA69BWe2QgWFO2CDg NhkOnbHtoXgXB3d++avVt0EKaZ0R+wJuUTeVBCvYvqrsRziLxGeAJnRJF3mZnR8nmZlN gsxJ7Tnc9mJPrJ6QeTbRQgYfgzrvQyvo5pyusZdFae8kY7OZIPfpgLqnVy7+NY/20qRp 7nlVPZu3mqWTbOyWmuMSEc5U2hW6/I2OmyCatimLwFkDiL8iKz84pXjoq2+xipCYsJi3 YvhNRpuDBnuHpm9j5S/tBRPTrG4vZ1N/lZBmah9eDxKBehz7Ecu8SnJQPp9HFvldRPDL hYZA== X-Gm-Message-State: AOJu0YxxL4VI0toTZ/H78SNn3wi2urFMPpiSRyqIVi9YXyJKNvcNKvJm AIqbMFZCcg3pBK1AV8ecT2NA/oUm8w9j5iLgy32kzTPvjPs0jnQ9E8f8JhpNYRhNJshgx18xS8T KwQ== X-Google-Smtp-Source: AGHT+IH+toPWCiPDFAxpzyjZiz0XzGzpUA6+VQs+q838rMwjQmsoPXGyEJ4VZpEN+6XxrI30IFrHifQqDGo= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:2352:b0:1f7:3bb3:abe2 with SMTP id d9443c01a7336-200ae5eee52mr1182405ad.12.1723232632391; Fri, 09 Aug 2024 12:43:52 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:19 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-8-seanjc@google.com> Subject: [PATCH 07/22] KVM: selftests: Precisely limit the number of guest loops in mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Run the exact number of guest loops required in mmu_stress_test instead of looping indefinitely in anticipation of adding more stages that run different code (e.g. reads instead of writes). Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 25 ++++++++++++++----- 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 80863e8290db..9573ed0e696d 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -19,12 +19,15 @@ static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stride) { uint64_t gpa; + int i; - for (;;) { + for (i = 0; i < 2; i++) { for (gpa = start_gpa; gpa < end_gpa; gpa += stride) vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); - GUEST_SYNC(0); + GUEST_SYNC(i); } + + GUEST_ASSERT(0); } struct vcpu_info { @@ -51,10 +54,18 @@ static void rendezvous_with_boss(void) } } -static void run_vcpu(struct kvm_vcpu *vcpu) +static void assert_sync_stage(struct kvm_vcpu *vcpu, int stage) +{ + struct ucall uc; + + TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC); + TEST_ASSERT_EQ(uc.args[1], stage); +} + +static void run_vcpu(struct kvm_vcpu *vcpu, int stage) { vcpu_run(vcpu); - TEST_ASSERT_EQ(get_ucall(vcpu, NULL), UCALL_SYNC); + assert_sync_stage(vcpu, stage); } static void *vcpu_worker(void *data) @@ -68,7 +79,8 @@ static void *vcpu_worker(void *data) rendezvous_with_boss(); - run_vcpu(vcpu); + /* Stage 0, write all of guest memory. */ + run_vcpu(vcpu, 0); rendezvous_with_boss(); #ifdef __x86_64__ vcpu_sregs_get(vcpu, &sregs); @@ -78,7 +90,8 @@ static void *vcpu_worker(void *data) #endif rendezvous_with_boss(); - run_vcpu(vcpu); + /* Stage 1, re-write all of guest memory. */ + run_vcpu(vcpu, 1); rendezvous_with_boss(); return NULL; From patchwork Fri Aug 9 19:43:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759222 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42CD116F265 for ; Fri, 9 Aug 2024 19:43:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232636; cv=none; b=JwnbeHMA1fqsqeCjGQ75TetXZ5p+Mt5Q1n94pa1ChQE5wiZ+SChfRCNMe3kB2dHAWrqgG0N5uhFTbEsX6b1Lu1Lqn0Bhn/tZTSwbOa9X6IgZtav8Ao8Wz1OrTlCNJSvPHG9XyrlPY43z0exz0BBSC9sAj4ZA6oYhbY2qScd5eqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232636; c=relaxed/simple; bh=JMLriyucuH1C15GtvvibdoIfA8+6so79QMDBwKK8/UQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HFPNlBpSGeHyBOpxwYG53wYlNrBmfnxpwfBd+mmkyi0BQa2I72oUzUtD/+oDQWcsKG/8/a4VYr4+Wl23R/MMfvdkymBqS7LTWAaeOC4gG0/MZ0IZ3hQiYtM9H09wu2JJ2oo0yDq0uy5f+cGfyxI32fqgkXLR10k3gdhxUUUga7g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yJzVP8fp; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yJzVP8fp" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2d1989d103eso3012763a91.2 for ; Fri, 09 Aug 2024 12:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232634; x=1723837434; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=KtOd9aj4hqeGD1AC+o3mLs+Sgxz7FerwNOXhm7lsLkU=; b=yJzVP8fpammFpUwlRKxnR30bVqCTcPr6UQpxuw18PmiM5fSSjMw+yv4gKxL4pi3sqF zhHLJAzpPap6QdBRQO6yPeYO+2Sdloc0CW5kdXRlnbNToXyyQgxA2yHm9Tv9beKhM3fA njYEpKrjTfqEZUZ2Cbu2moRRTsH1NRhAsExnPh27+dmmEEoUYkhcD6gr/fuVSkaoroAP aejQhHfhgbOH+AcKZNqGr4lHfca35Vk0WCVTq/mVCtgEX1NOplJGnEoJJ0jTwC9pgu4O +1i5/m0EGXGqbldIFrQgbVih2h0K3ZEWMEt7SZoQKVxn2QpgeiZDZAT2Hff9oVJ/eUXh UsXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232634; x=1723837434; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KtOd9aj4hqeGD1AC+o3mLs+Sgxz7FerwNOXhm7lsLkU=; b=L5uP9WssUx5b1MhTZ2jKmLJN2ItBEL8j6N1SxkxoUrH3u6yE30ESuC3EjLsJcWlSdu xJfLm0f/NBE2i7TPDmP6er+YwR9aQ/FpygobysdTM3wFg4nz9kkdVykhHHFMVIzIySxO /HzDn4wKOOY3zdQ91g17GiTxtgfTVkGUaVwKhd/HIpukNB1VZBNQt1HqKUrzZRt8zBse y64EKndGXeUuRWpduPqaj7YO+NAd3rvMSgfehu9FhA9EQDCKBkVpUu577SMWn51fbCWo ipXk0LJleNnhIa8bhfPiWGdLX/y/HQPm8W9OiXf2gqsmPQ0w5x8zv7dZ6m9pWuXRit0Y UMuQ== X-Gm-Message-State: AOJu0Yy+CLcVMowDA4Xa/8HALohqO/npDvlwRN/RqSiuwn8AP5vGzOjs IQPAKaBD9Zkam8mLcamQeIGM2gRNi/jeeDAo3jmWMEO1k55HJB7oCYf+ksCAWgUWl/q3CokWSn8 aAw== X-Google-Smtp-Source: AGHT+IGvAmg8+tqrGrCIZoGsSdnG8ZSFhwx3qgY9wt9WSt5Z7jq3JEgK8XnEDH316ti3bUud1crZuHLg05M= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:a88:b0:2c3:1985:e9c3 with SMTP id 98e67ed59e1d1-2d1e80b80femr12946a91.3.1723232634383; Fri, 09 Aug 2024 12:43:54 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:20 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-9-seanjc@google.com> Subject: [PATCH 08/22] KVM: selftests: Add a read-only mprotect() phase to mmu_stress_test From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Add a third phase of mmu_stress_test to verify that mprotect()ing guest memory to make it read-only doesn't cause explosions, e.g. to verify KVM correctly handles the resulting mmu_notifier invalidations. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 22 +++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 9573ed0e696d..50c3a17418c4 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -27,6 +27,10 @@ static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stride) GUEST_SYNC(i); } + for (gpa = start_gpa; gpa < end_gpa; gpa += stride) + *((volatile uint64_t *)gpa); + GUEST_SYNC(2); + GUEST_ASSERT(0); } @@ -94,6 +98,10 @@ static void *vcpu_worker(void *data) run_vcpu(vcpu, 1); rendezvous_with_boss(); + /* Stage 2, read all of guest memory, which is now read-only. */ + run_vcpu(vcpu, 2); + rendezvous_with_boss(); + return NULL; } @@ -174,7 +182,7 @@ int main(int argc, char *argv[]) const uint64_t start_gpa = SZ_4G; const int first_slot = 1; - struct timespec time_start, time_run1, time_reset, time_run2; + struct timespec time_start, time_run1, time_reset, time_run2, time_ro; uint64_t max_gpa, gpa, slot_size, max_mem, i; int max_slots, slot, opt, fd; bool hugepages = false; @@ -278,14 +286,20 @@ int main(int argc, char *argv[]) rendezvous_with_vcpus(&time_reset, "reset"); rendezvous_with_vcpus(&time_run2, "run 2"); + mprotect(mem, slot_size, PROT_READ); + rendezvous_with_vcpus(&time_ro, "mprotect RO"); + + time_ro = timespec_sub(time_ro, time_run2); time_run2 = timespec_sub(time_run2, time_reset); - time_reset = timespec_sub(time_reset, time_run1); + time_reset = timespec_sub(time_reset, time_run1); time_run1 = timespec_sub(time_run1, time_start); - pr_info("run1 = %ld.%.9lds, reset = %ld.%.9lds, run2 = %ld.%.9lds\n", + pr_info("run1 = %ld.%.9lds, reset = %ld.%.9lds, run2 = %ld.%.9lds, " + "ro = %ld.%.9lds\n", time_run1.tv_sec, time_run1.tv_nsec, time_reset.tv_sec, time_reset.tv_nsec, - time_run2.tv_sec, time_run2.tv_nsec); + time_run2.tv_sec, time_run2.tv_nsec, + time_ro.tv_sec, time_ro.tv_nsec); /* * Delete even numbered slots (arbitrary) and unmap the first half of From patchwork Fri Aug 9 19:43:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759223 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67A2C170849 for ; Fri, 9 Aug 2024 19:43:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232639; cv=none; b=OycwKFXedDVk6yYnkDd/pthYBWA8T7WESKz6PN6razokj7C50u3yjar/cexLM/dlRv8W03AM27EYk1PlAFNnW3g4UtMwbad5VaeeKSAy0sHSYy+pLVKQbjw+pSMRjhEWGxbwQtqx7sW3eKgjbnVU0nhyTlkBBKieiqc8sWc+pjI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232639; c=relaxed/simple; bh=WbIw5XKazA4cY8kYGA63IriSB3FyI1oMjErI9b0li8o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UxSKzQy+1u/cEn5nL71PmJtwp5Exqn85SwZP/z/h4m0JTOTIErUbRCrvshxZsRwMi2s9uJoasXz1QY3z9cUXpN6IZryrsmOofJBpnU5lzu9ttd5InYMvbg3msawMz6OysYNh17KLgfcYNHgndSnmEGhGbAxblpGx4jj0gpWYaAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gU6jeuVi; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gU6jeuVi" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-672bea19bedso56719937b3.3 for ; Fri, 09 Aug 2024 12:43:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232636; x=1723837436; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LA7LUE9c83FojgXGi3HkbRYYkaUFQd5wswdwV1F5jBI=; b=gU6jeuViOpEtFfoHDpTbdPptIImL9ZovcOCdszkOZsjx27gvFGSMuOaL1Nk4YRFdaW TBPjZo61b9A+tImg6G/19nlAI4Y1aXROj4xy04zWKcd7oKoqDVNUgVZ91wHiuLaoX3CX SET3qiE3+UqPd2JHDY8NxwAX2PEW2H8NjsqjPt8UWDCtXyL6EYu7EDlxQDL6UL2dWi7F ahB22lv6mIat8iSud9GoKZjbEsJEqBOfVhiY71XVAICbCw+u0u2Mh5vo8qkvNwPM0YdM lMcUBkQTSmFmBN6Toq8hGr2VFtCdcToPGgOwr4KQ5HiS9lRRHGsnlsISb1W/HihqFq2z c/9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232636; x=1723837436; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LA7LUE9c83FojgXGi3HkbRYYkaUFQd5wswdwV1F5jBI=; b=RaLmpaDGVVKcAkm6Rp6+Bg5QcDsAmn7p/wtkiBWcF++gLEXdvJMvcCsARzQxxrXJ+c gNDfXLtEpRJpD3SkxIIQDA6y5pjGL+z5uUqlgWP3xGvv05dG9w+P3NdixWuXEdVVmAAl t8RdyOqQCAS0sMbHjMBXcKu2zo5vXhtcvsJmcIohHvDzPlo9FE+zZlSO38IY4DIbk1PL fmCuXRXYXtzirJpkkWJz+NHq13eynJApQ/sM/KKcM9eiy8DIJ1hT5U1CzH7rhg1LtJjR Zmet1+7L/7zANQ5AOnwtieMXWBd9v+8K5c6ZDyz1oEUIokQcP7ASpjsOFCQy72u19JUI 55RA== X-Gm-Message-State: AOJu0YzuG57VAMAH2TduvPrVJIeQ50TQI+YWuDPM9zSVxOT98H8ZD9sh hD6V4WcvT12vHOeqzxSMop5VJ/UfxqXH1l2H1uOg6iY/WIQrriOa8Kq4AtsYLGpnv26k941X+hG aFA== X-Google-Smtp-Source: AGHT+IHdm1tAiwdHoiER1evk1B2BVo/sVRdO+cs/X5rKSY/fp4P6+azgk/JjvaFk5hFSNW8l0gSAdoLXOSY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a0d:f084:0:b0:697:9aae:1490 with SMTP id 00721157ae682-69ec4a1937dmr228707b3.1.1723232636462; Fri, 09 Aug 2024 12:43:56 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:21 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-10-seanjc@google.com> Subject: [PATCH 09/22] KVM: selftests: Verify KVM correctly handles mprotect(PROT_READ) From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Add two phases to mmu_stress_test to verify that KVM correctly handles guest memory that was writable, and then made read-only in the primary MMU, and then made writable again. Add bonus coverage for x86 to verify that all of guest memory was marked read-only. Making forward progress (without making memory writable) requires arch specific code to skip over the faulting instruction, but the test can at least verify each vCPU's starting page was made read-only. Signed-off-by: Sean Christopherson --- tools/testing/selftests/kvm/mmu_stress_test.c | 87 ++++++++++++++++++- 1 file changed, 84 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 50c3a17418c4..98f9a4660269 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -16,6 +16,8 @@ #include "guest_modes.h" #include "processor.h" +static bool mprotect_ro_done; + static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stride) { uint64_t gpa; @@ -31,6 +33,33 @@ static void guest_code(uint64_t start_gpa, uint64_t end_gpa, uint64_t stride) *((volatile uint64_t *)gpa); GUEST_SYNC(2); + /* + * Write to the region while mprotect(PROT_READ) is underway. Keep + * looping until the memory is guaranteed to be read-only, otherwise + * vCPUs may complete their writes and advance to the next stage + * prematurely. + */ + do { + for (gpa = start_gpa; gpa < end_gpa; gpa += stride) +#ifdef __x86_64__ + asm volatile(".byte 0xc6,0x40,0x0,0x0" :: "a" (gpa) : "memory"); +#else + vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); +#endif + } while (!READ_ONCE(mprotect_ro_done)); + + /* + * Only x86 can explicitly sync, as other architectures will be stuck + * on the write fault. + */ +#ifdef __x86_64__ + GUEST_SYNC(3); +#endif + + for (gpa = start_gpa; gpa < end_gpa; gpa += stride) + vcpu_arch_put_guest(*((volatile uint64_t *)gpa), gpa); + GUEST_SYNC(4); + GUEST_ASSERT(0); } @@ -78,6 +107,7 @@ static void *vcpu_worker(void *data) struct vcpu_info *info = data; struct kvm_vcpu *vcpu = info->vcpu; struct kvm_vm *vm = vcpu->vm; + int r; vcpu_args_set(vcpu, 3, info->start_gpa, info->end_gpa, vm->page_size); @@ -100,6 +130,49 @@ static void *vcpu_worker(void *data) /* Stage 2, read all of guest memory, which is now read-only. */ run_vcpu(vcpu, 2); + + /* + * Stage 3, write guest memory and verify KVM returns -EFAULT for once + * the mprotect(PROT_READ) lands. Only architectures that support + * validating *all* of guest memory sync for this stage, as vCPUs will + * be stuck on the faulting instruction for other architectures. Go to + * stage 3 without a rendezvous + */ + do { + r = _vcpu_run(vcpu); + } while (!r); + TEST_ASSERT(r == -1 && errno == EFAULT, + "Expected EFAULT on write to RO memory, got r = %d, errno = %d", r, errno); + +#ifdef __x86_64__ + /* + * Verify *all* writes from the guest hit EFAULT due to the VMA now + * being read-only. x86-only at this time as skipping the instruction + * that hits the EFAULT requires advancing the program counter, which + * is arch specific and currently relies on hand-coded assembly. + */ + vcpu->run->kvm_valid_regs = KVM_SYNC_X86_REGS; + for (;;) { + r = _vcpu_run(vcpu); + if (!r) + break; + TEST_ASSERT_EQ(errno, EFAULT); + WRITE_ONCE(vcpu->run->kvm_dirty_regs, KVM_SYNC_X86_REGS); + vcpu->run->s.regs.regs.rip += 4; + } + assert_sync_stage(vcpu, 3); +#endif + rendezvous_with_boss(); + + /* + * Stage 4. Run to completion, waiting for mprotect(PROT_WRITE) to + * make the memory writable again. + */ + do { + r = _vcpu_run(vcpu); + } while (r && errno == EFAULT); + TEST_ASSERT_EQ(r, 0); + assert_sync_stage(vcpu, 4); rendezvous_with_boss(); return NULL; @@ -182,7 +255,7 @@ int main(int argc, char *argv[]) const uint64_t start_gpa = SZ_4G; const int first_slot = 1; - struct timespec time_start, time_run1, time_reset, time_run2, time_ro; + struct timespec time_start, time_run1, time_reset, time_run2, time_ro, time_rw; uint64_t max_gpa, gpa, slot_size, max_mem, i; int max_slots, slot, opt, fd; bool hugepages = false; @@ -287,19 +360,27 @@ int main(int argc, char *argv[]) rendezvous_with_vcpus(&time_run2, "run 2"); mprotect(mem, slot_size, PROT_READ); + usleep(10); + mprotect_ro_done = true; + sync_global_to_guest(vm, mprotect_ro_done); + rendezvous_with_vcpus(&time_ro, "mprotect RO"); + mprotect(mem, slot_size, PROT_READ | PROT_WRITE); + rendezvous_with_vcpus(&time_rw, "mprotect RW"); + time_rw = timespec_sub(time_rw, time_ro); time_ro = timespec_sub(time_ro, time_run2); time_run2 = timespec_sub(time_run2, time_reset); time_reset = timespec_sub(time_reset, time_run1); time_run1 = timespec_sub(time_run1, time_start); pr_info("run1 = %ld.%.9lds, reset = %ld.%.9lds, run2 = %ld.%.9lds, " - "ro = %ld.%.9lds\n", + "ro = %ld.%.9lds, rw = %ld.%.9lds\n", time_run1.tv_sec, time_run1.tv_nsec, time_reset.tv_sec, time_reset.tv_nsec, time_run2.tv_sec, time_run2.tv_nsec, - time_ro.tv_sec, time_ro.tv_nsec); + time_ro.tv_sec, time_ro.tv_nsec, + time_rw.tv_sec, time_rw.tv_nsec); /* * Delete even numbered slots (arbitrary) and unmap the first half of From patchwork Fri Aug 9 19:43:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759224 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D273B194C62 for ; Fri, 9 Aug 2024 19:43:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232640; cv=none; b=oppEHR6KuDcYCbDOX3uk08oMNyOkbDIrdTNbEFLmM4Oqx8gLIh0jLyX0whO0qeFhvgZuhT90eb9vste1QuIEJ1sRqsO3J+LXffNSMHbwf29qvJ7fUtXeqKhT774weADQHo2420dVQYMy5OXQmzsiUHu448EzhlQj08mVE3Bl0PQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232640; c=relaxed/simple; bh=+GylkI/uIWrUOvqPBl8m0ev0LDK+4fi2QwLYgo7xKSQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FX7QN8GZpbz2EZ+w5pGpzqaJpOLS552CFQE8quPIveotnI4nCTzEVCI8lcOkosGg8r2EwNt/x/FioHMdrlxW8qmr+g6tiNEOmpIH36Tif/Zf56kUhGceEa+3+enzGU/Mj91H+16CIktZKWZLAdsWJHrQdWPmeB0pfUyvApQ3Lpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pDHcY69F; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pDHcY69F" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-6507e2f0615so2292876a12.1 for ; Fri, 09 Aug 2024 12:43:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232638; x=1723837438; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=FfIoOq8ezL3lF0COKJQzO4R/UUZXUE0fWXr809zSI60=; b=pDHcY69FQBoDOpmgX9GU7Ao38U8UCZoRyRFL1OayPd0huY0pcxvBJuTkt0CslTH2wY O1EGGgK1qMUm2uoa5VPUydKHIg7jVq9xVdG4BjoWrhNkaG9IHT2Fr2s0PVUoZCrLYv5o W0ZUuBLoFPVsUuP6WBNrILPxBc3StUJN8q6ionxxWAqrbWsgHLO+kZHZEkK+e9kSgyd1 G/wn9NGJu/WUI0SGZ6qYxbueDpz35n5JbqwrL+Uy+hANhKxZxQvsNH3qVasxHkvcYAC8 46lsucWkoI1INjt10wLVyn//vGBcH9+sIlmMQ1wPuJq7spFdhw7d1qtQxnvmmpb+YL1X vF6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232638; x=1723837438; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FfIoOq8ezL3lF0COKJQzO4R/UUZXUE0fWXr809zSI60=; b=wymFPDO43uLae+lPPNcsLXdr16cD+Q8uXOk7sOIe6Oet8Y78iRsVlkmkQLMNQV6GTr GRgMjyCeKBsG/g+woQNS+d8733JVd1rHBFzndv/fX2gjT59sPni6mecoA8wDcQwI3c1q PeAHGTudCAK83wGkvjPuCsON7uNB8PAK996XhcxXax0VtateuMwjW7/pruCDbLA0LOTx eGIgMNQjFhtXS6YRxul8m4J9nvJc+D5WuImea8VpclKCFMzv/evLDnYEEDYk2NCvub3o 5liTpZ2220xfOoIEdOMvfaiFmiUsXnPtFsYBxeABsMeYZZ5dGxIHczNoBypyljsUYbbG KYMA== X-Gm-Message-State: AOJu0YzhQRTFfA8u//X9rVhHjw/MsCcds1JowF8BmK0kXTNu+F4fycaI 9ZgmDmDxHd5rYa/FuMkzidhyqMjTIuz9xkaIwSbjMBK6A5LXgQCjtu+6nRybS5BTALfdutNow3m qmw== X-Google-Smtp-Source: AGHT+IHI2Sj/+82ITfhQRaMFsQWLje/k3ROR9iGp71RmEKU0cgVARHxIz4EvnVRZ5brISpMtlPzaid6nWq4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e80b:b0:1fa:f9ac:8b4a with SMTP id d9443c01a7336-200ae4c9f0amr1621115ad.3.1723232638087; Fri, 09 Aug 2024 12:43:58 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:22 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-11-seanjc@google.com> Subject: [PATCH 10/22] KVM: x86/mmu: Move walk_slot_rmaps() up near for_each_slot_rmap_range() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Move walk_slot_rmaps() and friends up near for_each_slot_rmap_range() so that the walkers can be used to handle mmu_notifier invalidations, and so that similar function has some amount of locality in code. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 107 +++++++++++++++++++++-------------------- 1 file changed, 54 insertions(+), 53 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 901be9e420a4..676cb7dfcbf9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1534,6 +1534,60 @@ static void slot_rmap_walk_next(struct slot_rmap_walk_iterator *iterator) slot_rmap_walk_okay(_iter_); \ slot_rmap_walk_next(_iter_)) + +/* The return value indicates if tlb flush on all vcpus is needed. */ +typedef bool (*slot_rmaps_handler) (struct kvm *kvm, + struct kvm_rmap_head *rmap_head, + const struct kvm_memory_slot *slot); + +static __always_inline bool __walk_slot_rmaps(struct kvm *kvm, + const struct kvm_memory_slot *slot, + slot_rmaps_handler fn, + int start_level, int end_level, + gfn_t start_gfn, gfn_t end_gfn, + bool flush_on_yield, bool flush) +{ + struct slot_rmap_walk_iterator iterator; + + lockdep_assert_held_write(&kvm->mmu_lock); + + for_each_slot_rmap_range(slot, start_level, end_level, start_gfn, + end_gfn, &iterator) { + if (iterator.rmap) + flush |= fn(kvm, iterator.rmap, slot); + + if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { + if (flush && flush_on_yield) { + kvm_flush_remote_tlbs_range(kvm, start_gfn, + iterator.gfn - start_gfn + 1); + flush = false; + } + cond_resched_rwlock_write(&kvm->mmu_lock); + } + } + + return flush; +} + +static __always_inline bool walk_slot_rmaps(struct kvm *kvm, + const struct kvm_memory_slot *slot, + slot_rmaps_handler fn, + int start_level, int end_level, + bool flush_on_yield) +{ + return __walk_slot_rmaps(kvm, slot, fn, start_level, end_level, + slot->base_gfn, slot->base_gfn + slot->npages - 1, + flush_on_yield, false); +} + +static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, + const struct kvm_memory_slot *slot, + slot_rmaps_handler fn, + bool flush_on_yield) +{ + return walk_slot_rmaps(kvm, slot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_yield); +} + typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct kvm_memory_slot *slot, gfn_t gfn, int level); @@ -6199,59 +6253,6 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, } EXPORT_SYMBOL_GPL(kvm_configure_mmu); -/* The return value indicates if tlb flush on all vcpus is needed. */ -typedef bool (*slot_rmaps_handler) (struct kvm *kvm, - struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot); - -static __always_inline bool __walk_slot_rmaps(struct kvm *kvm, - const struct kvm_memory_slot *slot, - slot_rmaps_handler fn, - int start_level, int end_level, - gfn_t start_gfn, gfn_t end_gfn, - bool flush_on_yield, bool flush) -{ - struct slot_rmap_walk_iterator iterator; - - lockdep_assert_held_write(&kvm->mmu_lock); - - for_each_slot_rmap_range(slot, start_level, end_level, start_gfn, - end_gfn, &iterator) { - if (iterator.rmap) - flush |= fn(kvm, iterator.rmap, slot); - - if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { - if (flush && flush_on_yield) { - kvm_flush_remote_tlbs_range(kvm, start_gfn, - iterator.gfn - start_gfn + 1); - flush = false; - } - cond_resched_rwlock_write(&kvm->mmu_lock); - } - } - - return flush; -} - -static __always_inline bool walk_slot_rmaps(struct kvm *kvm, - const struct kvm_memory_slot *slot, - slot_rmaps_handler fn, - int start_level, int end_level, - bool flush_on_yield) -{ - return __walk_slot_rmaps(kvm, slot, fn, start_level, end_level, - slot->base_gfn, slot->base_gfn + slot->npages - 1, - flush_on_yield, false); -} - -static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, - const struct kvm_memory_slot *slot, - slot_rmaps_handler fn, - bool flush_on_yield) -{ - return walk_slot_rmaps(kvm, slot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_yield); -} - static void free_mmu_pages(struct kvm_mmu *mmu) { if (!tdp_enabled && mmu->pae_root) From patchwork Fri Aug 9 19:43:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759225 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE3D91990C3 for ; Fri, 9 Aug 2024 19:44:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232643; cv=none; b=CkFEu/i4lPBLVev1tLluyWJ+YNbH7vQVTj2fdZxMruRtBW4vIiLyLy19TgiwUkKZQAtxSHrJvHcTG603cBnwngQBIlNz8c0d8i0KulP3n6/jVAkoiJialc3dJSMxd1PVjlJwWb3WrenSq2Xst3UJT/S1COWUUpZxYWI8XmAIl+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232643; c=relaxed/simple; bh=Q4/0S9zjurXdMR/Xc6/7FhGHVIB+yXDEQA05A1JT85k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YTHnXO3VApeHEa0gHQTLEItqNj/Rs3FxDrgqTEzp0L57l9/u12txDA9EoNKhOIr8HkIxl+OUUdwZcly3Z7DtpwGmhn95FdP3wjzddYmwmWxGyBepqCxWqKJ+PwQnj06JeM/cbuN7YEXz2pdaQ+2t0eL44Acwdf/nc97AWIRr0CQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yOPfdfbk; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yOPfdfbk" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2cb68c23a00so3335752a91.0 for ; Fri, 09 Aug 2024 12:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232641; x=1723837441; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=RZnIwpm2DX6oLiBTd7LSzopcPIvohhjKZPunqiEaNiE=; b=yOPfdfbkTACJtGhMaGp8/glp73j/d1XutUuiT1/hOrk0OHG2Co/fiHKEI+gpRxzY7P HI7Ng6xM24W+fzHZk14tpEPSlhZM1IDhxlrj7p6VBLQhllP4gAh9ZQsBmRW1d83yMiv4 E3V4t34SmWHKgTNXoKZq6qoDyotqBwVv0cpRq5Z0bxkoZ8mmVhctJP4O/ljfqaCamKt4 fQzCBe610zoObazsxBFS8IxbwVgoURES985c6oS0jIas/xkEsZHYXqAa3/625u/xaJLq becO5cYeFiS+KjddHSpsu8gGzOnHu8l4J/IkG5fSzEehedxR7GiqWSrDaKI8Etsn55sn 9GyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232641; x=1723837441; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=RZnIwpm2DX6oLiBTd7LSzopcPIvohhjKZPunqiEaNiE=; b=nBTLyhZA35Ovo8y5DR5pq2BMkp0ku7GLeTAb3lyV31LTjQX01alRLjIIqxblibI89i X2mqp3+DsFvfEqC3CLlet+X1buGBUl9AbSLqb3ZLdBdU5vJkhjzSfWKJLfq1XSVZGhKs fJTW3wNcBvrw8qQ2RruJHtdTpZmILZhG9DvTNeCX7dVrN+QnGovoLlV9fpVj9rHB4TbA Dy/WjVniq32zwDVyb3Qf4o9eOgqxeKR4x7rSHVYZbUlt1aoDeMKBUhTScqIEHzWwkNIQ 03plDvJeH9xBbYXF3jhTAb46g4k4fDgFVer1m/BAJlrrO1uDp+9hMcTANDY4vBPD/Fzv xFBQ== X-Gm-Message-State: AOJu0YxpZRU6mwwqI2gVoECQClvXbv4XCfYqIefr3BoMhHfQ7t5JLJle UhDHioKvqPko5ra735H6AvduMVuuF4ZmJZ4rvYmNbnzZXHJmm1kEwS0J7lzLKc/saDsAVa+Q+B3 vew== X-Google-Smtp-Source: AGHT+IEVMPLIN0gtxbuCx2PJeNj0X98BYvziqTLoHJHpzW4rfW380DSvpsuYxUtuP8Md6WisaeszFgzwm+A= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:68c4:b0:2ca:7f5a:db6c with SMTP id 98e67ed59e1d1-2d1c4c45398mr42369a91.3.1723232640835; Fri, 09 Aug 2024 12:44:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:23 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-12-seanjc@google.com> Subject: [PATCH 11/22] KVM: x86/mmu: Plumb a @can_yield parameter into __walk_slot_rmaps() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Add a @can_yield param to __walk_slot_rmaps() to control whether or not dropping mmu_lock and conditionally rescheduling is allowed. This will allow using __walk_slot_rmaps() and thus cond_resched() to handle mmu_notifier invalidations, which usually allow blocking/yielding, but not when invoked by the OOM killer. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 676cb7dfcbf9..a5a7e476f5bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1545,7 +1545,8 @@ static __always_inline bool __walk_slot_rmaps(struct kvm *kvm, slot_rmaps_handler fn, int start_level, int end_level, gfn_t start_gfn, gfn_t end_gfn, - bool flush_on_yield, bool flush) + bool can_yield, bool flush_on_yield, + bool flush) { struct slot_rmap_walk_iterator iterator; @@ -1556,6 +1557,9 @@ static __always_inline bool __walk_slot_rmaps(struct kvm *kvm, if (iterator.rmap) flush |= fn(kvm, iterator.rmap, slot); + if (!can_yield) + continue; + if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { if (flush && flush_on_yield) { kvm_flush_remote_tlbs_range(kvm, start_gfn, @@ -1577,7 +1581,7 @@ static __always_inline bool walk_slot_rmaps(struct kvm *kvm, { return __walk_slot_rmaps(kvm, slot, fn, start_level, end_level, slot->base_gfn, slot->base_gfn + slot->npages - 1, - flush_on_yield, false); + true, flush_on_yield, false); } static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, @@ -6528,7 +6532,7 @@ static bool kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_e flush = __walk_slot_rmaps(kvm, memslot, __kvm_zap_rmap, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, - start, end - 1, true, flush); + start, end - 1, true, true, flush); } } @@ -6816,7 +6820,7 @@ static void kvm_shadow_mmu_try_split_huge_pages(struct kvm *kvm, */ for (level = KVM_MAX_HUGEPAGE_LEVEL; level > target_level; level--) __walk_slot_rmaps(kvm, slot, shadow_mmu_try_split_huge_pages, - level, level, start, end - 1, true, false); + level, level, start, end - 1, true, true, false); } /* Must be called with the mmu_lock held in write-mode. */ From patchwork Fri Aug 9 19:43:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759226 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EF4B1991AD for ; Fri, 9 Aug 2024 19:44:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232644; cv=none; b=DYhi2iYrGS+6k8FKd0HikDWHMxRn6BAtIiV/uGG/MAs74RN31hQMCk3MFcCqu8Vwq8nsPHS+fo5tUwaWhfVmQe/et/jb2WKgFGL/hjOEHPFZIqWqBx0UddpliyPXgfLGvUUYLCx4h25V6YnFnC510ezkAP1qV8E4GJmauU5y10I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232644; c=relaxed/simple; bh=Ky1N4NEPPXYQ68tuOtFA6BSNneI8rYbZCaa3EnVYquw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CTHpx/6BNTShMCprSBiXHNUX4xXC7j1rvcwTnyKl5GHI2ZQ/i0uq3ID1RLxqKfX5RSXAz7FiALbPMGzEJjxXzAqZ/yfamHJkJwUCb9PLY0ok6avPmO3YcVlo/qe5R/J+Ww+wB0dyE4w6ZDF7cniNtx0u4QJTks9byYa1kq6sgtU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EIVw8Rlu; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EIVw8Rlu" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-71050384c9aso2463609b3a.1 for ; Fri, 09 Aug 2024 12:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232643; x=1723837443; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=KngE1DTnwbCcWR9aNh2KG9Wx5wVMsPDCUrAPTcj/AmI=; b=EIVw8RluigmxDMVw8j0b2Ek+6tEzRzap1dFnmb+4h3VXdMwG9dCQXL32e14jOFjvGq TcFb2itAsoBKoowacw1mYdnagzEUOnCjwVwOQhe1bzOZVFE341gyKyp0mmv7NVmfQVj9 qXcgXymrr4xrro76/E+7OWbsfhBXynxxJmc5JMTc95mvodngkABgAy2c8QrgeEGvDe2Q nJJ7FZD5Djp3HzAPlEoLWWovTtfnYyNTzl/CdC9ggrO3be9KFyFFT35hmaiGbuDW4dXd d1SH/3cGzB+KqD0CRd4uebTb7oY6fYtB1S5SnO7MIis0GZR+pFNNHRVNsUsVzXdn1MqC +kfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232643; x=1723837443; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=KngE1DTnwbCcWR9aNh2KG9Wx5wVMsPDCUrAPTcj/AmI=; b=aSPx2t0CD+GC6DIexDHE5q68aofE0bJx1yO5VTki3YeP2WDnVYXGn95/DHeowC8glV mk1rFdPfWCUp0IS/kFn+IT/GFiyMTmzqFGtTfeI53ZXc+nrxF2GDj4L+YoLs3qX2HfmF iRlOYWHb7JC6rBZMEJpPKAYM0iMySLmnnjIZrv9xtyRBOyBy8SHBgc6Aop6tpmZyUgM7 y2xTf/CARyrrYc4eAz0qzgiojGAP9CoaaP73mAXU0cKZRcCfh4LLsZW4BouBpY488IKR OuK2ySeTNW0cTSArpiGR+N3zquX5EWEDZvxUrVssb9zpZhlTw5ij8FM1ygcESNj+o69p Ijug== X-Gm-Message-State: AOJu0Yy9r8T94WOQ1eXqd+HYoPvxd2JXKuDqh+sclJSFelyLhe9DWT6x pjyVcdg/YvAdq/vzn+O7xv8jYb/gP6NevAhnCAAuNknDExTDMFeY6Z/aOg1Yc3x3glfpuPNp1oL j6w== X-Google-Smtp-Source: AGHT+IG7ko0vuFPPXt3xcZarFz9eU+f5poN4SmWQJRgKUX/WSEyph3BvOPDm1891EeOCuHmelP3kV0TthVI= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:91e4:b0:710:4e4c:a4ad with SMTP id d2e1a72fcca58-710cc32523dmr145683b3a.0.1723232642725; Fri, 09 Aug 2024 12:44:02 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:24 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-13-seanjc@google.com> Subject: [PATCH 12/22] KVM: x86/mmu: Add a helper to walk and zap rmaps for a memslot From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Add a dedicated helper to walk and zap rmaps for a given memslot so that the code can be shared between KVM-initiated zaps and mmu_notifier invalidations. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a5a7e476f5bb..4016f63d03e8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1592,6 +1592,16 @@ static __always_inline bool walk_slot_rmaps_4k(struct kvm *kvm, return walk_slot_rmaps(kvm, slot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_yield); } +static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm, + const struct kvm_memory_slot *slot, + gfn_t start, gfn_t end, bool can_yield, + bool flush) +{ + return __walk_slot_rmaps(kvm, slot, __kvm_zap_rmap, + PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, + start, end - 1, can_yield, true, flush); +} + typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct kvm_memory_slot *slot, gfn_t gfn, int level); @@ -6530,9 +6540,8 @@ static bool kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_e if (WARN_ON_ONCE(start >= end)) continue; - flush = __walk_slot_rmaps(kvm, memslot, __kvm_zap_rmap, - PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, - start, end - 1, true, true, flush); + flush = __kvm_rmap_zap_gfn_range(kvm, memslot, start, + end, true, flush); } } From patchwork Fri Aug 9 19:43:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759240 Received: from mail-qk1-f201.google.com (mail-qk1-f201.google.com [209.85.222.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 783BC168487 for ; Fri, 9 Aug 2024 19:50:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723233006; cv=none; b=kHGvyIqQXG2MxX3liwqAaO+58XwOqZ8qCvDz75uY9QECzIyU02V23prt6SYnNgb/md97YOX+H3n/Q66kkjHmHBQGOIz1NsGE55QL+jBhvg9TUf3kc9uR4GI83OWV8rNkR4HAZ/VL4OwEGsnefxAe2dIR4Rz/fSH84z80+SFeDGY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723233006; c=relaxed/simple; bh=3xSCN07is/d7+9qa5tb05xLdmkaS8Uh8BYF+HVNaYMo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=N/LU2wrByjNf7Ns0UCavkDtTuH0MPfqqTMlQUVEFvDxfk1qFOGJyWMlNfyIA4QviKCquu18VAjCZKNxeLGrGy/f53awVQS2OkHG9ri93RnDbMBXKaot7eMY4BNC55S/u06hb1y7bHwvWgksBwD3o7LsBIxlDq/ZVgEmn2pmMyU4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gDUa+LhR; arc=none smtp.client-ip=209.85.222.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gDUa+LhR" Received: by mail-qk1-f201.google.com with SMTP id af79cd13be357-7a1d71b96c4so291395185a.0 for ; Fri, 09 Aug 2024 12:50:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723233003; x=1723837803; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:from:to:cc:subject:date :message-id:reply-to; bh=C/Kjdpw4GZGWgN0ZxVg8kngRskxTTrucp/2D4JfCR5g=; b=gDUa+LhRd+dlWZY5BBv6+kvl4gNb5yOeGvj1k0Om3jIN08z8k+iFlaJ8JNnmFoiSeV DOGlvR/N21/BYPKpwUALRB6sTs4usz6grYzWchQA/1gxeqLHeVAmfDEucouIPBY06yaV 5Dol41TUCL0cq/COjQ4rPMD0Hvcy+kHISBLUnMTQSZztFL8wg87hzwjsTotbrMtqIHbh z4CxWHHIhobLoDA/pmVphyPE5vO8bPpVcpICJTdWB2nqnUh+xXnel9zJ5kIDm6FLg8FH tTJ+xf+hHHrAGJYSn7YZgnThUDw1gENJ9Q6eRZHjFMJ6rMJYUH1cVCqtKKZ1WwpeXS4n dNAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723233003; x=1723837803; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=C/Kjdpw4GZGWgN0ZxVg8kngRskxTTrucp/2D4JfCR5g=; b=JsdYzrajUOBPZW3+SG0IubX316Ko9KwK+Ay+tOnO6/CYBgaivYO6gyO3gOK/1dEtAS OGQe9GN8ZvSh5pNADgE4RPog6UBZZsk2bp9XY0uTpMkmDHBhdKvq7CThBtJ+9GBLc9OB j+0S7MPmhxvGulc1PwHfo3IMmm2PewykUMAoGGHTfTgRd05UDCYdytlLdstji9Mmdzdq TrTcMEo6Q0jJgbAwJpb0ee8cTzs5acj8Bu5aJJASvoKJwJGTHAFfrIAAnjNkGi6wl5A5 AxM/Iv3M7D91clbzDSHaz/7DzKaDSX26qe2TgkOLOnZoXhE2N2VgrAvLTYBISu9zbCYz QOVQ== X-Gm-Message-State: AOJu0YxwPm6GQ+9gTLkXwoHBfKsUY4m3LeLlBjzKRtKWf2+OZqiglafS nZF6jxlPaRY5YkgIDVLTg0e3KTBNesUDffw7n4m5At76lBY1UCPEGVbUMC2j8KR1FuEFHbu2EjB bnQ== X-Google-Smtp-Source: AGHT+IHlnphgTRNzeVdx0O31fPI0LuneFaWMABXGnKSnF6B8xUcCs6tohBFzlE1Hi/BkKbGcyYXHhyb9HbY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:d507:b0:1fc:2ee3:d46c with SMTP id d9443c01a7336-200ae2549b2mr2179795ad.0.1723232644737; Fri, 09 Aug 2024 12:44:04 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:25 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-14-seanjc@google.com> Subject: [PATCH 13/22] KVM: x86/mmu: Honor NEED_RESCHED when zapping rmaps and blocking is allowed From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Convert kvm_unmap_gfn_range(), which is the helper that zaps rmap SPTEs in response to an mmu_notifier invalidation, to use __kvm_rmap_zap_gfn_range() and feed in range->may_block. In other words, honor NEED_RESCHED by way of cond_resched() when zapping rmaps. This fixes a long-standing issue where KVM could process an absurd number of rmap entries without ever yielding, e.g. if an mmu_notifier fired on a PUD (or larger) range. Opportunistically rename __kvm_zap_rmap() to kvm_zap_rmap(), and drop the old kvm_zap_rmap(). Ideally, the shuffling would be done in a different patch, but that just makes the compiler unhappy, e.g. arch/x86/kvm/mmu/mmu.c:1462:13: error: ‘kvm_zap_rmap’ defined but not used Reported-by: Peter Xu Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4016f63d03e8..0a33857d668a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1453,16 +1453,10 @@ static bool kvm_vcpu_write_protect_gfn(struct kvm_vcpu *vcpu, u64 gfn) return kvm_mmu_slot_gfn_write_protect(vcpu->kvm, slot, gfn, PG_LEVEL_4K); } -static bool __kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot) -{ - return kvm_zap_all_rmap_sptes(kvm, rmap_head); -} - static bool kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level) + const struct kvm_memory_slot *slot) { - return __kvm_zap_rmap(kvm, rmap_head, slot); + return kvm_zap_all_rmap_sptes(kvm, rmap_head); } struct slot_rmap_walk_iterator { @@ -1597,7 +1591,7 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end, bool can_yield, bool flush) { - return __walk_slot_rmaps(kvm, slot, __kvm_zap_rmap, + return __walk_slot_rmaps(kvm, slot, kvm_zap_rmap, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, start, end - 1, can_yield, true, flush); } @@ -1626,7 +1620,9 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool flush = false; if (kvm_memslots_have_rmaps(kvm)) - flush = kvm_handle_gfn_range(kvm, range, kvm_zap_rmap); + flush = __kvm_rmap_zap_gfn_range(kvm, range->slot, + range->start, range->end, + range->may_block, flush); if (tdp_mmu_enabled) flush = kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush); From patchwork Fri Aug 9 19:43:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759227 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F384199256 for ; Fri, 9 Aug 2024 19:44:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232648; cv=none; b=P1qUlB/FTdM59fToAbkYncOFr6yQcTaeDy4FEdQCBCW3RGOyEhQvoPlGGEwZj6AmLEFWcKzBYFzKKe5QypZFBQJzZUNix5ESLCvPivoLWh9laY0XNcqCWqBAQyVlLJCDLlOXmrlfbMP30XoEruxxeAJPLdvL4pRLm6w7Slae1mw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232648; c=relaxed/simple; bh=Ed1G/Nb2CnyIN7eoTZ0EijA3UdmeGBcXP9PKO/DqlAo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XtZAqXQHI7YyC2QlBp3mxThFaaEL/GtgnRTqDm9nr7p2DkoO4ZoOsTBI9Up5GnINQn9yWppZxWEVZU8lHH/LRt/nUloDnYU1Rghy4b5cdb95t5ElWvRT6hdNc+MvHbGa2aLXmwh/FzxHd4OjOjfQtwdhXMzZxgg6O4eu8/uBzVQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WJov4la9; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WJov4la9" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7a30753fe30so2302559a12.3 for ; Fri, 09 Aug 2024 12:44:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232647; x=1723837447; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=xgf0knUlsf9OwxiOIKoJyEqysjyCp256HfjOAsBxKsQ=; b=WJov4la9g6mT6plMsSYQXNdHAX9fpke2GXA9PibfRBbl/km/cDZ6eW/g5uljUZLvAL lNLemc0pucMcfpvsH3IdogpZWpONwLKd8tBK/GBCNq6lgjtI6uANf8AlLT3edj234NSL +uZt+QPjGBTWNdcLCCvZnPRFFBAJy/EKXjwHm5AW/Qw5iu1vUFG2icWu9+044H4FjRm1 hzvIb+qDAKcfpUZau8lLIi/7iknVionVOe75FxTM9NUT6MwZTxHdv8oDAPV7E0++nobf 4cW2h7jf7KzoDaAUl74AlvfOcgkrgy66YrFQEWCQPu4OPZwnCsKRofdLV/UF2+XpFQn1 ZMSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232647; x=1723837447; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=xgf0knUlsf9OwxiOIKoJyEqysjyCp256HfjOAsBxKsQ=; b=e53xUZskM/81z/KDTSKpMhecUwcZci3sQ1Cbrsk7MPM4Qm01jNgDfB1hMNLlqkynkj V0IPG+Ndh/aFRkEtcpcWmUJq/vPWEsf6oSGOQnJgPBEoGYP83poTyKXnHaDrLS2fggLr W+70DQzE7QkBlmQxKN4WnjAoW+Jw4UOVs599+RvFdStEIK1LKPLVRXc0p1uhOolQpXSQ CifSpHpSZSv82eHtdye0r9p6c1W1QLQvBgLZGg3ewkusddN48/g5QqaZg+WulKYxG1ee FUzRCUJke8Rqw8WZLGZ2HlYwvAWWAzechEqkQzETwM8L0HT72S1mwqJwTOsLJjWL1CUz XzGg== X-Gm-Message-State: AOJu0YyO1YkZVKHE+xkpokCoZ9QWA9PEqM6uBSVGzFgQt/hFFXjGgmHA 50P4ViuXqGZ1jsWmjxQBHxTLd70gmXVlYjki+LDtqP4zUb6cYcDQsAHtWGyztNn11xRp40LMn7G ZTg== X-Google-Smtp-Source: AGHT+IH1270TZbw7dwgOHkJEckbD2SAane4OZgA6vncKd5lKiIhglKNNCAxVffQiCers7rx90OBnP67l4os= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:6d46:0:b0:718:665d:65c9 with SMTP id 41be03b00d2f7-7c3d2bd2a84mr4911a12.6.1723232646717; Fri, 09 Aug 2024 12:44:06 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:26 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-15-seanjc@google.com> Subject: [PATCH 14/22] KVM: x86/mmu: Morph kvm_handle_gfn_range() into an aging specific helper From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Rework kvm_handle_gfn_range() into an aging-specic helper, kvm_rmap_age_gfn_range(). In addition to purging a bunch of unnecessary boilerplate code, this sets the stage for aging rmap SPTEs outside of mmu_lock. Note, there's a small functional change, as kvm_test_age_gfn() will now return immediately if a young SPTE is found, whereas previously KVM would continue iterating over other levels. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 68 ++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 46 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 0a33857d668a..88b656a1453d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1596,25 +1596,6 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm, start, end - 1, can_yield, true, flush); } -typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, - int level); - -static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, - rmap_handler_t handler) -{ - struct slot_rmap_walk_iterator iterator; - bool ret = false; - - for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, - range->start, range->end - 1, &iterator) - ret |= handler(kvm, iterator.rmap, range->slot, iterator.gfn, - iterator.level); - - return ret; -} - bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) { bool flush = false; @@ -1634,31 +1615,6 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) return flush; } -static bool kvm_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level) -{ - u64 *sptep; - struct rmap_iterator iter; - int young = 0; - - for_each_rmap_spte(rmap_head, &iter, sptep) - young |= mmu_spte_age(sptep); - - return young; -} - -static bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level) -{ - u64 *sptep; - struct rmap_iterator iter; - - for_each_rmap_spte(rmap_head, &iter, sptep) - if (is_accessed_spte(*sptep)) - return true; - return false; -} - #define RMAP_RECYCLE_THRESHOLD 1000 static void __rmap_add(struct kvm *kvm, @@ -1693,12 +1649,32 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } +static bool kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) +{ + struct slot_rmap_walk_iterator iterator; + struct rmap_iterator iter; + bool young = false; + u64 *sptep; + + for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, + range->start, range->end - 1, &iterator) { + for_each_rmap_spte(iterator.rmap, &iter, sptep) { + if (test_only && is_accessed_spte(*sptep)) + return true; + + young = mmu_spte_age(sptep); + } + } + return young; +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; if (kvm_memslots_have_rmaps(kvm)) - young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + young = kvm_rmap_age_gfn_range(kvm, range, false); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1711,7 +1687,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) bool young = false; if (kvm_memslots_have_rmaps(kvm)) - young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + young = kvm_rmap_age_gfn_range(kvm, range, true); if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); From patchwork Fri Aug 9 19:43:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759228 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 457B6199398 for ; Fri, 9 Aug 2024 19:44:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232650; cv=none; b=ov3zQ4QpP3nkrbipOzXvY1fk00UkGrysbbxiyfzoNLDzoSX3KBcqZNqaz9hgLvjYl7aJUgtcx3Fo+8niu3rBQYTp8PkPxmX+7AZWPT0bKas4TaL6yTx8Ht3cZ7Lluty3BIjJt9daer5jeb3lrr5A8YVAisF4fZHEv/Zhjw4eX4M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232650; c=relaxed/simple; bh=VG0ehnLRBNQjGjdZ62CiB3/PrgwQ8Ke9eyr0F1VXaGU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lD+bOJdwCx5XeSs2dLoKWN9Zn+preR2HDgHQvVkYqnKSUgNPikRgmkY/2yngs3xraHE6cEXFbs39ZsKujLbyMuBwAk4AoLc0ciqOsFBgij44lVJs2m+olQ5HY/tHhff38YFWMdVAiYjhiy5d3g9i6vb5tGsTE7OsbcTHk+47Jqc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3qlLhUcC; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3qlLhUcC" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-71050384c9aso2463664b3a.1 for ; Fri, 09 Aug 2024 12:44:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232648; x=1723837448; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Ai8EH7U6wTbiOPvc4tDQEcYQ+E/EtVyC6HGLhPebPF8=; b=3qlLhUcC97lB6vCpdUPWtkuyd4ObXkDIAeSa+M9klnKyX5L6H2S67M6HxaE6pIJQV6 VHdblS5NGUk7ROWxOvP08JH655dAuNTTe8bDHkbjBLuLlJWNHZOvNYsedBgMLnZ3Ih42 Ad+pyOTzngUWy5i90jrJOZ0+SSHdUBuePexfbsgSQEV1H8ND7msrLyv0YAvH8y7jBIWm oFzpkQT2F+GqtzX8yg8tr1Ydq6lDdvbJZAY+TqklSgsy00ybax5BxOMqpTSelcGvuWzp oWOsMgmdN5ioNYnLAAavhTjP8buGZ16QCyHZ6whtp79/Xf2+Tfxio2j6bc0BV9nowmCY BmAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232648; x=1723837448; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ai8EH7U6wTbiOPvc4tDQEcYQ+E/EtVyC6HGLhPebPF8=; b=MEiFdUuRknpLt5ojPVmCkMQTMWwbFec+3sIk5sPblHnDuEVKr4lQ0Vpzt6Vw5zC8Xv 8aUYLnoxCARY4SUyq8o1BWha6fdKBjDiC1pJtINZLTzH3V51AXGP6OUraFSTvYGgwSZm fWVkOQRRJ/eq7PU+2mYdDkf++qIGVRBCHOSFeiP0t7Z5iLMYLXk9mex9msMYG5c/LDDi 2W6iMyy771XUVHB2L9vUFaWr2mB6AqYPAQjiDPDBbmbem5EMKyIQHgZKH8Dik0tMwuS3 4fcttmBJB0cQ45WCcafyvmEgbeSSM2ndf5ROLwQhe3LJy42lgyaeHE44PmrN1iOLq2f6 5XcQ== X-Gm-Message-State: AOJu0YwY1TFcjkJ148ehMjiqHdAazg2EWMGz2+PiJW5CtCxYskhSJTnM 7rlXeYIfveS56KSL+quwUmD335814lsQwWL3YU/HyPmtW2Kso6N4cli8JTrRC6YeZmv+lClQvdN wNw== X-Google-Smtp-Source: AGHT+IHNNFbniYTOQTi7bihux9iGSGXPTBj4ES/WY0YDfyXV3hzhWFmxyp3a86iRwwurPT1lphBR34jXitQ= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:9167:b0:70d:1b0d:a15d with SMTP id d2e1a72fcca58-710ccb04a71mr122147b3a.3.1723232648511; Fri, 09 Aug 2024 12:44:08 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:27 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-16-seanjc@google.com> Subject: [PATCH 15/22] KVM: x86/mmu: Fold mmu_spte_age() into kvm_rmap_age_gfn_range() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Fold mmu_spte_age() into its sole caller now that aging and testing for young SPTEs is handled in a common location, i.e. doesn't require more helpers. Opportunistically remove the use of mmu_spte_get_lockless(), as mmu_lock is held (for write!), and marking SPTEs for access tracking outside of mmu_lock is unsafe (at least, as written). I.e. using the lockless accessor is quite misleading. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 50 +++++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 88b656a1453d..c536a069d6b9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -614,32 +614,6 @@ static u64 mmu_spte_get_lockless(u64 *sptep) return __get_spte_lockless(sptep); } -/* Returns the Accessed status of the PTE and resets it at the same time. */ -static bool mmu_spte_age(u64 *sptep) -{ - u64 spte = mmu_spte_get_lockless(sptep); - - if (!is_accessed_spte(spte)) - return false; - - if (spte_ad_enabled(spte)) { - clear_bit((ffs(shadow_accessed_mask) - 1), - (unsigned long *)sptep); - } else { - /* - * Capture the dirty status of the page, so that it doesn't get - * lost when the SPTE is marked for access tracking. - */ - if (is_writable_pte(spte)) - kvm_set_pfn_dirty(spte_to_pfn(spte)); - - spte = mark_spte_for_access_track(spte); - mmu_spte_update_no_track(sptep, spte); - } - - return true; -} - static inline bool is_tdp_mmu_active(struct kvm_vcpu *vcpu) { return tdp_mmu_enabled && vcpu->arch.mmu->root_role.direct; @@ -1660,10 +1634,30 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, for_each_slot_rmap_range(range->slot, PG_LEVEL_4K, KVM_MAX_HUGEPAGE_LEVEL, range->start, range->end - 1, &iterator) { for_each_rmap_spte(iterator.rmap, &iter, sptep) { - if (test_only && is_accessed_spte(*sptep)) + u64 spte = *sptep; + + if (!is_accessed_spte(spte)) + continue; + + if (test_only) return true; - young = mmu_spte_age(sptep); + if (spte_ad_enabled(spte)) { + clear_bit((ffs(shadow_accessed_mask) - 1), + (unsigned long *)sptep); + } else { + /* + * Capture the dirty status of the page, so that + * it doesn't get lost when the SPTE is marked + * for access tracking. + */ + if (is_writable_pte(spte)) + kvm_set_pfn_dirty(spte_to_pfn(spte)); + + spte = mark_spte_for_access_track(spte); + mmu_spte_update_no_track(sptep, spte); + } + young = true; } } return young; From patchwork Fri Aug 9 19:43:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759229 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C193199EA1 for ; Fri, 9 Aug 2024 19:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232653; cv=none; b=NNISY1D7AFeY4wvJf7uRKsUUd03EE0+a1qc6s5BadSkMVNWPfGmCCcSsEPSThLlljkE4aGbzW2XDwsxkX9xRA+Ga7T3tdx2s5eBa/vEcCVFh5Nt0WMq2BQtV6mknE76FOKUI7CguoGd/l1Uz439vTwzUr7O+spjyA/2WdDiN2ns= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232653; c=relaxed/simple; bh=aE+XjLSYWb149BZo4fuwEt6PAsJA4S1vOO2cjFVxeZw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hnY2466ezqSad8keXSE0bJVR+PQcrDnx0mXFR2G0YJDqq/lcnTQVGxsXzJGab3zQT80NFg/MwrDDHcnjqW0/rkWvmn7uYFatlXJYhJJHXv3wp4ZJ/NIZbDIHPegUvfMwVtKceqlkF/1bAXw7gGvhvpXhxL67SpiG8gTOpP1avzo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QQP9GjY+; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QQP9GjY+" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-66890dbb7b8so55878077b3.0 for ; Fri, 09 Aug 2024 12:44:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232650; x=1723837450; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=GD7gycGsmp7NjONVo+Jkt5X+cTf6SoGnpMVjud9Ddns=; b=QQP9GjY+WoiTj2kofWBAusfKONlPvEtPCxIeJuyN4ZihH7zFfe/XPLtryV2o8qn8mx R+rLn7JG10drMi/oFbfw/jQvdNNEaW+8M01e8GKin1KUWnDNUUCvogIGH4tkPYZu/yGO Ireh6K9jYRkb0mX+4rQVqNlhMz7vORYY300xEEPEU5KRkjeaJbgMskmxBOGMYHd9bxTS jlaHKa1+ves/F9zIqeFLSxWG8KzoX6NBNuQS0n3jCCPok6ulpT6zZMHEqzbxPdKwVUGA MTnv0imu4dlp45J7CsWYPuDA8Fn4di3vWDbWWV4y5h2lH2aCjFyMUK5EuA/4O34EnXph KZ0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232650; x=1723837450; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GD7gycGsmp7NjONVo+Jkt5X+cTf6SoGnpMVjud9Ddns=; b=lJHZRVyEC/ahU84DUHpak/RmFC9GISejML2Ati047hu/MMN8uaIj04eZFJvjdq62yY +CYcuhYsZJ1nRAvrSrZyedjRU1HGMranSnXa/MBUj2k5VzNYmbh+0AgspzmaiBgG/hWA BGeB3v/A9nxCsWONR5dr+fxqIVAui11y0361TB0alC8cJW+LEw/9PkNyvx0UB4c3Pbk5 OiIADiDIikdjS8f4VEXNitkAwpPnQ3o99xZs6WZHEZ4BZmnDx2WMSeLaXSRDrinSADqR zAxRXzqi9KCFVxxcmjzm+qOp+i8XQAjfZDSHZmHYzpELdl6yFonv5FQkNxPPuzw5SrDk YNJw== X-Gm-Message-State: AOJu0YzSAUekg37BrFAey1XN30qfWDnbrjWQkm/ccasdw/1jJYfMsKZp PGMfG/iBTok7KS/73pWkBSKOudNzSw6eCc08/SewaLa8UpfAmRLZSEDJafdk+rKRQ6oHKPiBuOY rUw== X-Google-Smtp-Source: AGHT+IG5SwO0DcW0swDPhRWsKN2wuZhEs3mbWTMQ2R+czhdomrVn6Ywvmw+UK+vkinS9d2PnRXVcF1Gcbp4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:690c:290f:b0:615:32e1:d82c with SMTP id 00721157ae682-69ec8d85594mr560077b3.6.1723232650446; Fri, 09 Aug 2024 12:44:10 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:28 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-17-seanjc@google.com> Subject: [PATCH 16/22] KVM: x86/mmu: Add KVM_RMAP_MANY to replace open coded '1' and '1ul' literals From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Replace all of the open coded '1' literals used to mark a PTE list as having many/multiple entries with a proper define. It's hard enough to read the code with one magic bit, and a future patch to support "locking" a single rmap will add another. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c536a069d6b9..73569979130d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -912,6 +912,7 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu * in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct * pte_list_desc containing more mappings. */ +#define KVM_RMAP_MANY BIT(0) /* * Returns the number of pointers in the rmap chain, not counting the new one. @@ -924,16 +925,16 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, if (!rmap_head->val) { rmap_head->val = (unsigned long)spte; - } else if (!(rmap_head->val & 1)) { + } else if (!(rmap_head->val & KVM_RMAP_MANY)) { desc = kvm_mmu_memory_cache_alloc(cache); desc->sptes[0] = (u64 *)rmap_head->val; desc->sptes[1] = spte; desc->spte_count = 2; desc->tail_count = 0; - rmap_head->val = (unsigned long)desc | 1; + rmap_head->val = (unsigned long)desc | KVM_RMAP_MANY; ++count; } else { - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); count = desc->tail_count + desc->spte_count; /* @@ -942,10 +943,10 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, */ if (desc->spte_count == PTE_LIST_EXT) { desc = kvm_mmu_memory_cache_alloc(cache); - desc->more = (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc->more = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); desc->spte_count = 0; desc->tail_count = count; - rmap_head->val = (unsigned long)desc | 1; + rmap_head->val = (unsigned long)desc | KVM_RMAP_MANY; } desc->sptes[desc->spte_count++] = spte; } @@ -956,7 +957,7 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct pte_list_desc *desc, int i) { - struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); int j = head_desc->spte_count - 1; /* @@ -985,7 +986,7 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, if (!head_desc->more) rmap_head->val = 0; else - rmap_head->val = (unsigned long)head_desc->more | 1; + rmap_head->val = (unsigned long)head_desc->more | KVM_RMAP_MANY; mmu_free_pte_list_desc(head_desc); } @@ -998,13 +999,13 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_head->val, kvm)) return; - if (!(rmap_head->val & 1)) { + if (!(rmap_head->val & KVM_RMAP_MANY)) { if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_head->val != spte, kvm)) return; rmap_head->val = 0; } else { - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); while (desc) { for (i = 0; i < desc->spte_count; ++i) { if (desc->sptes[i] == spte) { @@ -1037,12 +1038,12 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, if (!rmap_head->val) return false; - if (!(rmap_head->val & 1)) { + if (!(rmap_head->val & KVM_RMAP_MANY)) { mmu_spte_clear_track_bits(kvm, (u64 *)rmap_head->val); goto out; } - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); for (; desc; desc = next) { for (i = 0; i < desc->spte_count; i++) @@ -1062,10 +1063,10 @@ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) if (!rmap_head->val) return 0; - else if (!(rmap_head->val & 1)) + else if (!(rmap_head->val & KVM_RMAP_MANY)) return 1; - desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); return desc->tail_count + desc->spte_count; } @@ -1127,13 +1128,13 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, if (!rmap_head->val) return NULL; - if (!(rmap_head->val & 1)) { + if (!(rmap_head->val & KVM_RMAP_MANY)) { iter->desc = NULL; sptep = (u64 *)rmap_head->val; goto out; } - iter->desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + iter->desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); iter->pos = 0; sptep = iter->desc->sptes[iter->pos]; out: From patchwork Fri Aug 9 19:43:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759230 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C57019A280 for ; Fri, 9 Aug 2024 19:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232654; cv=none; b=KkKxLsYgoi3GYlZAZBY8WcXn1ScMoW5Im/BObqXtH7li/xDASLB5zjfR+vmT2BnoBX/UlRi4VFUEg+Md9hHCD/2K2WKoz6XwmB4o97QW+YPmrmjrHNYEHVzS4liRxsDhgp7UiuQ+BLLkDKG/4rkBJ664YZjJu5UEhUqT4McudqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232654; c=relaxed/simple; bh=KiAcec9zQkYgeR0W9i8MR49CVqVo71Q4X9zgQyrpB/0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tpnNzIyqzZzRg/vH20mJSz6t6DsyvFzGkukufVZ3sHMtkIROg4aJ1XMjv5FLABOcE68vaZwgLJFbK2fsUad+0VrYxjD4zYZNpLAL3Vqg+0ixlZ4NhQMFzbttCT/rfqJ6W7CUDEI1JfOBd7won2f3JDV+PZRE71jewG4hzmG2cMA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JeOmsrn8; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JeOmsrn8" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7a242496897so2350059a12.2 for ; Fri, 09 Aug 2024 12:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232652; x=1723837452; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LJq/GJ0moPmdVQ8FJN/LFNxKQSz1PiS8pWAScNqsM+k=; b=JeOmsrn85iNSMEOfGAbluITjOVjH7/ODm5wI5Xc5P6E/CDN5hcORuyFfFbjPhBAsne 3TASZGUndyPHczwmB/cfFV79gMgrpXLzwD5iXaOQrNIGXLQgboG+yVaI5a72DDjEpaXR rwMnwc0EYi612YYOmEc9OlpTKeOlyNuU/W7EaJCCSK7WFp6juLnrYwnuEMdANW36t34m 4+W8AC21Tg1UhUHUWxz/Ts8vCSIpJRrszFmJed8+LPAOIquxyGXMIg2w0sXVoHovIcYO wEePlIftCbuFQ9xewWaiKu+KmZR9MHT+wJH8A6XLUCRQRKn3jn26PQeOQHnl+Tq5zrAH JGKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232652; x=1723837452; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LJq/GJ0moPmdVQ8FJN/LFNxKQSz1PiS8pWAScNqsM+k=; b=m27UTK1olgsfLXaG7lWBe6z1/4Hfyj/ZO/3MwEhyN9cS6eSFF//0dU+ihw/kEJ8tQX Ge89xoMhjiJhDXJWFJuXY+Bz/PfFljkFWyEOgerujsHgKynpjXPRcU7plF/rMCCxT1Fw upFtFktrA8hjg5ND7qjyKVWa6LqhzvCe626eNtRqFl+va39lZqrncGe/ZKNfj7N7Sp5W +o6u3KjedPUnWjUoBD0jV3b9GVzSAP2k1i3nmwYqivoqBSMBa+KAqQx0bSEFhyzcf3yp lQ6lMp+ZJRiQnVk3Hh2EBdqZZgdqdXiaNRko6JZ9+7t00i1Z/eNC4VCCQsOxNSH4J9Oc Gy2Q== X-Gm-Message-State: AOJu0YxWywn5OaB7VoqKSOZPc2Bni2dMuSFckfCiQq3d3ql36GTZYIm1 YsDOC7asf6GIsS82EUo4V9MjZUExsIj0VDKTaYCkjxGIrkpWTa4xZJZ+8qt7t4jSF2k+Ab2vE7u 5Qg== X-Google-Smtp-Source: AGHT+IG9iS1KQa10Rf8CvbcbSW/02Vokoqft8BTRa6S8EoVMPZ81K0TgdGJVD2ND164posnnJw8HgWteDns= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a63:3714:0:b0:717:a912:c302 with SMTP id 41be03b00d2f7-7c3d2b78b02mr5155a12.1.1723232652349; Fri, 09 Aug 2024 12:44:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:29 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-18-seanjc@google.com> Subject: [PATCH 17/22] KVM: x86/mmu: Refactor low level rmap helpers to prep for walking w/o mmu_lock From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Refactor the pte_list and rmap code to always read and write rmap_head->val exactly once, e.g. by collecting changes in a local variable and then propagating those changes back to rmap_head->val as appropriate. This will allow implementing a per-rmap rwlock (of sorts) by adding a LOCKED bit into the rmap value alongside the MANY bit. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 83 +++++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 33 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 73569979130d..49c1f6cc1526 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -920,21 +920,24 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, struct kvm_rmap_head *rmap_head) { + unsigned long old_val, new_val; struct pte_list_desc *desc; int count = 0; - if (!rmap_head->val) { - rmap_head->val = (unsigned long)spte; - } else if (!(rmap_head->val & KVM_RMAP_MANY)) { + old_val = rmap_head->val; + + if (!old_val) { + new_val = (unsigned long)spte; + } else if (!(old_val & KVM_RMAP_MANY)) { desc = kvm_mmu_memory_cache_alloc(cache); - desc->sptes[0] = (u64 *)rmap_head->val; + desc->sptes[0] = (u64 *)old_val; desc->sptes[1] = spte; desc->spte_count = 2; desc->tail_count = 0; - rmap_head->val = (unsigned long)desc | KVM_RMAP_MANY; + new_val = (unsigned long)desc | KVM_RMAP_MANY; ++count; } else { - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY); count = desc->tail_count + desc->spte_count; /* @@ -943,21 +946,25 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, */ if (desc->spte_count == PTE_LIST_EXT) { desc = kvm_mmu_memory_cache_alloc(cache); - desc->more = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc->more = (struct pte_list_desc *)(old_val & ~KVM_RMAP_MANY); desc->spte_count = 0; desc->tail_count = count; - rmap_head->val = (unsigned long)desc | KVM_RMAP_MANY; + new_val = (unsigned long)desc | KVM_RMAP_MANY; + } else { + new_val = old_val; } desc->sptes[desc->spte_count++] = spte; } + + rmap_head->val = new_val; + return count; } -static void pte_list_desc_remove_entry(struct kvm *kvm, - struct kvm_rmap_head *rmap_head, +static void pte_list_desc_remove_entry(struct kvm *kvm, unsigned long *rmap_val, struct pte_list_desc *desc, int i) { - struct pte_list_desc *head_desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + struct pte_list_desc *head_desc = (struct pte_list_desc *)(*rmap_val & ~KVM_RMAP_MANY); int j = head_desc->spte_count - 1; /* @@ -984,9 +991,9 @@ static void pte_list_desc_remove_entry(struct kvm *kvm, * head at the next descriptor, i.e. the new head. */ if (!head_desc->more) - rmap_head->val = 0; + *rmap_val = 0; else - rmap_head->val = (unsigned long)head_desc->more | KVM_RMAP_MANY; + *rmap_val = (unsigned long)head_desc->more | KVM_RMAP_MANY; mmu_free_pte_list_desc(head_desc); } @@ -994,24 +1001,26 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc; + unsigned long rmap_val; int i; - if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_head->val, kvm)) - return; + rmap_val = rmap_head->val; + if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) + goto out; - if (!(rmap_head->val & KVM_RMAP_MANY)) { - if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_head->val != spte, kvm)) - return; + if (!(rmap_val & KVM_RMAP_MANY)) { + if (KVM_BUG_ON_DATA_CORRUPTION((u64 *)rmap_val != spte, kvm)) + goto out; - rmap_head->val = 0; + rmap_val = 0; } else { - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); while (desc) { for (i = 0; i < desc->spte_count; ++i) { if (desc->sptes[i] == spte) { - pte_list_desc_remove_entry(kvm, rmap_head, + pte_list_desc_remove_entry(kvm, &rmap_val, desc, i); - return; + goto out; } } desc = desc->more; @@ -1019,6 +1028,9 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, KVM_BUG_ON_DATA_CORRUPTION(true, kvm); } + +out: + rmap_head->val = rmap_val; } static void kvm_zap_one_rmap_spte(struct kvm *kvm, @@ -1033,17 +1045,19 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc, *next; + unsigned long rmap_val; int i; - if (!rmap_head->val) + rmap_val = rmap_head->val; + if (!rmap_val) return false; - if (!(rmap_head->val & KVM_RMAP_MANY)) { - mmu_spte_clear_track_bits(kvm, (u64 *)rmap_head->val); + if (!(rmap_val & KVM_RMAP_MANY)) { + mmu_spte_clear_track_bits(kvm, (u64 *)rmap_val); goto out; } - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); for (; desc; desc = next) { for (i = 0; i < desc->spte_count; i++) @@ -1059,14 +1073,15 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { + unsigned long rmap_val = rmap_head->val; struct pte_list_desc *desc; - if (!rmap_head->val) + if (!rmap_val) return 0; - else if (!(rmap_head->val & KVM_RMAP_MANY)) + else if (!(rmap_val & KVM_RMAP_MANY)) return 1; - desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); return desc->tail_count + desc->spte_count; } @@ -1109,6 +1124,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) */ struct rmap_iterator { /* private fields */ + struct rmap_head *head; struct pte_list_desc *desc; /* holds the sptep if not NULL */ int pos; /* index of the sptep */ }; @@ -1123,18 +1139,19 @@ struct rmap_iterator { static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { + unsigned long rmap_val = rmap_head->val; u64 *sptep; - if (!rmap_head->val) + if (!rmap_val) return NULL; - if (!(rmap_head->val & KVM_RMAP_MANY)) { + if (!(rmap_val & KVM_RMAP_MANY)) { iter->desc = NULL; - sptep = (u64 *)rmap_head->val; + sptep = (u64 *)rmap_val; goto out; } - iter->desc = (struct pte_list_desc *)(rmap_head->val & ~KVM_RMAP_MANY); + iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); iter->pos = 0; sptep = iter->desc->sptes[iter->pos]; out: From patchwork Fri Aug 9 19:43:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759231 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DB07168498 for ; Fri, 9 Aug 2024 19:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232656; cv=none; b=TDHpDIDP4d/u1nk7iyhWgPmkEzivn/c42nHPWUXTSRoZHkqrU7U939DjimEuHzMTbVm+uGAkO3XMsM6rMNvjjFAd6TAaUsLzq9hAI3wgOQWq8mvfHnw+YVzWyeDA57iIOkkAwWDAmc356gG5++FRDoxcSgZkUGGLy0yip0GyvNQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232656; c=relaxed/simple; bh=lGLMFgBOB+zT1nvZAyf/Zc7DkG5GaKVPXMH41Wm07lY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qayJVZY8a9GtCSVeUY8R74Lyrzoh68iJA8JaVgdZozGe1OkNmnSwdsDk0499a83LTA2RYnQd/Y91c6X/8f8sErOq2zmR3kk3t3PUw3gR3/pA4macwrs93rGMPO9UNLgeiNj+FreNZg35e2b4icFqjW0pk/WSArx3TWCLQ1DhcMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tDvNCmaS; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tDvNCmaS" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-200618d71dfso20777805ad.1 for ; Fri, 09 Aug 2024 12:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232654; x=1723837454; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=gQyjT/J4Ba5hX1ofSr3HGlFYc34S1qyVyBiZGiETo24=; b=tDvNCmaSDyqOauFuB0GWJbupGiwGO8rmxX2bpgQzlrI5g57Kcf7X1Faa8FSlpolcLL GvFtAEyKjHolXIqtxftBtftG/Z+k8t1KRJE42VhaKYK4Jw7866V0YIpmVGdSRHR5Fi3I SsiE9MshFRX+GuDK4ZoVsRyj48km10uh7kVg/juZWCCgbyxhWzWg2+HBXOfOIHA0Up4n aXYMr6EBW5K7GwI2+fXxrE70TbcG83xrjQdqxh0I3sAMqPqDuxEsgEZf/gf0jNT+bwEl M7p/NEh1zNOc3mC3UwznIb5eBLgT3UEDUkzeogRIHbQB0EC4A+I/swdOwWHZe1cIDi3K V4aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232654; x=1723837454; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gQyjT/J4Ba5hX1ofSr3HGlFYc34S1qyVyBiZGiETo24=; b=cZ2BnPxzpC+C2RoiyZ81tu8SC7/cqJ1x6m6+AU06U4psKouswGeUttjZjRj71deojN PG+WSfQt87Ltyp2a30Z5NL50nhwGQbD34FY2HW+eLNLYpcI6czAqAA/x3fpO9zNjA61K m9dmCv1YfZ/++TUOsvmS4Fyez8cr9vF2Kg16JJjEQcIdj3aA6v13uTW6V4mUwtMXYNX/ aIUr/XN0uESP02pV+K33x6Y44FELexnwa9P6mETyHlWSmY/GrlPxhvMI3dcFYrPruzUe OqPlk68mO8vbOtJMZ95dZ7lqDV+LK9MCkBE+DvqJJM8iXBoMRlShMgvbY8W3hysQ9I5w quRQ== X-Gm-Message-State: AOJu0Yyj3Gd30LSOcOCAUY426uACy/bGFZOVCI2+83XXG72tJ8DwMhkN 4Olnmev7Hw5LVCU3mO6qAT355zoXQyn4qcgzai/tLP2tnvD22TOjY4j1MMCtwruE6QcxdQf82XJ igw== X-Google-Smtp-Source: AGHT+IE7aWrPpp/r00UqCmma5DjJiz0mB/4c0ahEJTDpFl71+olIXEHExH9+iNKK22JWV2AmgsyXWas3vJg= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:f689:b0:1fa:1a78:b5bc with SMTP id d9443c01a7336-200ae2a6d55mr1531575ad.0.1723232654489; Fri, 09 Aug 2024 12:44:14 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:30 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-19-seanjc@google.com> Subject: [PATCH 18/22] KVM: x86/mmu: Use KVM_PAGES_PER_HPAGE() instead of an open coded equivalent From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Use KVM_PAGES_PER_HPAGE() instead of open coding equivalent logic that is anything but obvious. No functional change intended, and verified by compiling with the below assertions: BUILD_BUG_ON((1UL << KVM_HPAGE_GFN_SHIFT(PG_LEVEL_4K)) != KVM_PAGES_PER_HPAGE(PG_LEVEL_4K)); BUILD_BUG_ON((1UL << KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M)) != KVM_PAGES_PER_HPAGE(PG_LEVEL_2M)); BUILD_BUG_ON((1UL << KVM_HPAGE_GFN_SHIFT(PG_LEVEL_1G)) != KVM_PAGES_PER_HPAGE(PG_LEVEL_1G)); Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 49c1f6cc1526..8ca7f51c2da3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1499,7 +1499,7 @@ static bool slot_rmap_walk_okay(struct slot_rmap_walk_iterator *iterator) static void slot_rmap_walk_next(struct slot_rmap_walk_iterator *iterator) { while (++iterator->rmap <= iterator->end_rmap) { - iterator->gfn += (1UL << KVM_HPAGE_GFN_SHIFT(iterator->level)); + iterator->gfn += KVM_PAGES_PER_HPAGE(iterator->level); if (iterator->rmap->val) return; From patchwork Fri Aug 9 19:43:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759232 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8921E19ADB9 for ; Fri, 9 Aug 2024 19:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232659; cv=none; b=d03US/22YHagHvm8AWDUTkmJ5yen8dEgcrLG5gajfz2mcGpOYvDDOFhPUCrgqtX674zJp7ZHvt8dRxq7Uj+59an+yZiMgl3TZTIKq1j78+lqFLT/GDZIfJYbVujS2W4OaGJevmKltDe9jswH7SxFPfB8v9xqoBBeIhH0ii92I78= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232659; c=relaxed/simple; bh=5rReX2g7eJHcUC+j0NjSl5Msvx0o5Y1O/Iq1eeDsNj4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IoF7pdMD+aHRrfFq8GqvHbE+SY/WG/EzWY+EVpeMZGXkQa34+J7a3Oio0XFblCDOUvXP/qTDOeyBpoaIctqrB0JJTWUOfE2ocpfo7fSspNzHVdTaqJAuMQKI7kUEuNx51RV+xGaLDtupD+I9Mj1N5wMWHnwFMHsI9hEUJ4oPF8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UGmUI3G0; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UGmUI3G0" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6698f11853aso48179447b3.0 for ; Fri, 09 Aug 2024 12:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232656; x=1723837456; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=kaQkfw19FaVNluamR3vKVuhloxrU0TCLWsyQPJoUOaw=; b=UGmUI3G0LPbglxhO0HahDxJaWVCfKe2K1LDPTW0hprlQiKr4n4ItnsaK7ttuAIOyGA WaLQW4G7HMbvnfb7XULss7ilkhWNYwz6IzwtmFz3r7HgIMG1a5tVIg6eYjOnOkjZZYpN CAdUTLn16zZSU3Vqr5pAdMsydj8qIIIMucciSlVE2WJw0xTB0DNwswId5yOFQen4yuFY LG36TKXU/sNFMv40d8rfgytSYowHforLdGojtETZxR3NNhTRDh+pxr0WZAQSeROrvi3+ HCSDN2jtmUcgq/NipXxX69oynrW/s7DeKhLpyPB/sGu1pZzYDo/EcMF8dNLDJ3RKEg3f Sl4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232656; x=1723837456; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kaQkfw19FaVNluamR3vKVuhloxrU0TCLWsyQPJoUOaw=; b=PQAEhMrB1omTGRxJxMwOj1FOEgYGHxpViBQyZ8ZCiFVT2l7AIE2MG6W+yiqD3ng2++ O1Vfa4Wm0ToJdXJNF5O0LJGOYQ0YxYvxEA80Cztsqld+1UiN439tUh+uDiBC8qpFyNTj JwxcIRbBSdnlrJkHCszOiHlBfzWSheBJtagu2WewgYmD7RHaPNY2t6pL7jPOnwGHCdPG m6DgSD+/00/Xv7iD/QwSa28cMLlUON4UWmJymXjXi2bSGwhLaFutG+ulTroeLO26hUg2 UPopfoDbNEM4Wf2h/iB1CRKvvrsAmJlqeFNUHSNsjeCNGh8d8VwDlwTjARdaaFcfVtmC GvhQ== X-Gm-Message-State: AOJu0Yz+0JSSJXRJQCkkE7twFiD8Tgmx0VAMIU1zLcl+sMrTHlCiqFNt TRvmNGo5XNErRfNJdYZuWy9uWB6IfD63h5DqHzS8wdDDZ/SePk5If4drbs0pnilznuIxnMQrtDG tRQ== X-Google-Smtp-Source: AGHT+IEBMWPkgI1k2a5Dd4JPhwd4nlairCp4gJ2z3JSwf2Dhnd/SN6RWtBOMd6oz6Ibb0/SIzT9/HBsYOT8= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a5b:e8e:0:b0:e0b:cce3:45c7 with SMTP id 3f1490d57ef6-e0eb99ff70amr4436276.9.1723232656534; Fri, 09 Aug 2024 12:44:16 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:31 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-20-seanjc@google.com> Subject: [PATCH 19/22] KVM: x86/mmu: Add infrastructure to allow walking rmaps outside of mmu_lock From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Steal another bit from rmap entries (which are word aligned pointers, i.e. have 2 free bits on 32-bit KVM, and 3 free bits on 64-bit KVM), and use the bit to implement a *very* rudimentary per-rmap spinlock. The only anticipated usage of the lock outside of mmu_lock is for aging gfns, and collisions between aging and other MMU rmap operations are quite rare, e.g. unless userspace is being silly and aging a tiny range over and over in a tight loop, time between contention when aging an actively running VM is O(seconds). In short, a more sophisticated locking scheme shouldn't be necessary. Note, the lock only protects the rmap structure itself, SPTEs that are pointed at by a locked rmap can still be modified and zapped by another task (KVM drops/zaps SPTEs before deleting the rmap entries) Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 80 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 8ca7f51c2da3..a683b5fc4026 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -909,11 +909,73 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu * About rmap_head encoding: * * If the bit zero of rmap_head->val is clear, then it points to the only spte - * in this rmap chain. Otherwise, (rmap_head->val & ~1) points to a struct + * in this rmap chain. Otherwise, (rmap_head->val & ~3) points to a struct * pte_list_desc containing more mappings. */ #define KVM_RMAP_MANY BIT(0) +/* + * rmaps and PTE lists are mostly protected by mmu_lock (the shadow MMU always + * operates with mmu_lock held for write), but rmaps can be walked without + * holding mmu_lock so long as the caller can tolerate SPTEs in the rmap chain + * being zapped/dropped _while the rmap is locked_. + * + * Other than the KVM_RMAP_LOCKED flag, modifications to rmap entries must be + * done while holding mmu_lock for write. This allows a task walking rmaps + * without holding mmu_lock to concurrently walk the same entries as a task + * that is holding mmu_lock but _not_ the rmap lock. Neither task will modify + * the rmaps, thus the walks are stable. + * + * As alluded to above, SPTEs in rmaps are _not_ protected by KVM_RMAP_LOCKED, + * only the rmap chains themselves are protected. E.g. holding an rmap's lock + * ensures all "struct pte_list_desc" fields are stable. + */ +#define KVM_RMAP_LOCKED BIT(1) + +static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +{ + unsigned long old_val, new_val; + + old_val = READ_ONCE(rmap_head->val); + if (!old_val) + return 0; + + do { + /* + * If the rmap is locked, wait for it to be unlocked before + * trying acquire the lock, e.g. to bounce the cache line. + */ + while (old_val & KVM_RMAP_LOCKED) { + old_val = READ_ONCE(rmap_head->val); + cpu_relax(); + } + + /* + * Recheck for an empty rmap, it may have been purged by the + * task that held the lock. + */ + if (!old_val) + return 0; + + new_val = old_val | KVM_RMAP_LOCKED; + } while (!try_cmpxchg(&rmap_head->val, &old_val, new_val)); + + /* Return the old value, i.e. _without_ the LOCKED bit set. */ + return old_val; +} + +static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, + unsigned long new_val) +{ + WARN_ON_ONCE(new_val & KVM_RMAP_LOCKED); + WRITE_ONCE(rmap_head->val, new_val); +} + +static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) +{ + return READ_ONCE(rmap_head->val) & ~KVM_RMAP_LOCKED; +} + /* * Returns the number of pointers in the rmap chain, not counting the new one. */ @@ -924,7 +986,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, struct pte_list_desc *desc; int count = 0; - old_val = rmap_head->val; + old_val = kvm_rmap_lock(rmap_head); if (!old_val) { new_val = (unsigned long)spte; @@ -956,7 +1018,7 @@ static int pte_list_add(struct kvm_mmu_memory_cache *cache, u64 *spte, desc->sptes[desc->spte_count++] = spte; } - rmap_head->val = new_val; + kvm_rmap_unlock(rmap_head, new_val); return count; } @@ -1004,7 +1066,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, unsigned long rmap_val; int i; - rmap_val = rmap_head->val; + rmap_val = kvm_rmap_lock(rmap_head); if (KVM_BUG_ON_DATA_CORRUPTION(!rmap_val, kvm)) goto out; @@ -1030,7 +1092,7 @@ static void pte_list_remove(struct kvm *kvm, u64 *spte, } out: - rmap_head->val = rmap_val; + kvm_rmap_unlock(rmap_head, rmap_val); } static void kvm_zap_one_rmap_spte(struct kvm *kvm, @@ -1048,7 +1110,7 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, unsigned long rmap_val; int i; - rmap_val = rmap_head->val; + rmap_val = kvm_rmap_lock(rmap_head); if (!rmap_val) return false; @@ -1067,13 +1129,13 @@ static bool kvm_zap_all_rmap_sptes(struct kvm *kvm, } out: /* rmap_head is meaningless now, remember to reset it */ - rmap_head->val = 0; + kvm_rmap_unlock(rmap_head, 0); return true; } unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { - unsigned long rmap_val = rmap_head->val; + unsigned long rmap_val = kvm_rmap_get(rmap_head); struct pte_list_desc *desc; if (!rmap_val) @@ -1139,7 +1201,7 @@ struct rmap_iterator { static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { - unsigned long rmap_val = rmap_head->val; + unsigned long rmap_val = kvm_rmap_get(rmap_head); u64 *sptep; if (!rmap_val) From patchwork Fri Aug 9 19:43:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759233 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1364B19B3E1 for ; Fri, 9 Aug 2024 19:44:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232660; cv=none; b=bZP4ztqMJaCx7We3UuUOQRWKb7irleGhDrBJvuI6yzyycH1c13FbYR11IPv2ANoYnTE+A656P4A9tIF1L1Hj6+VFs125MBdPjQqG+O0M4VWC3Alp+m5qxObdBipLjO5Rmv7FsH61lqdk7a3gIWeOSKB7XxBSTDsJ59QdqmJOQvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232660; c=relaxed/simple; bh=UJIPN/Wqw44nvsetoo32cIAvD96dImRh3JnAszJC0xg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PBlGDHw1oYonVMDWQ4UZsRxby32UvEY1EJsg6QUwixhw9iTIkghwtcxoZ64x8F2YIFyTeUguUyu+Bg3GVSSL85kSsx0kaKzF29CdfKeJhJwAmxRCCLR1jMXwtCl2XuvcnsBh39xmzLL4FFVKxUaqk1pxClzw+lOFGgEf9Vja+Oc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MCNWDJlp; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MCNWDJlp" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-1fc53227f21so28240305ad.2 for ; Fri, 09 Aug 2024 12:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232658; x=1723837458; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=/UfIOuTzGEIV2Af0nH4NhXy855i+DLQnfbOw4mjIiPM=; b=MCNWDJlp/eBX60WFUNv999DoOKjZAIiTfD6dX5B43XmFxdoJXe86Ng0oE9uaGtAyXw YpZMW6YKqMduUVt8xbcv3RpfviAG2X7QYOyEFcTTXuyB/vcrJFU6d/LCaff5jcJukoo4 tpUb6cA8QoWSW4OJoUYHzQ9q604cURYqDIHex99DHuhsS2A9Y1G7yH2DWztDGkXSzvsp Ne14ArelML/Pc/yDL0h+Mw58YNmOJ53KIL5M7tsU9X8UK2HiFEhb42qx1uoLcBp0vp9E RiVWZLs+jOnGp4SwYcq3nnh5zaZBjIhCrRp+YSyfi00OZM4f3uoYfdoeScgXId6olF56 lR2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232658; x=1723837458; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/UfIOuTzGEIV2Af0nH4NhXy855i+DLQnfbOw4mjIiPM=; b=a8VphWN46FWhFentXY+zugTQA3bQPV6Fm+libnHjhbaEJyQHSbY1Ft6vtysR/2rDsf rI1A4AokzTxXqAQfuG+THRLTj3L+0w61VsV999hdeTpY94IFioQTt3NGC2g2Lt+5bEPs G0FgbW7pQdMjTrud2I5iualzq2m37ExFpzlQVFJcle3BlOBDad8sr1qNqXGHUOob6bO4 ne5nkc0eeOF+YGZI2S8e/ams2RHsj/r+gwdpBeE4el8v52VHig/3IQ0nbuSKLwsvAV7U WAMxl6ER1AVDWyzz93QfwWaNd2/700VFoarsmPf/bkGYOUpTGpZRXBjJepK671/mcD/5 jQwg== X-Gm-Message-State: AOJu0Yzh/QEUx+OBEuWK4E/EE/s6M+W0RQdr7KMb8/u6a3LNqDDohDHV SKtcucdt0KilnUjwDCz8XMXMijFtRDOn42qrp1xpdWF8QCrID8tMM5DK0p4shKFiQ8eXf4o7MD0 jLQ== X-Google-Smtp-Source: AGHT+IGIFyto8PfbQoPuRgcHzc8y0pWxF/kkpM1SeNnq1V1bACAA4fUhirUzmNwmEGbz15VyaKrug3Yf37Q= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e74e:b0:1ff:39d7:a1a4 with SMTP id d9443c01a7336-200ae5d8a1cmr1519355ad.12.1723232658496; Fri, 09 Aug 2024 12:44:18 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:32 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-21-seanjc@google.com> Subject: [PATCH 20/22] KVM: x86/mmu: Add support for lockless walks of rmap SPTEs From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton Add a lockless version of for_each_rmap_spte(), which is pretty much the same as the normal version, except that it doesn't BUG() the host if a non-present SPTE is encountered. When mmu_lock is held, it should be impossible for a different task to zap a SPTE, _and_ zapped SPTEs must be removed from their rmap chain prior to dropping mmu_lock. Thus, the normal walker BUG()s if a non-present SPTE is encountered as something is wildly broken. When walking rmaps without holding mmu_lock, the SPTEs pointed at by the rmap chain can be zapped/dropped, and so a lockless walk can observe a non-present SPTE if it runs concurrently with a different operation that is zapping SPTEs. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 86 +++++++++++++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 23 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a683b5fc4026..48e8608c2738 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -932,10 +932,16 @@ static struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu */ #define KVM_RMAP_LOCKED BIT(1) -static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +static unsigned long __kvm_rmap_lock(struct kvm_rmap_head *rmap_head) { unsigned long old_val, new_val; + /* + * Elide the lock if the rmap is empty, as lockless walkers (read-only + * mode) don't need to (and can't) walk an empty rmap, nor can they add + * entries to the rmap. I.e. the only paths that process empty rmaps + * do so while holding mmu_lock for write, and are mutually exclusive. + */ old_val = READ_ONCE(rmap_head->val); if (!old_val) return 0; @@ -960,17 +966,53 @@ static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) new_val = old_val | KVM_RMAP_LOCKED; } while (!try_cmpxchg(&rmap_head->val, &old_val, new_val)); - /* Return the old value, i.e. _without_ the LOCKED bit set. */ + /* + * Return the old value, i.e. _without_ the LOCKED bit set. It's + * impossible for the return value to be 0 (see above), i.e. the read- + * only unlock flow can't get a false positive and fail to unlock. + */ return old_val; } +static unsigned long kvm_rmap_lock(struct kvm_rmap_head *rmap_head) +{ + /* + * TODO: Plumb in @kvm and add a lockdep assertion that mmu_lock is + * held for write. + */ + return __kvm_rmap_lock(rmap_head); +} + static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, unsigned long new_val) { - WARN_ON_ONCE(new_val & KVM_RMAP_LOCKED); + KVM_MMU_WARN_ON(new_val & KVM_RMAP_LOCKED); WRITE_ONCE(rmap_head->val, new_val); } +/* + * If mmu_lock isn't held, rmaps can only locked in read-only mode. The actual + * locking is the same, but the caller is disallowed from modifying the rmap, + * and so the unlock flow is a nop if the rmap is/was empty. + */ +__maybe_unused +static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_head) +{ + return __kvm_rmap_lock(rmap_head); +} + +__maybe_unused +static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, + unsigned long old_val) +{ + if (!old_val) + return; + + KVM_MMU_WARN_ON(old_val != (rmap_head->val & ~KVM_RMAP_LOCKED)); + WRITE_ONCE(rmap_head->val, old_val); +} + + static unsigned long kvm_rmap_get(struct kvm_rmap_head *rmap_head) { return READ_ONCE(rmap_head->val) & ~KVM_RMAP_LOCKED; @@ -1202,23 +1244,18 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, struct rmap_iterator *iter) { unsigned long rmap_val = kvm_rmap_get(rmap_head); - u64 *sptep; if (!rmap_val) return NULL; if (!(rmap_val & KVM_RMAP_MANY)) { iter->desc = NULL; - sptep = (u64 *)rmap_val; - goto out; + return (u64 *)rmap_val; } iter->desc = (struct pte_list_desc *)(rmap_val & ~KVM_RMAP_MANY); iter->pos = 0; - sptep = iter->desc->sptes[iter->pos]; -out: - BUG_ON(!is_shadow_present_pte(*sptep)); - return sptep; + return iter->desc->sptes[iter->pos]; } /* @@ -1228,14 +1265,11 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, */ static u64 *rmap_get_next(struct rmap_iterator *iter) { - u64 *sptep; - if (iter->desc) { if (iter->pos < PTE_LIST_EXT - 1) { ++iter->pos; - sptep = iter->desc->sptes[iter->pos]; - if (sptep) - goto out; + if (iter->desc->sptes[iter->pos]) + return iter->desc->sptes[iter->pos]; } iter->desc = iter->desc->more; @@ -1243,20 +1277,26 @@ static u64 *rmap_get_next(struct rmap_iterator *iter) if (iter->desc) { iter->pos = 0; /* desc->sptes[0] cannot be NULL */ - sptep = iter->desc->sptes[iter->pos]; - goto out; + return iter->desc->sptes[iter->pos]; } } return NULL; -out: - BUG_ON(!is_shadow_present_pte(*sptep)); - return sptep; } -#define for_each_rmap_spte(_rmap_head_, _iter_, _spte_) \ - for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \ - _spte_; _spte_ = rmap_get_next(_iter_)) +#define __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + for (_sptep_ = rmap_get_first(_rmap_head_, _iter_); \ + _sptep_; _sptep_ = rmap_get_next(_iter_)) + +#define for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (!is_shadow_present_pte(*(_sptep_))) \ + BUG(); \ + else + +#define for_each_rmap_spte_lockless(_rmap_head_, _iter_, _sptep_, _spte_) \ + __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ + if (is_shadow_present_pte(_spte_ = mmu_spte_get_lockless(sptep))) static void drop_spte(struct kvm *kvm, u64 *sptep) { From patchwork Fri Aug 9 19:43:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759234 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3390F19CCE7 for ; Fri, 9 Aug 2024 19:44:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232662; cv=none; b=WFkcoM1GkvLex+BuHLwuoZRPQ//9IlkwNwjktTEqyyURa9l9hLEGaCd11bPX+MMPnh2SXEQkeYMEZICk8rWQF4cOe71KC49zEmULehvGhiROmdbBOEBtqxUaThgdRSisJJSRb/m0TQW6YnBxxK8va5EOiAaRopqgCdeqckh/cu4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232662; c=relaxed/simple; bh=Asf1nEQnWkrE4726hXYeTADyaj7scYaajKYH4kYYdl0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sAEzAcAvH2GPhDhNM1fkRoDFfaE9Vhnq7ogtXcZiacVQq6L3srpGB2aAs1cgvoeYLgNOEPQD/NhlHvS21W5hOL1OkovfpGCY2g9WwNPpDwDC8O+51oWYRCY/+m39+01nz8EB/FB2rq8ceLKMJbUKNVTd+ZwFVfnbJ3Sz8uZrEvM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RifMYGAA; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RifMYGAA" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-70ec5bf33f3so1982012b3a.0 for ; Fri, 09 Aug 2024 12:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232660; x=1723837460; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=mm5+yuh4XjI5vBcow3bFf5ciAXmemsZ5SboXxBoCy4Q=; b=RifMYGAA1p0+yzdi1CLrmPLouhb1lsvy6btRM34D/MZrmpPOCe4voQvmSZRlhQt+vN TbI6UQII7mH9qSuMkPuq4thkKJxMqTWTldW/Jj/9VuPVwaL+Gv8LjHNMDZ9//+ZvO9iP YU3i/LemxChyS488r77gOSflGimzkG0ODZADKByoMNPF2bk9pMSPZAqGs/CIH9FISW7v V3eArI0AaA5QuYVNvmd8Lx5BtuCdJNVvPj/4HoEpG7Q5cl27Zyome3aDb4ftSac0fXp3 SqcMOUYUhrTPFqLh7BHwwRPPlMDQ6hF90sbVtUuezFo27VLHMrXO35+Su2/GLiAX83Dy jVAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232660; x=1723837460; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mm5+yuh4XjI5vBcow3bFf5ciAXmemsZ5SboXxBoCy4Q=; b=iAIooyAaNib9KMVADsuddsPjzAYUIpzKY9VILkPR+VyOUTmHgcMSlgJdC/sU9v6lWg OANkrvLLcukj9ZF2YHBc9iLvaczS44OYGyt6MFCYEtNEVF8V1CaPolwHTumrpYPegh+Z qDPjC0Fa1yQp0brCdRUQbu3FEzTcRBhf/Y62XOcVgUNowy5hPDxmo+knTmXPW2Vg2P7V aQmQGcQoVDh0EtEjlLV+Ow2yLDKMQ1h605VWvzvVynvPcKBUDGaXmvyBNU+POl5xh6Ob 47j0+xsrqKRowTZM/KKFiMBeWP4E01eeMmunqZMc3LBat3jqoEL6s6+TH1kgkicc0c+q emBw== X-Gm-Message-State: AOJu0Yz006VTGIDF9/iUooMsyQNqUUjU7WmmNulZYl1FSWadZKBsCJ0f haUIfjxYSuqdUjUECjv4SPxFNkKNtmNehXIqdjruma6MmYXDGKIgBpMz5ICV7R8OAcPxskjXb35 fGg== X-Google-Smtp-Source: AGHT+IF/O6pTeFfoAlWjuVv6pzCpox1wsfwaTSxqhna+jt0NCRVn34Qv+aJ7MSoWohdt3Pb9tXUIOw8xzCs= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a00:91e4:b0:710:4d6c:a504 with SMTP id d2e1a72fcca58-710dcb27574mr83474b3a.4.1723232660388; Fri, 09 Aug 2024 12:44:20 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:33 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-22-seanjc@google.com> Subject: [PATCH 21/22] KVM: x86/mmu: Support rmap walks without holding mmu_lock when aging gfns From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton DO NOT MERGE, yet... Cc: James Houghton Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 63 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 59 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 48e8608c2738..9df6b465de06 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -995,13 +995,11 @@ static void kvm_rmap_unlock(struct kvm_rmap_head *rmap_head, * locking is the same, but the caller is disallowed from modifying the rmap, * and so the unlock flow is a nop if the rmap is/was empty. */ -__maybe_unused static unsigned long kvm_rmap_lock_readonly(struct kvm_rmap_head *rmap_head) { return __kvm_rmap_lock(rmap_head); } -__maybe_unused static void kvm_rmap_unlock_readonly(struct kvm_rmap_head *rmap_head, unsigned long old_val) { @@ -1743,8 +1741,53 @@ static void rmap_add(struct kvm_vcpu *vcpu, const struct kvm_memory_slot *slot, __rmap_add(vcpu->kvm, cache, slot, spte, gfn, access); } -static bool kvm_rmap_age_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, bool test_only) +static bool kvm_rmap_age_gfn_range_lockless(struct kvm *kvm, + struct kvm_gfn_range *range, + bool test_only) +{ + struct kvm_rmap_head *rmap_head; + struct rmap_iterator iter; + unsigned long rmap_val; + bool young = false; + u64 *sptep; + gfn_t gfn; + int level; + u64 spte; + + for (level = PG_LEVEL_4K; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) { + for (gfn = range->start; gfn < range->end; + gfn += KVM_PAGES_PER_HPAGE(level)) { + rmap_head = gfn_to_rmap(gfn, level, range->slot); + rmap_val = kvm_rmap_lock_readonly(rmap_head); + + for_each_rmap_spte_lockless(rmap_head, &iter, sptep, spte) { + if (!is_accessed_spte(spte)) + continue; + + if (test_only) { + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + return true; + } + + /* + * Marking SPTEs for access tracking outside of + * mmu_lock is unsupported. Report the page as + * young, but otherwise leave it as-is. + */ + if (spte_ad_enabled(spte)) + clear_bit((ffs(shadow_accessed_mask) - 1), + (unsigned long *)sptep); + young = true; + } + + kvm_rmap_unlock_readonly(rmap_head, rmap_val); + } + } + return young; +} + +static bool __kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) { struct slot_rmap_walk_iterator iterator; struct rmap_iterator iter; @@ -1783,6 +1826,18 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, return young; } + +static bool kvm_rmap_age_gfn_range(struct kvm *kvm, + struct kvm_gfn_range *range, bool test_only) +{ + /* FIXME: This also needs to be guarded with something like range->fast_only. */ + if (kvm_ad_enabled()) + return kvm_rmap_age_gfn_range_lockless(kvm, range, test_only); + + lockdep_assert_held_write(&kvm->mmu_lock); + return __kvm_rmap_age_gfn_range(kvm, range, test_only); +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; From patchwork Fri Aug 9 19:43:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 13759235 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05CDA19CD0B for ; Fri, 9 Aug 2024 19:44:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232664; cv=none; b=gRrspGhy/wiFANl9ncxQy7pDNppPahzx85vQWleriy+NqmdDoiNNgzZHrEIqOE7C6RbvqTO59mC4OxPtJlbOhFTXMchGO2esTNzOzSmgnv41gYIUqV3U8Qq9e6WMRe3gkSBic88H84DYausBq2ChaFIbF0FoALHmDjlfLdlDvT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723232664; c=relaxed/simple; bh=DAzHpkMD+wsg+IgGyvY1zUDoM4DEg8WSG8aKxUnrcM0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=URsr5V0j0WXzC2OMCATC7b+nLzRwxIqfFH13ISmqpg1zwuVEJVLpHVwKFskKuRy6R4joBahicdYAHLZaxLzpCZeoMqwMDDW6Kc832VBrhs4zFfHfw0L9LQUA3d2kegIm0hvXJcI9j+qMpybbR2C3Z1WmhfUixwViWzSLG4QD+O0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SNmfM4vo; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SNmfM4vo" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7a3c6dd559aso2628728a12.1 for ; Fri, 09 Aug 2024 12:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723232662; x=1723837462; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=8pI2Dk2CV1cTItnOXgPwVCgdPRwBcss9J2eOD4s9n2E=; b=SNmfM4voWXCyX9uc9PSsvFRT3o6CR1p7akCFejZ3L16cl0dRqL4ZZAHEpyQIHLz7OB oHxKLn61By1WM6WA4UzQWCZw8vPjAot5+XnYQZElx1cMYOKbBcJi3ZonX5oNDPgjk9MD rFUP1825C4QGb6A9X2oB3bjzorR/sOYtuyj7OtfCR4j9I9/4qY2WgnQGAKA83xopTliv Anm1wsZ0D9ErP0cZjnLFZwZdLaHxLS//YNzB3SeC2WAG3QuzLGywocwgPRQIDz0oGDEy t1dL+qCgLbPE7QdDkPyKxKOVm03BvAip+HArlxweMRIls9mJ5mN5lGfPEJgh1xNU/ubb jMLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723232662; x=1723837462; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8pI2Dk2CV1cTItnOXgPwVCgdPRwBcss9J2eOD4s9n2E=; b=H8Q9ugmJ++v5g8Y7GF/ZpcFAVf36/PjQJDUFOlwVkZWuBe9ERERFDPWyvFF6Fxg7tq CPTfdAroxZwG/4uCCTTUpbAudYHELXh9jPUUnkFGO2on20qdxn82oH7Myoh51+v5tMu9 690GmAIEwGRAERWVxvdZKIYV/NIij8VCNNf07f0xpjUHlwAvdgNU+v4MmzZzdF8GqUUe EY5HtW863eCpD8fosbezykHFuiClAI3yLCr+anud1BsVNDQLg4HaodJx+KWPDPN4uYww SIP70T2rpmW4tBhru4gme8jcSImXtmnYCl8RCZlBsjMZjxKsKi06+Q+RmszmtoUPfOTq 2ubg== X-Gm-Message-State: AOJu0YwJZCetFGj2nq1GAobyBnleYvDLSQNz0kuQJw+CFeOQwCTcA1qR sRyTw357dr+fmTK5C87+bOAhWxwVgPcqDD+QVJsRs2QsGWN5A3Lbkhgjdb8f5TzF54w0D2OSEcz dUA== X-Google-Smtp-Source: AGHT+IFfyFmWIFbiqu94F+TDoLpelUA2Pyb+SCf4Vx96bmcRj1adsvoVMyySdsBmrnDzs3PRgMYuDO6IScE= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:9c0c:b0:2c9:967d:94a4 with SMTP id 98e67ed59e1d1-2d1e80e6a81mr12876a91.5.1723232662118; Fri, 09 Aug 2024 12:44:22 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 9 Aug 2024 12:43:34 -0700 In-Reply-To: <20240809194335.1726916-1-seanjc@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240809194335.1726916-1-seanjc@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240809194335.1726916-23-seanjc@google.com> Subject: [PATCH 22/22] ***HACK*** KVM: x86: Don't take mmu_lock when aging gfns From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Oliver Upton , Marc Zyngier , Peter Xu , James Houghton DO NOT MERGE, this is a horrific hack, breaks TDP MMU, etc. Not-signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 2 ++ virt/kvm/kvm_main.c | 7 ++++--- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index c115d26844f7..e5c5d0f9a69d 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -5300,6 +5300,8 @@ static __init int svm_hardware_setup(void) if (!boot_cpu_has(X86_FEATURE_NPT)) npt_enabled = false; + npt_enabled = false; + /* Force VM NPT level equal to the host's paging level */ kvm_configure_mmu(npt_enabled, get_npt_level(), get_npt_level(), PG_LEVEL_1G); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index f18c2d8c7476..28f3493d6391 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -8419,6 +8419,8 @@ __init int vmx_hardware_setup(void) !cpu_has_vmx_invept_global()) enable_ept = 0; + enable_ept = 0; + /* NX support is required for shadow paging. */ if (!enable_ept && !boot_cpu_has(X86_FEATURE_NX)) { pr_err_ratelimited("NX (Execute Disable) not supported\n"); diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..1b9b5dea2ac8 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -642,10 +642,11 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.slot = slot; if (!r.found_memslot) { - r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) + if (!IS_KVM_NULL_FN(range->on_lock)) { + r.found_memslot = true; + KVM_MMU_LOCK(kvm); range->on_lock(kvm); + } if (IS_KVM_NULL_FN(range->handler)) goto mmu_unlock;