From patchwork Thu Dec 21 03:19:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13500957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C38AC3DA6E for ; Thu, 21 Dec 2023 03:19:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9FE68D0007; Wed, 20 Dec 2023 22:19:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D4EF48D0001; Wed, 20 Dec 2023 22:19:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA21D8D0007; Wed, 20 Dec 2023 22:19:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 97CC08D0001 for ; Wed, 20 Dec 2023 22:19:23 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6C40712035A for ; Thu, 21 Dec 2023 03:19:23 +0000 (UTC) X-FDA: 81589369806.22.C1A467B Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf09.hostedemail.com (Postfix) with ESMTP id A02F6140002 for ; Thu, 21 Dec 2023 03:19:21 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=eBM4Z1H8; spf=pass (imf09.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703128761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mGmYN/slxAgBeZjp+ZQs88mMzGzd5XnAVtjIWHCDIHA=; b=ObDtCiS/nnjWF5hzVRyLbhe1MRg+VEQrrxBNW+hoI/cPeuc1Gdk3BmQ5It9pTQMvWGPj11 r06R03uUMM+4qDiLZqf3bs9/OX0kB0ajXR7EXlSCgCWdZ88AaZwZXfqfMQvN8p07nQ2ISl e33fweZPYDM7fif+6wD5nvtQUsi76Ow= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703128761; a=rsa-sha256; cv=none; b=X+B7cITM8ONs+RA+HiX8IhdRv+AGBLyV5GfqTuZ39jz+H0O/XofKzBJcFBuN1PmuvVaBiz 0divoDGgRpZZDKjRGmDVVxdZBfBpQh1VXJZU673gm0wDNmZijOE9i/keBAfSH8skEsjxL7 5yDajwj+XMekKQRGQplSqCGPRZGEdz8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=soleen.com header.s=google header.b=eBM4Z1H8; spf=pass (imf09.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7811db57cb4so17311185a.0 for ; Wed, 20 Dec 2023 19:19:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; t=1703128761; x=1703733561; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=mGmYN/slxAgBeZjp+ZQs88mMzGzd5XnAVtjIWHCDIHA=; b=eBM4Z1H8iLce0ZGsy7afF6i6GA2sjEnT91ff82zv8llyre46u186S8jnoIFHwPsTIi cUSisM+7+t8N1i99QIsBHNbafSHqLkM+Z0mSXkj94+of4DX7h7cg/i+79mXKaIWBxdmY NXyN82AsLm5gOX1Q4JKC7aKFzZBLe+pCFgfcOAtmQqsXUxsdzqtjNnqJmnNxj2ieQ5xI Tino263wPbJqt90jfN08MaSYhJXMyzIBu7mq8lWLjoiPlIpdySEKzRyBJ58CqHVx7DuB bMghjC5jyAgHkJppBx0/B9sMiaGcdg55wrtoV2UMkGfOgL9Z0oPN888mprAs21LruOog IRGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703128761; x=1703733561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mGmYN/slxAgBeZjp+ZQs88mMzGzd5XnAVtjIWHCDIHA=; b=aUkv1iM1b6n1+H8rYB+JAyNZJPPVm7UzTz7I9n9QtftGRn1/UYthsoXIxRvnR+taQm Ymy1szg5h88H1tjGwjqvN57e6GDu/0yZTvHJJg53ZCXvV/YY/4dwTL1degnum1ZKcSwe AP807fM/3ubCnUB7XWyhZOWo02QJ/+QiHh10m2DwI98gKQ3BuKzn5QS2peTsffQy49Lj oyH9y6CXRxR5T5fG7sHUki3XGbAnA8GLH3tLdyMJqelu317/XYqoa22g0+6v3R3GlXmt aSAnmJLpQFSnRMLdcDBita4P9Ujd7uPHVGXVhx+vRPgjPcKOsiwyxegpum5UXJV+et5u hiwQ== X-Gm-Message-State: AOJu0YwXwQgLyy0YzYhq59LSWwcLgFwegDXn4O9lFCVZgmrB0qM5gfG/ Q/uZstbC8X+OgOfZcbKIkZt42Q== X-Google-Smtp-Source: AGHT+IGNYrBgDx+nvC9AuuwI67vV1vM61BysWepXxnQdaZCSvAikEGaalUvebM9KlNFAOPYa+XowDw== X-Received: by 2002:ae9:e64d:0:b0:781:1ae:5aa1 with SMTP id x13-20020ae9e64d000000b0078101ae5aa1mr52489qkl.29.1703128760786; Wed, 20 Dec 2023 19:19:20 -0800 (PST) Received: from soleen.c.googlers.com.com (55.87.194.35.bc.googleusercontent.com. [35.194.87.55]) by smtp.gmail.com with ESMTPSA id m18-20020a05620a221200b0077d85695db4sm371893qkh.99.2023.12.20.19.19.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 19:19:19 -0800 (PST) From: Pasha Tatashin To: akpm@linux-foundation.org, linux-mm@kvack.org, pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, rientjes@google.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, iommu@lists.linux.dev Subject: [RFC 2/3] iommu/intel: synchronize page table map and unmap operations Date: Thu, 21 Dec 2023 03:19:14 +0000 Message-ID: <20231221031915.619337-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog In-Reply-To: <20231221031915.619337-1-pasha.tatashin@soleen.com> References: <20231221031915.619337-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A02F6140002 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 4xko8kqkmqn88y3x3n7964r4yrzm9tiq X-HE-Tag: 1703128761-827027 X-HE-Meta: U2FsdGVkX1+sAT7wm69AFnXxpV5TGDscH9+VByluKC0L3w0emqUyvzCl2f4koXUE7K6n0svJEn1hxyJlX+hwv3c16/mbvTnBLZ+roj8koKPweEIFrDycUrcNc/p5fsC0NsQJx/l4gdSmHarK9jr9qdLmiQFPOkiM2tp58GhUOB5haYOBsy/0ZSY9eYkJy6nD3902kO5R0dNdjFDJExtqr8Dtf5dCOJnVpAhhCgvD7yGofNj5MJaedvkZWarIurag/46C/gm0wy7ZV4c/svhInFZjHahxf48Fu+z5u1L85CEPQOeJf8+nGbYtTMmcUNbmv1TEWhzBSfqe2dW+lkQu3PvxnyxG042aD4q6LsOWEVsYkTqKgHGnqLc6/AWlYUGiw++FZkFH9IvvXBAUuMeq6w9mYPRwQshWky52/wn21RHGAqg2vWxr3P+hY7rWfkGcof+jYGg9uqHxfFx2lC0SUGuN3b0J3LeHSSGJRx/1CA5Z4PKRvFVH2ZTAsoxZdLEpXpFMl7FiUGOuKdMP8OPGzqv3XgiPuESK+qP/zoUSHNVnxUw5/spBTWbUuZbD/SykjmFtnf/+4zDut2RLYCzHk09oVFHkFslLeMORw0o5XihYdaatj5T9WNFO+piHbeBTKR44kHDoBtvaD5h5evGrLmrgv2l/Qzm7lTFHJjLypHMjMG2Y06+HQO3POcs8YX76xCW8JlSc0LEH9BZysz+DgPy+0+9EZU7F6SJTcajkkC53S6LgmIuZNzs2sxsKSXL5DFXO2gWPkjEM8ROuctxSzqVQUWDDz/ZvjsRH5EF/6eNHeSHoQP3oozM8+fDfHCxphnTQge4ignnqd6vztHv3dAVHrJkiJKPvr57q2HajcK5dDkqj70c16f+0S/gsdaKF60x4T4RV314U+DmcqWtk69dJ4cqtXEayTi/AHg5rpaYAxMxFtXpVViCeYfjc/4QcJ1mKzjsC3taeUMtBbdA ht80I7/Z P2eV+t7li3/RNAK1WmF+D31UY8ARHlq4lu550iDl6GcMkA2MJbqRN1sz2jNJVDP0Kruw6vGbzmc3gCe/UmLZHBbJS+KB4dYs/k1gW1RhNsqhAIF29cG0SMSYdTrsklHT5wDPDAYQgZNFnFLPLPitZqWzZNwYWk26oXA/tNfhstlWBHmRDi1B0umrn3IoGiIKgIdE210bMEZvxScv7tmNXEz3vg8/Rf3cg68mdLbbO+25b2YA8HOm4jq7kGxMRY4HnIAMyDDJCep3AsQh2p897wiADA2F25Sd7jRc5m2IpistTdwIzKMuz6hkrZQJdRAY0ji1wTUcBI0snX9bdk92ZOcI8LPVfb+4dyIMpXAERGZi8AONxIeKsCQsLJ05ZmCTCJhyK8HMuw3J5Va+Jpd+DQ9evcckUW54bK2JS6yWHOUCrscxip5MCE+sP/BwiEI22e6nt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since, we are going to update parent page table entries when lower level page tables become emtpy and we add them to the free list. We need a way to synchronize the operation. Use domain->pgd_lock to protect all map and unmap operations. This is reader/writer lock. At the beginning everything is going to be read only mode, however, later, when free page table on unmap is added we will add a writer section as well. Signed-off-by: Pasha Tatashin --- drivers/iommu/intel/iommu.c | 21 +++++++++++++++++++-- drivers/iommu/intel/iommu.h | 3 +++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 4688ef797161..733f25b277a3 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1082,11 +1082,13 @@ static void dma_pte_free_pagetable(struct dmar_domain *domain, unsigned long last_pfn, int retain_level) { + read_lock(&domain->pgd_lock); dma_pte_clear_range(domain, start_pfn, last_pfn); /* We don't need lock here; nobody else touches the iova range */ dma_pte_free_level(domain, agaw_to_level(domain->agaw), retain_level, domain->pgd, 0, start_pfn, last_pfn); + read_unlock(&domain->pgd_lock); /* free pgd */ if (start_pfn == 0 && last_pfn == DOMAIN_MAX_PFN(domain->gaw)) { @@ -1179,9 +1181,11 @@ static void domain_unmap(struct dmar_domain *domain, unsigned long start_pfn, WARN_ON(start_pfn > last_pfn)) return; + read_lock(&domain->pgd_lock); /* we don't need lock here; nobody else touches the iova range */ dma_pte_clear_level(domain, agaw_to_level(domain->agaw), domain->pgd, 0, start_pfn, last_pfn, freelist); + read_unlock(&domain->pgd_lock); /* free pgd */ if (start_pfn == 0 && last_pfn == DOMAIN_MAX_PFN(domain->gaw)) { @@ -2217,6 +2221,7 @@ __domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn, pteval = ((phys_addr_t)phys_pfn << VTD_PAGE_SHIFT) | attr; + read_lock(&domain->pgd_lock); while (nr_pages > 0) { uint64_t tmp; @@ -2226,8 +2231,10 @@ __domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn, pte = pfn_to_dma_pte(domain, iov_pfn, &largepage_lvl, gfp); - if (!pte) + if (!pte) { + read_unlock(&domain->pgd_lock); return -ENOMEM; + } first_pte = pte; lvl_pages = lvl_to_nr_pages(largepage_lvl); @@ -2287,6 +2294,7 @@ __domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn, pte = NULL; } } + read_unlock(&domain->pgd_lock); return 0; } @@ -4013,6 +4021,7 @@ static int md_domain_init(struct dmar_domain *domain, int guest_width) domain->pgd = alloc_pgtable_page(domain->nid, GFP_ATOMIC); if (!domain->pgd) return -ENOMEM; + rwlock_init(&domain->pgd_lock); domain_flush_cache(domain, domain->pgd, PAGE_SIZE); return 0; } @@ -4247,11 +4256,15 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain, unsigned long start_pfn, last_pfn; int level = 0; + read_lock(&dmar_domain->pgd_lock); /* Cope with horrid API which requires us to unmap more than the size argument if it happens to be a large-page mapping. */ if (unlikely(!pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, - &level, GFP_ATOMIC))) + &level, GFP_ATOMIC))) { + read_unlock(&dmar_domain->pgd_lock); return 0; + } + read_unlock(&dmar_domain->pgd_lock); if (size < VTD_PAGE_SIZE << level_to_offset_bits(level)) size = VTD_PAGE_SIZE << level_to_offset_bits(level); @@ -4315,8 +4328,10 @@ static phys_addr_t intel_iommu_iova_to_phys(struct iommu_domain *domain, int level = 0; u64 phys = 0; + read_lock(&dmar_domain->pgd_lock); pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &level, GFP_ATOMIC); + read_unlock(&dmar_domain->pgd_lock); if (pte && dma_pte_present(pte)) phys = dma_pte_addr(pte) + (iova & (BIT_MASK(level_to_offset_bits(level) + @@ -4919,8 +4934,10 @@ static int intel_iommu_read_and_clear_dirty(struct iommu_domain *domain, struct dma_pte *pte; int lvl = 0; + read_lock(&dmar_domain->pgd_lock); pte = pfn_to_dma_pte(dmar_domain, iova >> VTD_PAGE_SHIFT, &lvl, GFP_ATOMIC); + read_unlock(&dmar_domain->pgd_lock); pgsize = level_size(lvl) << VTD_PAGE_SHIFT; if (!pte || !dma_pte_present(pte)) { iova += pgsize; diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index f1ea508f45bd..cb0577ec5166 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -618,6 +618,9 @@ struct dmar_domain { struct { /* virtual address */ struct dma_pte *pgd; + + /* Synchronizes pgd map/unmap operations */ + rwlock_t pgd_lock; /* max guest address width */ int gaw; /*