From patchwork Wed Feb 5 23:17:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13962016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 628BFC02192 for ; Wed, 5 Feb 2025 23:18:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F32C76B0095; Wed, 5 Feb 2025 18:18:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE41F6B0096; Wed, 5 Feb 2025 18:18:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAAD76B0099; Wed, 5 Feb 2025 18:18:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B9DB26B0095 for ; Wed, 5 Feb 2025 18:18:23 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 61C8180D29 for ; Wed, 5 Feb 2025 23:18:23 +0000 (UTC) X-FDA: 83087456886.17.F431A61 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id AA81E40003 for ; Wed, 5 Feb 2025 23:18:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KhGSm9ji; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of alex.williamson@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=alex.williamson@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738797501; a=rsa-sha256; cv=none; b=pIdA2EK+N3vjRzAzNaSWTUFjYZdIkUiqGU4oXVItvkytpWCL2o10Mlo43fHRifCrN3PsJg KoiquZs2zRIi9mYgUfWd1mgGi8mHE5/UTN3yfuP1qwrkyGVhS1aHldX4Z0JosCd7GPSs8K +snV8s7EzVG/fsPv2Uf8/Jrs9HccHUM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KhGSm9ji; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of alex.williamson@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=alex.williamson@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738797501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8W4JP+Fju0ACXLthBx+3G3GONe8TLTlsBTTA1Q3RDZE=; b=qYC2XLUnL85tunL5/2w/tkl5NYwbqY3sJyxIqxjzYACrop0ZqHIhKw/aj5zQez0NTE9iCP Yo1cu/yNtGgX6l/7QUeDqVnYmsWyUXumgMLABCwbqCOBeCn2kjomHzC8OtUOAoO255hoiG eDwWyPVPboACg6pmAU9gxY0bzTtnoEg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738797501; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8W4JP+Fju0ACXLthBx+3G3GONe8TLTlsBTTA1Q3RDZE=; b=KhGSm9jiAYbc9klwd8G76Z8RtmqqTI4FiDwnPisBWqlosWWihrX62nQQXKBdvtxUxQQgO6 MQkUMeYkh8ufp4csvAcYPnOdwV9x1DcUGun+zo8QBnM7btsPHb4qLaZLrPAPXF00csBFmU MSUB5wFdZk2/Hy5IGYywyN9Ht1BTWVE= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-561-FkNDRqmMM9yE_UzeBVYLPQ-1; Wed, 05 Feb 2025 18:18:17 -0500 X-MC-Unique: FkNDRqmMM9yE_UzeBVYLPQ-1 X-Mimecast-MFC-AGG-ID: FkNDRqmMM9yE_UzeBVYLPQ Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 871FF1800985; Wed, 5 Feb 2025 23:18:16 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.81.141]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9D68A1800570; Wed, 5 Feb 2025 23:18:14 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, akpm@linux-foundation.org, linux-mm@kvack.org Subject: [PATCH 4/5] mm: Provide page mask in struct follow_pfnmap_args Date: Wed, 5 Feb 2025 16:17:20 -0700 Message-ID: <20250205231728.2527186-5-alex.williamson@redhat.com> In-Reply-To: <20250205231728.2527186-1-alex.williamson@redhat.com> References: <20250205231728.2527186-1-alex.williamson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: AA81E40003 X-Stat-Signature: ikxomcjdnwftds931woad6uayxe3szer X-HE-Tag: 1738797501-291336 X-HE-Meta: U2FsdGVkX1/xLcZkJ8+LGr16maYBmX5TOvcQ0JjVs0I0RmK7+ep0aZdGAAgdrfgH3IbuQsbP/1zyMlgJOP2p0ZdicyE06xavhbpn8GamedSPpk2o/1O2f0OsVnDqa4sv81yDJkjnYZXbO2Zsa9EokA+C8og3C6w0dK1eGbSS6ToAIr72a2R9h9xYKhXw+jIyeGtuzPMg/+jTHzDaWzsmkjhCWFLQUQLlRyMvUi92j55Fy5oFSoZS2rgbvjJ7OcqMbYuVK4bO0hvyHZpRORc3GUeI93MKuz3I+6cpYYqSGaLvAWPluAUdptkGQwJOIpcOeaNrKhXNUAjihtcutwc2B3lJVgH7gImR2EtV8EqnqvgofOvue/VB9vRoc+YDE2sK/ADVzvpWfEAfkhce35xqKcLZbY1S1pCnV9xfUc4bngP/9f1ilCD179P3e17Cq8Y2RYWQGpEUhVbMaEkPia8VoyXZgrEp5NVvim9kDUA/w3emlxbecsIRVI/brOsy1ffIxiesPdovIvXGgT2ljjWHxEl1taenw0C1GEmPwIh3ezD7CzeYHmK8SiuuTxR0JtRcyL5olz921vM/Vwvzu5eHBXXWL/NrvyEcroV7y72X0d9KSIDYF0rXuhnuEoqIDKkXool+zhzYP9f2XSCiqr5dqmAbXoEfEQnpz/DCpLEg5CIvMC4WGdbcfjYqQsecTwD1y5bCxu4bhZLD9D5lejHgmHrTaoNy2M1uhQhl6XD08Z1A+0eHw7I+Q5wC3FrsZTi/+7/cBOmtMehS03Rslum6+HFIENF/BAcyr5cPiQ8cPxB1MIkrkFwr3vA+BTivpUWFxwBrQBDiBHI7IZjahEi4p3Xr4S86J1spCOzY03oZoaPJyqMbcfiyF8rgmHjWELM80JAn5ooR9PsPLQtpb4xiL9pGXSXDShKfhIVv/bEXdm/g3F4fygTRoYICLv7je3aR8TmpW3/AiQE1eJ3Tyht HjPXyMeF kW4fI/DkiTyzxRoXe80YJVCfTyKAQL1mfNS8CbS7uSVTSTPyY6qy3vXUjpIRaVZkjuirON1+A1ApXGivUpDbt1Gig4QChiCfh9F8R+hj13asJBeLWgRxotD2rRR82CZtUdht3oHMSR0MiqDPI7ZvBbyERJRcQrdi9aoMuZZG4QEkwReJ2xEVgQ/DC9BzOA7AG/qTgi+LB1XbvdvUid0J+u/W3+si0ZPDlh10XvN+ktyZKXbrNBgwXe3bBWNB3bF2D9PHtEAWB7gSMHkAfNNoFxNHHjsKmUUdt2lK1AnP8TUBw/B8/egXRDksAYkQ+lWhelgCx4ueadMDxZH2FoXOhHtf70AYXsRJsSdPekr1IvK+zX0hg9EbGVMHFcoMGFpJPXRbD5TySPpaMaGNK+avkVX8rXb0VFuhZOXC4czvRnxQpr6SBGLZGpl5QaXRucX2ddlCk92KgoO4ACqXEU1VxupjbllBkaxZD7k2hLfeXjK5MmaE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000373, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: follow_pfnmap_start() walks the page table for a given address and fills out the struct follow_pfnmap_args in pfnmap_args_setup(). The page mask of the page table level is already provided to this latter function for calculating the pfn. This page mask can also be useful for the caller to determine the extent of the contiguous mapping. For example, vfio-pci now supports huge_fault for pfnmaps and is able to insert pud and pmd mappings. When we DMA map these pfnmaps, ex. PCI MMIO BARs, we iterate follow_pfnmap_start() to get each pfn to test for a contiguous pfn range. Providing the mapping page mask allows us to skip the extent of the mapping level. Assuming a 1GB pud level and 4KB page size, iterations are reduced by a factor of 256K. In wall clock time, mapping a 32GB PCI BAR is reduced from ~1s to <1ms. Cc: Andrew Morton Cc: linux-mm@kvack.org Signed-off-by: Alex Williamson --- include/linux/mm.h | 2 ++ mm/memory.c | 1 + 2 files changed, 3 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b1c3db9cf355..0ef7e7a0b4eb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2416,11 +2416,13 @@ struct follow_pfnmap_args { * Outputs: * * @pfn: the PFN of the address + * @pgmask: page mask covering pfn * @pgprot: the pgprot_t of the mapping * @writable: whether the mapping is writable * @special: whether the mapping is a special mapping (real PFN maps) */ unsigned long pfn; + unsigned long pgmask; pgprot_t pgprot; bool writable; bool special; diff --git a/mm/memory.c b/mm/memory.c index 398c031be9ba..97ccd43761b2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6388,6 +6388,7 @@ static inline void pfnmap_args_setup(struct follow_pfnmap_args *args, args->lock = lock; args->ptep = ptep; args->pfn = pfn_base + ((args->address & ~addr_mask) >> PAGE_SHIFT); + args->pgmask = addr_mask; args->pgprot = pgprot; args->writable = writable; args->special = special; From patchwork Wed Feb 5 23:17:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 13962017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2623C02199 for ; Wed, 5 Feb 2025 23:18:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FC556B0099; Wed, 5 Feb 2025 18:18:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4851E6B009A; Wed, 5 Feb 2025 18:18:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FC99280001; Wed, 5 Feb 2025 18:18:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 10B8F6B0099 for ; Wed, 5 Feb 2025 18:18:26 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7360C12090E for ; Wed, 5 Feb 2025 23:18:25 +0000 (UTC) X-FDA: 83087456970.17.B9D093D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id B892A40006 for ; Wed, 5 Feb 2025 23:18:23 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VA41LJHX; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of alex.williamson@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=alex.williamson@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738797503; a=rsa-sha256; cv=none; b=vLlqMpulToX2rOu9voIuDzZFWa1nXAS0aiWRf4cykq/vdp2iGinO3FB3XdfVu47vh8+vJf kcyIGZgL4wPXOnHdGoDRQa0E4JTAgtoEfbLeECo4fVeIUdM9kA/Z5P5JSQIr96tvxB94aG c0KbAOL3k5EQMuoul2GIFVey4qmw9VU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VA41LJHX; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of alex.williamson@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=alex.williamson@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738797503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FZa80BhZdx9gmyyMrwsj4JyFeb2ASk0SBEuM3S7wkWc=; b=QaaTUrRxsV4m0Fn54iY4K0qsJiUXfmf7bSPmQbr8x8xJSnlytX+vgJiVDC3UVBkT9Gtv/G tK9JEybJg7yUtN+BbVpywc9y0oTXEW2xnBw6ev2C2piZ39WV6pD43J6VaBSvqakecb/uZS jJcxU4iHRFhz+pXZ3eIfNVHu5pybbec= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738797503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FZa80BhZdx9gmyyMrwsj4JyFeb2ASk0SBEuM3S7wkWc=; b=VA41LJHXtOLrxc5rqjO+kFP3/UHo6s/Q+9Q/M+PgRSVy0G/lS4MAILOPxAz6XqnQszD1tU U4OqxFSRN/xZEIyWmv0FVIaJ7mM3SV8HlhLQcs4TU1tq0KYzu6Rx4QAiEzeLKKeOKswd/N KWg4d3N45sXwPCgtHZ50hsh8Oo1QW7U= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-TONjRLI-McyooT1dQwr4iQ-1; Wed, 05 Feb 2025 18:18:20 -0500 X-MC-Unique: TONjRLI-McyooT1dQwr4iQ-1 X-Mimecast-MFC-AGG-ID: TONjRLI-McyooT1dQwr4iQ Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B0063180087D; Wed, 5 Feb 2025 23:18:18 +0000 (UTC) Received: from omen.home.shazbot.org (unknown [10.22.81.141]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DC67C1800570; Wed, 5 Feb 2025 23:18:16 +0000 (UTC) From: Alex Williamson To: alex.williamson@redhat.com Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, peterx@redhat.com, mitchell.augustin@canonical.com, clg@redhat.com, akpm@linux-foundation.org, linux-mm@kvack.org Subject: [PATCH 5/5] vfio/type1: Use mapping page mask for pfnmaps Date: Wed, 5 Feb 2025 16:17:21 -0700 Message-ID: <20250205231728.2527186-6-alex.williamson@redhat.com> In-Reply-To: <20250205231728.2527186-1-alex.williamson@redhat.com> References: <20250205231728.2527186-1-alex.williamson@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspam-User: X-Rspamd-Queue-Id: B892A40006 X-Rspamd-Server: rspam10 X-Stat-Signature: ipzcnkern531wywysmo3yhxgb5ejyqhf X-HE-Tag: 1738797503-359150 X-HE-Meta: U2FsdGVkX1/4wcH/iwUIpZcC117iZ2XNLwwky9dFfXa6ElufvfGpx10HbMwUa0CUhf6sCB1OTeNs1G5Gqj0X0XHSUFRPZ0zgfGTySpVjbeA1JruiGGN/zLQdI3pYr0BbwZNH/+Ln+t2hqR77+wmTH4vVMpQJ8GG+Kd7aHZp7MAz7BzT0rYEyIe+WLwj8emiS5tUG+43401crHx/BPXJmI01ziJS4ExtjjtGjk4Y6HFzQYkVhYFwvZ7UV/tkssQFsKlfkeMk6FuxRRQ56e4DuAeEuaUyNW/oKUbYEUMd3ppIL8jLC9ly5rNbFphGV1198M5k1vjPhRqpTugIWq9Mab6Vopr2mMYWnz1OOnm5EqE//I/McdFm4Yn6K4effN0Ss7tfFhpIoWAiHbbdZuJJxRfL5bS6IFg3/MkorxPRDkqIO5CI/q6Vd0GUcd5tpEZ8iNZNpFJHz6hGLHcRWlPD3mHy0Q8O4jekMSC+5SlHaz+1lgSH5MDQwsue69WzVV/RuQtw3y8ZDXdLRSuCEkb4xKGzZ2VKgKLa2M7iSyzVbcNl7IVaLZIsN56XYuYasAuMBM+rm3q5TyRGp5w5FxAjZbjwt9BxA5RFdGSlD4iI2ygRgS+yo6s3VotDOtO8Fc3U62JcR2mwXNTp2CVeDZA9aWNRQbhXOgn1gWcKFaEi91J2hdgvM3CX+NkKXTPsOL39LYLtYK1mqGfbESKMQ790Vfuwxok1N6bhiC0C76qoYrkmqGtyfJHnnooFaoM2ZsqJbTgBlkWGkbqhKmlpNu8RcY7igqXbTkI/0SMhsjtaHOy2ghTkIeEF1mcefJQBW3XyEdXuNI/dcdIeOaRPZEciVdsjcEP2oLdQmwPJunFOJ9nhP+kd0ut8U/dq4nvDvof0a6iWlyIfJHUM5iXMrgM+o18xAkdjYXtUNl4A+DE1KAxrDmZlvQS2n9L/DFEiVkuxDfRHtN5XQwhVVa5HjeOZ d4ZuESYX F294jjqvmYCfPTtp9xUXQoj03zMMG/xtTzc02Zx6YVaQLWN3qz61k7wKXH82FMgyR9g7xwt/CyU6BBIYkAlSp2Ma8BkmbRq4JZEbpr7AnIAATNRDca9y67Qs9YI08KDyZlgsUN4YxZSKDYMUDC60fPXtjCACJlTYIXOM5Hy4qUfuGXQwF3WA+7S0D5Eyo2hsEBNL2sVak5ZvzYlHz8JJu52aBPYzmnEY4+8OUzyLhEslZthZITODYrs02Qw2Pp8j2OixQK/bx3JWtLGXcOHJ6P968bblaPnuq9pLbWoIEVbeuwxzKy30CAXMpvFd4JC2QXi0h X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: vfio-pci supports huge_fault for PCI MMIO BARs and will insert pud and pmd mappings for well aligned mappings. follow_pfnmap_start() walks the page table and therefore knows the page mask of the level where the address is found and returns this through follow_pfnmap_args.pgmask. Subsequent pfns from this address until the end of the mapping page are necessarily consecutive. Use this information to retrieve a range of pfnmap pfns in a single pass. With optimal mappings and alignment on systems with 1GB pud and 4KB page size, this reduces iterations for DMA mapping PCI BARs by a factor of 256K. In real world testing, the overhead of iterating pfns for a VM DMA mapping a 32GB PCI BAR is reduced from ~1s to sub-millisecond overhead. Signed-off-by: Alex Williamson --- drivers/vfio/vfio_iommu_type1.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 939920454da7..6f3e8d981311 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -520,7 +520,7 @@ static void vfio_batch_fini(struct vfio_batch *batch) static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, unsigned long vaddr, unsigned long *pfn, - bool write_fault) + unsigned long *pgmask, bool write_fault) { struct follow_pfnmap_args args = { .vma = vma, .address = vaddr }; int ret; @@ -544,10 +544,12 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm, return ret; } - if (write_fault && !args.writable) + if (write_fault && !args.writable) { ret = -EFAULT; - else + } else { *pfn = args.pfn; + *pgmask = args.pgmask; + } follow_pfnmap_end(&args); return ret; @@ -590,15 +592,23 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, vma = vma_lookup(mm, vaddr); if (vma && vma->vm_flags & VM_PFNMAP) { - ret = follow_fault_pfn(vma, mm, vaddr, pfn, prot & IOMMU_WRITE); + unsigned long pgmask; + + ret = follow_fault_pfn(vma, mm, vaddr, pfn, &pgmask, + prot & IOMMU_WRITE); if (ret == -EAGAIN) goto retry; if (!ret) { - if (is_invalid_reserved_pfn(*pfn)) - ret = 1; - else + if (is_invalid_reserved_pfn(*pfn)) { + unsigned long epfn; + + epfn = (((*pfn << PAGE_SHIFT) + ~pgmask + 1) + & pgmask) >> PAGE_SHIFT; + ret = min_t(int, npages, epfn - *pfn); + } else { ret = -EFAULT; + } } } done: