From patchwork Thu Jul 4 04:30:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13723183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E19F6C3065C for ; Thu, 4 Jul 2024 04:32:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28B606B0098; Thu, 4 Jul 2024 00:32:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 17B526B0099; Thu, 4 Jul 2024 00:32:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E98466B009A; Thu, 4 Jul 2024 00:31:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BEDFA6B0098 for ; Thu, 4 Jul 2024 00:31:59 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 50DD2C0A37 for ; Thu, 4 Jul 2024 04:31:59 +0000 (UTC) X-FDA: 82300797558.20.607A16D Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf25.hostedemail.com (Postfix) with ESMTP id 283F1A000E for ; Thu, 4 Jul 2024 04:31:56 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=h+c3YldW; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=frfAxxwF; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=h+c3YldW; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=frfAxxwF; spf=pass (imf25.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720067493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hCzlv35dbgXloc5Cmstin2CCiJG29oVOef/hhtOkA+o=; b=8Q/G0ePPzsjb5bQjTGTWlikmjRvt10ULJL1cRF8w+beLsr2rJwZ/1dnsd8s8OFbhRUB0Uc qADVuA85OL+hvZIpxYNDOTQ6Fx7rKWOxbfGhXT91nHHrL9kOni0FfuMkBaZFf4DEKyXBhf Dhk6CZ/Cer4tU9kJhx9n1PgMVIBQCvs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720067493; a=rsa-sha256; cv=none; b=npYl3l5zRwF4weZvPC3gGZ+mYEHW0xpBGghraY4xRHazt6RzesYFHn7PErn+PhNg+m+2fp OjjjGqjWZCFUWITDV8N9RbhRPwrUp4W0+qq8FVOPUsJy3B7HxYUfUwRxLLbSepgA/tWCcM 5wDBLeFA6gyik3BSPHGObeK4X1LJbUU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=h+c3YldW; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=frfAxxwF; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=h+c3YldW; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=frfAxxwF; spf=pass (imf25.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BD80C1FCFB; Thu, 4 Jul 2024 04:31:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1720067515; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hCzlv35dbgXloc5Cmstin2CCiJG29oVOef/hhtOkA+o=; b=h+c3YldWsDCWou0FH1uqAUtnGxvDkmkrw5MQS+eAbm6nc4xHjoC7SiMmO1KgT1ogxMucRW JsZAInX2H9tHjZJ29ry01mlBA/1iNay0rlLOoCB6DCVmON6lUqsmtsj4w0Z5P4bSRXrmdd E1ywbhdn4+EiM/LJbQhJoM39l9zXjNI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1720067515; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hCzlv35dbgXloc5Cmstin2CCiJG29oVOef/hhtOkA+o=; b=frfAxxwFBvqVUDaANiozJD91Zhfd1mCsF2hzZSdarkYr3nve6JYZaO/PFTJ73t684ex8JF l+vQvnp2Om9NkABQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1720067515; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hCzlv35dbgXloc5Cmstin2CCiJG29oVOef/hhtOkA+o=; b=h+c3YldWsDCWou0FH1uqAUtnGxvDkmkrw5MQS+eAbm6nc4xHjoC7SiMmO1KgT1ogxMucRW JsZAInX2H9tHjZJ29ry01mlBA/1iNay0rlLOoCB6DCVmON6lUqsmtsj4w0Z5P4bSRXrmdd E1ywbhdn4+EiM/LJbQhJoM39l9zXjNI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1720067515; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hCzlv35dbgXloc5Cmstin2CCiJG29oVOef/hhtOkA+o=; b=frfAxxwFBvqVUDaANiozJD91Zhfd1mCsF2hzZSdarkYr3nve6JYZaO/PFTJ73t684ex8JF l+vQvnp2Om9NkABQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id A7EBF13889; Thu, 4 Jul 2024 04:31:54 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eHJfJLolhmbnDAAAD6G6ig (envelope-from ); Thu, 04 Jul 2024 04:31:54 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Peter Xu , Muchun Song , David Hildenbrand , SeongJae Park , Miaohe Lin , Michal Hocko , Matthew Wilcox , Christophe Leroy , Oscar Salvador Subject: [PATCH 05/45] mm/pagewalk: Enable walk_pmd_range to handle cont-pmds Date: Thu, 4 Jul 2024 06:30:52 +0200 Message-ID: <20240704043132.28501-6-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240704043132.28501-1-osalvador@suse.de> References: <20240704043132.28501-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 283F1A000E X-Stat-Signature: 6jqbpshurxugbata7f56yik4cwcjyphz X-HE-Tag: 1720067516-466164 X-HE-Meta: U2FsdGVkX1/2rtjZwwG3o7RWvZqBnfcf05T7oPTGfQ3jeNQflRznQP5WqDDTJN2ghw+4GpnQNads0AS5ViF90bmrH4AF7mcmBUL9WyniQcmhT7isobcjS078FqiAN2YH5yGhKcAw4Nn5PQukKdoiYF+Km6x9T2Ua23B8WiPmXtYMxHFn2T9Kosen15z1klGz/88wUC27knKoreG1a8oLlEw0cKuVgUT3waKfBuPOOwnxeC6Q1wVEsQ02/UalZV0NShtHPoR5sdbXIXsWUkeO2eKllIHNKGdqObHrsMsAmoKbO9pfy7z75eW6ls8HqRyOCDyZRbmg3cDfTo/TvvIreZSc5bHI9UcGxbWXTmGwDMm/nlIKIodezti55gL0DV9g+EUE/1AlDxKq8qvxB86H16YVSSU+FXhlWPoAts8Ei0LF9o6rB3OV8YIqo5MxJxff6CTXKxE6ql5Mog+pS+4lz2EMrKXMK9gpOX8OgtTF4rKl+ikxUS+foIccMpgj0cEk0NWxoBv2fgAdjyj3tmz7CSXatoj/Qkw8/cXlX8gvBre/XaOM5fy11rw0ZjOU2uASnymwqvvlyqrDxon23CwNjVE8V0P24DnUQCSAgr6qB9qXpmoyiGEd4ek2W90k5q+C4n6EJclgIb/2SXvPg3A+tCnnExo52xUU6xWRArwIC+fALqznnSu1JnFPcqE0hmc/OPZBS633k72aglWA37PiLgQbOcgTSjPjt/hKTnR5egg1ulRLA2nhdJrKXxPvUBBgfCnK1XLl69odCJLPT3baTQLbrncPjASNCix5jnxFKsXoc4NyIEhEvoXy7QXfP89RscWXzySvdiE6Tq9kldi78yQok7oicrUyFiCodZcuG0FK3ls4FK42mP0X8dKd4XA8jCGdUrp0MRcpYL23+oQRhPdnEey58u/eSTHZy/5rkxDapL2bygUYhJZddbWzpnHJHhtFHvwWwcZIFYFPHCx GWNO9eY/ La/1CRfPi3ooXQi71XdUZf/+pj+YF1nc8w82c04WyVQ0mm/WWEUeDNheT9qVzjXDLbvM3F4WpSp2vw5BTtgnizM7winEJ/3v6K2pKi7XWXd0xFJUI40lS1clJuK4QkVDa1AHvHiJoA1u2MXcIbB41kcGr6TRNRYNQdaseJnCe6PLz8zOSoFkeVtyLWjU+r82BmM8qGa6bFZUEfZVeOs4jpRnWKRsB+h2GWraGvH9b5wUizvkJYnBrwwL8GnlajOB6z62HJ+32ZMKwfLjPE4N5WebOYg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: HugeTLB pages can be cont-pmd mapped, so teach walk_pmd_range to handle those. This will save us some cycles as we do it in one-shot instead of calling in multiple times. Signed-off-by: Oscar Salvador --- include/linux/pgtable.h | 12 ++++++++++++ mm/pagewalk.c | 12 +++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2a6a3cccfc36..3a7b8751747e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1914,6 +1914,18 @@ typedef unsigned int pgtbl_mod_mask; #define __pte_leaf_size(x,y) pte_leaf_size(y) #endif +#ifndef pmd_cont +#define pmd_cont(x) false +#endif + +#ifndef CONT_PMD_SIZE +#define CONT_PMD_SIZE 0 +#endif + +#ifndef CONT_PMDS +#define CONT_PMDS 0 +#endif + /* * We always define pmd_pfn for all archs as it's used in lots of generic * code. Now it happens too for pud_pfn (and can happen for larger diff --git a/mm/pagewalk.c b/mm/pagewalk.c index d93e77411482..a9c36f9e9820 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -81,11 +81,18 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, const struct mm_walk_ops *ops = walk->ops; int err = 0; int depth = real_depth(3); + int cont_pmds; pmd = pmd_offset(pud, addr); do { again: - next = pmd_addr_end(addr, end); + if (pmd_cont(*pmd)) { + cont_pmds = CONT_PMDS; + next = pmd_cont_addr_end(addr, end); + } else { + cont_pmds = 1; + next = pmd_addr_end(addr, end); + } if (pmd_none(*pmd)) { if (ops->pte_hole) err = ops->pte_hole(addr, next, depth, walk); @@ -126,8 +133,7 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, if (walk->action == ACTION_AGAIN) goto again; - - } while (pmd++, addr = next, addr != end); + } while (pmd += cont_pmds, addr = next, addr != end); return err; }