From patchwork Thu Nov 14 06:59:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13874613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69961D65C52 for ; Thu, 14 Nov 2024 07:01:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2E836B0096; Thu, 14 Nov 2024 02:01:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B8746B0098; Thu, 14 Nov 2024 02:01:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 832526B0099; Thu, 14 Nov 2024 02:01:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6033C6B0096 for ; Thu, 14 Nov 2024 02:01:05 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 218B3C0F7B for ; Thu, 14 Nov 2024 07:01:05 +0000 (UTC) X-FDA: 82783802766.18.5833FED Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf22.hostedemail.com (Postfix) with ESMTP id C7B70C0422 for ; Thu, 14 Nov 2024 07:00:04 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=kAqNFb1y; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731567596; a=rsa-sha256; cv=none; b=Yd6ynB3H+L1qUJX8phA+/LddG9tX1kkEmwFBz6d9/XDTekkJzCBls5juNDqhaC4SFfFmlc Gq4W0qoyp6XwWIil/CI59h7+z6TEu5b+g+ZF97E0HWZ98sp2gHKeR/mTtbvtACefmepk2K LEkbOc0Yl+YOSmAV8j2uLIM4b0iw4uI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=kAqNFb1y; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf22.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731567596; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FQVB91K7fAT15psCxftJJTVCdE7dh3d9kskucoAkCus=; b=6KQfwkCZNp7rKV4C39u5B0HwPhtjW4Hh6SZzGXc1gi4ugVR3qrEt+BsQdQF2wPH4KXXsXi I6jZxh4a8COpEeNOzdjTKV0dbefM0cBDlex6OmH5Kq9THp5uzCOuQFxh4Ybn0+yYaztK5B zcDyrVlVgVZkpgupJGgBc6/xTb964iU= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-720aa3dbda5so171096b3a.1 for ; Wed, 13 Nov 2024 23:00:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1731567658; x=1732172458; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FQVB91K7fAT15psCxftJJTVCdE7dh3d9kskucoAkCus=; b=kAqNFb1yShQOOWIUPay/spuAmY3cywCbLSvPF/+qnpdN4YDaiI4y4KwlE5gwI7Go8w ERqz7FTwK4Gjmvjjvco7ZUn2kcqkxggb9YeD4T7NnTAZDiC1/EiDhg0UNfHBQfEDAoxd G49bwuZuWUqI/QG3Ub+n7i9xHJA+ZTGaI7I1wPF35R2jgaHHOXc0H+S35TIhz0LP4HIl QJfTIPgUNDjeWBsrQvkoqmXmTEqENXPXNCvyTxmbjkiPIUmb23JpJyl2e302ISB2ooFy WpJDg5CoI2hhKanYnOL3QzniuzOvgjDt1hwOYCTUchIXAShcvAn8hY+hXMrZ7qUy8D/G 24+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731567658; x=1732172458; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FQVB91K7fAT15psCxftJJTVCdE7dh3d9kskucoAkCus=; b=t6uKlyGQcwuPWOTlEQ9dy39VFBVNuiKlV8ZOzJdZ4qnQsjJbfIY3oDXq5yLPU/62v4 yqe+WAtVgFIqBc8pY3dSG6LL5PJYSU6qt4W3oPazTPStTzirZldiLE49Jeuu5bbfzYPK AHkXDg3U9wC5DZXzXeYXbBo8zgckajgONsm4xHJHHWjXoRrLOOikjQvjE9CREEAnH3Sg 9VC9207rEcm1dhHe4ZcIvXX3r9tpLTFRnyYtpyGoHCQrIQ85RaAN8KKCXyL//X7937X1 taeIf96EKL424SOO5RtVjqkBTeFp4a5u3Xvk6FU11uLmNtRQnzBKWEcCy54KryFCPLF0 ryOw== X-Forwarded-Encrypted: i=1; AJvYcCUkQDj8ofYg6EDdgw+kNCs3xfw7UAWTvqS+yVbiDASkG5etU3f+4Vp8hxJ6Du8j14/DWbInf3MOgg==@kvack.org X-Gm-Message-State: AOJu0Yw2DrYaVETSpUqO2lA/aBZXYVuRTkzinK9iMxfPVY4fpYtPvK57 9bHOA0umcck9WZobWkOIq5Bo6O2m3EBei7/MlA4MB3cxI3yWnXQ/RIQwCP5YJdc= X-Google-Smtp-Source: AGHT+IEEOci5upVGjhq10Ml46AvoKHnyJ01pyJF7RhtSq5zmKJYKrXkqxyrBi8tCjSUuj7BETevvWg== X-Received: by 2002:a17:90b:43:b0:2d8:e524:797b with SMTP id 98e67ed59e1d1-2e9e4bed427mr12110579a91.18.1731567658258; Wed, 13 Nov 2024 23:00:58 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211c7d389c2sm4119065ad.268.2024.11.13.23.00.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 23:00:57 -0800 (PST) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, peterx@redhat.com Cc: mgorman@suse.de, catalin.marinas@arm.com, will@kernel.org, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, lorenzo.stoakes@oracle.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, zokeefe@google.com, rientjes@google.com, Qi Zheng Subject: [PATCH v3 4/9] mm: introduce skip_none_ptes() Date: Thu, 14 Nov 2024 14:59:55 +0800 Message-Id: <574bc9b646c87d878a5048edb63698a1f8483e10.1731566457.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: C7B70C0422 X-Rspamd-Server: rspam11 X-Stat-Signature: h17hhkga17x7xnuftqdx16ya75j78ygc X-HE-Tag: 1731567604-418194 X-HE-Meta: U2FsdGVkX1+limiG3PdEjqRKHgyHRFrtSxxkLBaG5MQW6hrZAQcv9S+e/aDQcSMKgOm0JB2lywN5EzEAVUudOwR6FBC7JysHX6/kueEh92KmqQYXIQzJqt2sv/zHekBNzTB10xPucjKYQEWOGKEwnkLxoouEjIwR+FFy67vWaTB/4e0XBnR1Q1+0ywkdnSC7/gOuOat1oSG2vlQAnOCZaNoEP//vz2+2Arr8YiyzBPdw+zPmUewwmtUXQC9madJAIGFSqcbjo0G1Rwg/Tno5mSbNrgafmiUcgqOTXLGv2lOo6NvQ43GCQs0eGEHiC+ft25TE0ri3PifOh0URbxOxwclo+82XT0MixiT/IYYZC4MOvYtWj6q9aT0LvIWe8rwvgBQuAoxlv4AhgBAwJhHVOKx+y5sbBl1ScnkZoCP1iA2xglXdBFiXnSyzMg9gu17l3wE+LNQz4sTgJJrqPx5RZLryOmbeKDjH/flhCooMT7aMKEiEVEhdVxUew2c16fPouAhADjwl9ioMtdlw7vfiGHlVz0wcT0TFmJd5DLUopZZX4ZbX5LQ24FtnM3ZuqCoI1+bOREGNlYvrQkZGaG/IWZFxgjbUevePfYdsFFREFDVx28AAiCmlc7vzFSKeexZt/zH2+W6R7MqjJMe7rZg875TxnnMTZJblw2byPmtr3eTiQ3PqDc0KZ4Txej2difM0dxxkCX3hbcDSVUuJ+A0GAh7b0FfX+050qweak8QiO+uaxEgHSxIn5kLGwPS1pdEPcyLWC4j96D03HjlxF3EhKYOOILaEakVhKze4mn3EKCVmWGw+Z++A1yJFrGwQPpmuENyT8hqBqNCIAwB5u/mFxTUm1M3IzdqtPFiZ7r5XPaZc0EzkMSlH8Zd3de1ja2Z5SZlWktIV/A04llsgRL5jpoopGE9XDyGnckyZT+H+Hp3OwXBVNmiGmQB1a5rbLvpQe2Ghr3ypFhK1ka1GTNa UKSVRDyW O/IfIuf1rnTlpcj5EJwO1ZPEWAVs30zHkwzHBViOBr6nAucEKWbLMutQKsyeVaTI1lIrDsyCROMpa2CzurlQkH99BiVWkzzpOrCLC5Zo8bTHT5KjI+bNRBSun0jg4wNQhypJqu52RtvHFpYeFeJNciTX7KDJ5lHEOiYuK4qFApv+sjGe/rJXwnvWTZI0Vx53BAiDA2gC3eu8wIQSNCgfYxLvdun2Y1sQtYZgvr8Eey+W/97LDPKZ0vdvNS+od1Wku3mfL/Tns6KcQoDBbBAlN7xnlbp3hkHq1mw/iAPWuil2+w8IAJvixHdIPBfv9n6ZDnU6XDffQeCC1LhnVHJbMdZMSfLH11Oril/scedxMmjkxFUewBVRtMCfYwJ5zjkmSMLSV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This commit introduces skip_none_ptes() to skip over all consecutive none ptes in zap_pte_range(), which helps optimize away need_resched() + force_break + incremental pte/addr increments etc. Suggested-by: David Hildenbrand Signed-off-by: Qi Zheng --- mm/memory.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index bd9ebe0f4471f..24633d0e1445a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1657,6 +1657,28 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, return nr; } +static inline int skip_none_ptes(pte_t *pte, unsigned long addr, + unsigned long end) +{ + pte_t ptent = ptep_get(pte); + int max_nr; + int nr; + + if (!pte_none(ptent)) + return 0; + + max_nr = (end - addr) / PAGE_SIZE; + nr = 1; + + for (; nr < max_nr; nr++) { + ptent = ptep_get(pte + nr); + if (!pte_none(ptent)) + break; + } + + return nr; +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1682,13 +1704,17 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_t ptent = ptep_get(pte); int max_nr; - nr = 1; - if (pte_none(ptent)) - continue; - if (need_resched()) break; + nr = skip_none_ptes(pte, addr, end); + if (nr) { + addr += PAGE_SIZE * nr; + if (addr == end) + break; + pte += nr; + } + max_nr = (end - addr) / PAGE_SIZE; if (pte_present(ptent)) { nr = zap_present_ptes(tlb, vma, pte, ptent, max_nr,