From patchwork Wed Feb 1 08:17:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13123913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81E12C38142 for ; Wed, 1 Feb 2023 08:16:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76B646B007E; Wed, 1 Feb 2023 03:16:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F79C6B0080; Wed, 1 Feb 2023 03:16:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40F036B0081; Wed, 1 Feb 2023 03:16:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 27CA36B007E for ; Wed, 1 Feb 2023 03:16:02 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 01ACB1408F4 for ; Wed, 1 Feb 2023 08:16:01 +0000 (UTC) X-FDA: 80418014964.02.BECCEB8 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf15.hostedemail.com (Postfix) with ESMTP id C606AA001D for ; Wed, 1 Feb 2023 08:15:59 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=D5JsGAfK; spf=pass (imf15.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675239360; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A8TUmg8Ng3w8EXE+p8s+IzBj1XvOj0ff6BNpqb3Ep6Q=; b=qxS9sNSXCbGlvr9xNoZKSbhuqS+tRTW5cpQw6/ZjugKpR0NVBH+Tsuq8XNx5k1WzdWTq+G ajDJWJ6MxCdFLrGooZ0RpKeHkU6mX27TiXdcByv8y1sDSQtpLg87dq+1xWdFmpdVHs0aPt DYTKKvaHT6JlKRIuPXm0T+okuAsQwrc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=D5JsGAfK; spf=pass (imf15.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675239360; a=rsa-sha256; cv=none; b=Y6zTGllwkuBykSybesH9njPI/YXA97RAmz6akYkdGDSCD0GYkKDe+MqElOsITIJKC0dehP PE7DIjx0mdysPvhWAvupSKiEqiI/9ompaW9zH1+8i7k/YZCR1ggwos23jT3zZuCBPlx8KT z4JrCzx5kZsAMATb/F3A9kvccGSQRto= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675239359; x=1706775359; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FELofmlHKe7JG0HLem2mmDqUAOxt0yESMF6m7PRi5mg=; b=D5JsGAfKDIYaG5tuwKLK95+b2UT7nkchAH+TnwR+seRg9PhGSawGdLOG MGW4aqLp7y+rLrwdzBh3dXzJbL5iLYjxqG0LRfn+sIK5GTduLkVs365hw i2uymqVWA6i9BQDINpbCV7VPJXblMW6x8sdy/LnFn7jsv5QgI7yHsJejc yjOx++SXAZf+e3liuBC09730cWXSnuZHKEKCvLykutvfdskToKe5pkJ3M dCkew558aovJFHQPHeIMyaH/udDP1vUZt3ueMTYtqTLqII/gIgf6lZFBR L+NzVNdRyVX/La+8zNvc9FN7LRLc5BHo7RizB0kBYVqN2bdFKc6qs3bQq Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10607"; a="390471713" X-IronPort-AV: E=Sophos;i="5.97,263,1669104000"; d="scan'208";a="390471713" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Feb 2023 00:15:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10607"; a="753580383" X-IronPort-AV: E=Sophos;i="5.97,263,1669104000"; d="scan'208";a="753580383" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by FMSMGA003.fm.intel.com with ESMTP; 01 Feb 2023 00:15:57 -0800 From: Yin Fengwei To: willy@infradead.org, david@redhat.com, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH v2 5/5] filemap: batched update mm counter,rmap when map file folio Date: Wed, 1 Feb 2023 16:17:37 +0800 Message-Id: <20230201081737.2330141-6-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230201081737.2330141-1-fengwei.yin@intel.com> References: <20230201081737.2330141-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Stat-Signature: 6uuiar6hapwqdomr78jx4ckisgkiuexa X-Rspam-User: X-Rspamd-Queue-Id: C606AA001D X-Rspamd-Server: rspam06 X-HE-Tag: 1675239359-586800 X-HE-Meta: U2FsdGVkX1+/U7w8URMEOZ8uoZoM2E+7cFWXG1K78/KZ0P3URVHGTCII9tWHtFxmP3zm+/yzOXzs6bnlP4f1fDyjIuV0Lt6oSNXWqhgX5P/7SaftB+32NRG+4ptw4j8+Gd0Lcu8Fhzyp95aKjkQ/QGTD1ph6gBM3+USslb/7Rmto5Kplud9uzXAh5++NErwepFh25BHD6UjUkxO1qXo3xWi/F3zYCcXFmJoMkkOcoqE4/3WSoBtKjTWnbOKmYzzteiQaYiGZJEfIZqeVxExMX9mk1DaSlYHyJnyEPk5l+ax73FsjSTFFZ5JHhJVQJZSvsmbjffDOuVENR6MHUjTvKEUvdyKpGHDjWlSEUNnoiJZzgSEXV8m9hiciMMIIQmSPBf4E0umYW3q8PFfxxhegKs4EsHesY48ydcr0bMElw4D6kvymfp0CPx/5RRBa0B5arOAVO8ES+/XY0+4PrRv3QdzrhL4y6+qCD+rEoMIlaVi7dWwcdUZ72LHQaw/GmM6qzqk1VxTJVwZbLq443imAK/V58uHDX80vDBI1U5c/qObe15vC6qBYZvTyYYIeDUT5U6P+VGzYCYYNC9a0HsrKtOeXEcHM+cwxhR4WW8iOv3NC906vMGFKWb9iVXNSZSbgttsg7I1vFRGlowW2j+RWuXk8FaH/UCvq2/H98+2ar91s1fYZHawxE7WWzouN4qe4nuHMM5Qrut0j3+Q7kXxXEJThHb6/ILZbcIe+T/8J/MYM/6VKanhIesgEhDO8OOIUk/AT5xz17Xn17dOlaeMuUuRVxQEE620BVj8Ljs37CE6Fw82mmUxqIjqcKsxTwTMtSkXXhH/p9j/WqzVbM+0OQu+TSOgKY90nyN4yRpqYRABr/EkYYwYGKNCtuII5MCk/37ARhP1Jl3GvUsq6uwfr8GpE+hsytqqHIXNAZFa8Eo6SSIOrbB1sMTLdhz4WF3th7+EdB9t6h/6HsFt4iWM zbRL3yQY dRbGiSi4jDU5iaPd/gD0nO531ausgFwmh7KvxNA+Au2JuzINLSlX80SMO17WzRt5V4jLam6dYPc34WVTVYAP7kLvBgJtygguuV0TJVUDjfs67XLz6ocnPEGh64wJcVdHZ85y3cT/1FcJrhdj/44MTeMnq3bjK/GYMSSvltLa17hCFdJQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use do_set_pte_range() in filemap_map_folio_range(). Which batched updates mm counter, rmap. With a self cooked will-it-scale.page_fault3 like app (change file write fault to read fault) got 15% performance gain. Perf data collected before/after the change: 18.73%--page_add_file_rmap | --11.60%--__mod_lruvec_page_state | |--7.40%--__mod_memcg_lruvec_state | | | --5.58%--cgroup_rstat_updated | --2.53%--__mod_lruvec_state | --1.48%--__mod_node_page_state 9.93%--page_add_file_rmap_range | --2.67%--__mod_lruvec_page_state | |--1.95%--__mod_memcg_lruvec_state | | | --1.57%--cgroup_rstat_updated | --0.61%--__mod_lruvec_state | --0.54%--__mod_node_page_state The running time of __mode_lruvec_page_state() is reduced a lot. Signed-off-by: Yin Fengwei --- mm/filemap.c | 37 +++++++++++++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 95f634d11581..b14f077d1c55 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3364,11 +3364,22 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct file *file = vma->vm_file; struct page *page = folio_page(folio, start); unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - unsigned int ref_count = 0, count = 0; + unsigned int ref_count = 0, count = 0, nr_mapped = 0; do { - if (PageHWPoison(page)) + if (PageHWPoison(page)) { + if (nr_mapped) { + vmf->pte -= nr_mapped; + do_set_pte_range(vmf, folio, + start + count - nr_mapped, + addr - nr_mapped * PAGE_SIZE, + nr_mapped); + + } + + nr_mapped = 0; continue; + } if (mmap_miss > 0) mmap_miss--; @@ -3378,16 +3389,34 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, * handled in the specific fault path, and it'll prohibit the * fault-around logic. */ - if (!pte_none(*vmf->pte)) + if (!pte_none(*vmf->pte)) { + if (nr_mapped) { + vmf->pte -= nr_mapped; + do_set_pte_range(vmf, folio, + start + count - nr_mapped, + addr - nr_mapped * PAGE_SIZE, + nr_mapped); + + } + + nr_mapped = 0; + continue; + } if (vmf->address == addr) ret = VM_FAULT_NOPAGE; ref_count++; - do_set_pte(vmf, page, addr); + nr_mapped++; } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + if (nr_mapped) { + vmf->pte -= nr_mapped; + do_set_pte_range(vmf, folio, start + count - nr_mapped, + addr - nr_mapped * PAGE_SIZE, nr_mapped); + } + /* * Restore the vmf->pte. Otherwise, it's possible vmf->pte point * to next page table entry if the last sub page in the range is