From patchwork Tue May 15 00:57:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 10399699 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D6B28601F9 for ; Tue, 15 May 2018 00:58:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAC2527EED for ; Tue, 15 May 2018 00:58:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E50028609; Tue, 15 May 2018 00:58:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E59A27EED for ; Tue, 15 May 2018 00:58:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 899186B026B; Mon, 14 May 2018 20:58:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 822136B026C; Mon, 14 May 2018 20:58:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C4E66B026D; Mon, 14 May 2018 20:58:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg0-f71.google.com (mail-pg0-f71.google.com [74.125.83.71]) by kanga.kvack.org (Postfix) with ESMTP id 2A5F26B026B for ; Mon, 14 May 2018 20:58:52 -0400 (EDT) Received: by mail-pg0-f71.google.com with SMTP id z16-v6so6990124pgv.16 for ; Mon, 14 May 2018 17:58:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id; bh=pRuI5lwnHeirjie1WCjZHhRoO2yEgcmYgm+nQ/sHXq8=; b=nHXnRYQ//ZFOiiFWJvZT8Q7c8Rv6Nm0IyUtEuDKPhJWRTeyc6CxZIX4kDZKXrwhGjt 5XOz6oTKCwZjViaUKrBXZQVxTpLlT6KGcRZI3g4OFEBN/zhZ5FqA2WH2uF4YUdYFC/V0 V7Q1FKRpOaFVKe5J02+q9MKOj3KmnYFM2Zr9N/SNGDa+tiC52dypPytqOgaydrC+ecy+ UOd8aT5TSMRNN/jRMnHpKrjVz7UFVosPIzKFmbUhArwQirYOcCMbGohXIl/NjRl7I89w boWhIbUGWKvYoEaYGa8KtP+ZGx4GV9vdDmw34mRWu0CuASLhuXGu+Jmbm8K8XB4/Kvky /cQw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Gm-Message-State: ALKqPwdRWTOiaJOiYD6D5bBXNekSD7OmWLFuueKlup2fOOgDU55BXpWd fxS+y8PyTIKcDviULg0r8m6ajNBav3PKc1NyGwjZDZRExva7l8tN0VXcaDWsLyornuQcLHpanOB /9jzoytVaqg7HD7sB2m8sQq1zK9wlMYwKQYgatgM9Rfmt/NlqW+gjPj65hk78H++knA== X-Received: by 2002:a17:902:7782:: with SMTP id o2-v6mr12365198pll.247.1526345931729; Mon, 14 May 2018 17:58:51 -0700 (PDT) X-Google-Smtp-Source: AB8JxZryHtrAyQ9kPod81T0YnX+h9N2OHWT+JVnZ6x1qBdF+5lgLhgtJrm0hFXWVejMEwFPNz7Yf X-Received: by 2002:a17:902:7782:: with SMTP id o2-v6mr12365167pll.247.1526345930783; Mon, 14 May 2018 17:58:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526345930; cv=none; d=google.com; s=arc-20160816; b=D4lRWxr3LYNPC3uKgJ8OsVmlFPprpd2t4Ukoaiqm8sDijRQirw8zmuCVCO3bCYDLgR wpaSPD3l+DPHFlZqXU7h4tR98iDHjBwL2dg/tTJOtlXwi8fMS69VUoNZoHfBv3gaF8Kf fh/rmUTQxyuiFwrpX9fS1n21jk2s3OFOJ4BKDPmgftDAtHAhhK2LsY5031EmxcIQIS1X crav8qwBsPDphWpgxCZ2J1BLcHX80luO+4UexIB0MF1Eu9V0knfJUWRSX31aqCa8zHZW FCwaTUQNKfECPQVJiLXCuFQsTDXYoD7sSJb1AMZwIGCKxSFrQ653nio3d1k3Xr1Vv+u/ QYaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from:arc-authentication-results; bh=pRuI5lwnHeirjie1WCjZHhRoO2yEgcmYgm+nQ/sHXq8=; b=VzINwmlzx49AbDkx4CKDvj8T5a0rKAXh2JOz4eTdrRspcYesOmigAtO9mqpS7Dh79+ U6pAzbzHQDygJzIZYtnpvsd/T5sxaJo2YLieC7WxfsIzbtQ4rU2MnurTzO2hzS90Dxox ygHh5AP3xVFmkLnmSbeXm3vDmH/nExZqr8t+Q9ttE9Hl/ZoMYbYgGc95R2gbfzCJfBVo QXUXBQ5LeUYlTlHft/pO/zcLX6g+l9knBQKsigeLYYa3m21Yhq6+RgyeCcWUaECbV25L Nuk/uz5iiXb0GtJTts37qzoMBrpruQj8dIBy6FrSbHQwqGxGealMhZd3yiT8sILjsfCs MSLg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id j10-v6si10761466pfn.87.2018.05.14.17.58.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 14 May 2018 17:58:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ying.huang@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 May 2018 17:58:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,402,1520924400"; d="scan'208";a="228568524" Received: from yhuang-gentoo.sh.intel.com ([10.239.193.219]) by fmsmga006.fm.intel.com with ESMTP; 14 May 2018 17:58:45 -0700 From: "Huang, Ying" To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Andrea Arcangeli , "Kirill A. Shutemov" , Andi Kleen , Jan Kara , Michal Hocko , Matthew Wilcox , Hugh Dickins , Minchan Kim , Shaohua Li , Christopher Lameter , "Aneesh Kumar K.V" , Punit Agrawal , Anshuman Khandual Subject: [PATCH -mm] mm, hugetlb: Pass fault address to no page handler Date: Tue, 15 May 2018 08:57:56 +0800 Message-Id: <20180515005756.28942-1-ying.huang@intel.com> X-Mailer: git-send-email 2.16.1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Huang Ying This is to take better advantage of huge page clearing optimization (c79b57e462b5d, "mm: hugetlb: clear target sub-page last when clearing huge page"). Which will clear to access sub-page last to avoid the cache lines of to access sub-page to be evicted when clearing other sub-pages. This needs to get the address of the sub-page to access, that is, the fault address inside of the huge page. So the hugetlb no page fault handler is changed to pass that information. This will benefit workloads which don't access the begin of the huge page after page fault. With this patch, the throughput increases ~28.1% in vm-scalability anon-w-seq test case with 88 processes on a 2 socket Xeon E5 2699 v4 system (44 cores, 88 threads). The test case creates 88 processes, each process mmap a big anonymous memory area and writes to it from the end to the begin. For each process, other processes could be seen as other workload which generates heavy cache pressure. At the same time, the cache miss rate reduced from ~36.3% to ~25.6%, the IPC (instruction per cycle) increased from 0.3 to 0.37, and the time spent in user space is reduced ~19.3% Signed-off-by: "Huang, Ying" Cc: Andrea Arcangeli Cc: "Kirill A. Shutemov" Cc: Andi Kleen Cc: Jan Kara Cc: Michal Hocko Cc: Matthew Wilcox Cc: Hugh Dickins Cc: Minchan Kim Cc: Shaohua Li Cc: Christopher Lameter Cc: "Aneesh Kumar K.V" Cc: Punit Agrawal Cc: Anshuman Khandual Reviewed-by: Mike Kravetz Acked-by: David Rientjes Acked-by: Michal Hocko --- mm/hugetlb.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 129088710510..3de6326abf39 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3677,7 +3677,7 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping, static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, unsigned int flags) + unsigned long faddress, pte_t *ptep, unsigned int flags) { struct hstate *h = hstate_vma(vma); int ret = VM_FAULT_SIGBUS; @@ -3686,6 +3686,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct page *page; pte_t new_pte; spinlock_t *ptl; + unsigned long address = faddress & huge_page_mask(h); /* * Currently, we are forced to kill the process in the event the @@ -3749,7 +3750,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, ret = VM_FAULT_SIGBUS; goto out; } - clear_huge_page(page, address, pages_per_huge_page(h)); + clear_huge_page(page, faddress, pages_per_huge_page(h)); __SetPageUptodate(page); set_page_huge_active(page); @@ -3871,7 +3872,7 @@ u32 hugetlb_fault_mutex_hash(struct hstate *h, struct mm_struct *mm, #endif int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long faddress, unsigned int flags) { pte_t *ptep, entry; spinlock_t *ptl; @@ -3883,8 +3884,7 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, struct hstate *h = hstate_vma(vma); struct address_space *mapping; int need_wait_lock = 0; - - address &= huge_page_mask(h); + unsigned long address = faddress & huge_page_mask(h); ptep = huge_pte_offset(mm, address, huge_page_size(h)); if (ptep) { @@ -3914,7 +3914,7 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, entry = huge_ptep_get(ptep); if (huge_pte_none(entry)) { - ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, flags); + ret = hugetlb_no_page(mm, vma, mapping, idx, faddress, ptep, flags); goto out_mutex; }