From patchwork Wed Oct 31 08:19:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kuo-Hsin Yang X-Patchwork-Id: 10662137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5872E3E9D for ; Wed, 31 Oct 2018 08:20:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 462B32A36D for ; Wed, 31 Oct 2018 08:20:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 397E42A380; Wed, 31 Oct 2018 08:20:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 841F02A381 for ; Wed, 31 Oct 2018 08:20:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58DEC6B0303; Wed, 31 Oct 2018 04:20:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 53E406B0305; Wed, 31 Oct 2018 04:20:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E0C96B0310; Wed, 31 Oct 2018 04:20:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id EBE176B0303 for ; Wed, 31 Oct 2018 04:20:02 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id n5-v6so11764613plp.16 for ; Wed, 31 Oct 2018 01:20:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:mime-version:content-transfer-encoding; bh=EuyJeAjt0VzpU8BPMav3ygqDxkI3/Tz4YSQEhr29gtk=; b=HbdGDXhvERDsuYBBTO5e+zhoa6omKfTIYf3eHXnz+MKFtnuj89wxUyBmf7wvg7cH5W OofMPZEXbccHXJTKV4XIhFi78dMX58bkYN/HT7gcHiiJ7lD83yx+ZwwnOPUqQGUHu7Vw OC8+jex2tqH6UCpzYUuqy5zmldVl9MM/acqOL2EV2vAp4Gk5y25HZld8kkBSxSR6+qZ8 uzPnHXorJvNJZ40C0oRjI08AO708/7MjO/ju5hOtAPuUeVA07Bqex6eeVsaIgCklE1zx dNy5m7746GNOR03TF9rxKE+g/i1vDNUmZS8XW5pb07U8rJGgRmz+b63o9tTdhhnZtJRs q5vg== X-Gm-Message-State: AGRZ1gI2ZNekuXRfkmq586Kr1trMDId9a2b8S1e4fxgqCEXdY2izws/L +4PlOc6hlARbF27vNnnhcbdktaxtlQc4tXpYtCifYExy4yAnopx9OYMfxlA64DOAKtjl1mNx3NG Jb9g2i2vkX7CoJzQzttuZkzcJPVEZEu5ieQ8SW8AbuF48lshoBZKGdEOuWgJALyaBXD8gYobDUT vGvCUK2j/SiL+HjpmUZUzpUUYCxYqUNo4xY/3ReHG10dRBviQ51IHeh6/15rStNpYUCQnDAwver azoKo2jF3sSDtw9ErCVYkWyBZ8O83RufOomeYbUmbSI3sr0UrcAYlKYuKooumzjeA406TiVjfgo EjzoPTP6UHsemTsHHC7izGEweTfnr1lHecmJYJaW3Driu8XkApCwsJZyEbBmh04h4y4Yp9uiNzY 4 X-Received: by 2002:a17:902:6e08:: with SMTP id u8-v6mr2346385plk.64.1540974002538; Wed, 31 Oct 2018 01:20:02 -0700 (PDT) X-Received: by 2002:a17:902:6e08:: with SMTP id u8-v6mr2346335plk.64.1540974001468; Wed, 31 Oct 2018 01:20:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1540974001; cv=none; d=google.com; s=arc-20160816; b=laM18ClDa8c8d0LAXFcEQ/q8A7KhqroZJyaJH9gf/KJvVcoGnKrI8Mk8KviZC+vzsL Yum7vBV0DgHG9I2BSoGyFxvdtRrWvlEruLeYzFzAM1Hfv2nYMvTEnkuH7OMJRWybhgNd kSH+Dpk5np5S37+7jIG5wD699LLnPgFVLyfbQ3vBcjj+3qbNNyiYo2g6hVLuF861r7Ej b/pb5pp5XE7rUGxUJCTkE7rfdU4ziD9ETPpBkJJOPkJVepc7nPbBf4O0tksvguj5YbiD P4kDOC5LupFJVaWfeZYs1B8ec9SoNhYhne+rfXNXsQl8O6INToQQLNomxBlikSADDfVL dveQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature; bh=EuyJeAjt0VzpU8BPMav3ygqDxkI3/Tz4YSQEhr29gtk=; b=BaJql705KAyU8EyDGk/g9RVafcaEB+7yT+99E+geyouKujn8LspHkd78KNy52JwNpx ysTJqnW48EvStDhz6/aac61wJKuXcqlj9QG+dVSW22hxfgshZjgrpWZgT1s7btqgw1ow tT9aTXnGLc126sszxpexT7bZV/Ez3v6diDd37QxgAFxZwN3DrzPnZUnpcgR3buSrobHX UdHg7Deou2kmc75t5oVIVBUB+qUpU3lzJ8xtXKgFT6Xzhz4kgIqENKcr5RC3prqstriV /XKndUHm013l9/7QV9gMbOFw/dB/GYJ/o0kldEuZZtUNQOtjuRRNk7dwCDjdUurSKekU Xvzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=RJTUP437; spf=pass (google.com: domain of vovoy@chromium.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=vovoy@chromium.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 193-v6sor26396848pgb.32.2018.10.31.01.20.01 for (Google Transport Security); Wed, 31 Oct 2018 01:20:01 -0700 (PDT) Received-SPF: pass (google.com: domain of vovoy@chromium.org designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=RJTUP437; spf=pass (google.com: domain of vovoy@chromium.org designates 209.85.220.65 as permitted sender) smtp.mailfrom=vovoy@chromium.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=EuyJeAjt0VzpU8BPMav3ygqDxkI3/Tz4YSQEhr29gtk=; b=RJTUP437sWK17zl93fXi0TDylzWa4+k64vu5s+v1WVQsf2etc3TbM92QfrSPdxZO1e i1CHmJSye8pZ2NBs8YRZo2F0JsVkr3cpiA4M4U+kYQKg3bZp+lMhD19YPEXRlkL0KCxU 60PnbEOp4XrIi4hkStiZsvdHvamS+xdSj76b8= X-Google-Smtp-Source: AJdET5fETXW8+jDNL4QL8Unz75pxaoAtCqJ6oAXZ8NgilsTzq56PUl2Bn2yfksLjsMIitGaX0rPQrA== X-Received: by 2002:a63:a84a:: with SMTP id i10mr2199477pgp.263.1540974000906; Wed, 31 Oct 2018 01:20:00 -0700 (PDT) Received: from vovoy-z840.tpe.corp.google.com ([2401:fa00:1:b:d89e:cfa6:3c8:e61b]) by smtp.gmail.com with ESMTPSA id f14-v6sm4833177pgr.87.2018.10.31.01.19.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Oct 2018 01:20:00 -0700 (PDT) From: Kuo-Hsin Yang To: linux-kernel@vger.kernel.org, intel-gfx@lists.freedesktop.org, linux-mm@kvack.org Cc: Kuo-Hsin Yang , Chris Wilson , Michal Hocko , Joonas Lahtinen , Peter Zijlstra , Andrew Morton , Dave Hansen Subject: [PATCH v3] mm, drm/i915: mark pinned shmemfs pages as unevictable Date: Wed, 31 Oct 2018 16:19:45 +0800 Message-Id: <20181031081945.207709-1-vovoy@chromium.org> X-Mailer: git-send-email 2.19.1.568.g152ad8e336-goog MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The i915 driver uses shmemfs to allocate backing storage for gem objects. These shmemfs pages can be pinned (increased ref count) by shmem_read_mapping_page_gfp(). When a lot of pages are pinned, vmscan wastes a lot of time scanning these pinned pages. In some extreme case, all pages in the inactive anon lru are pinned, and only the inactive anon lru is scanned due to inactive_ratio, the system cannot swap and invokes the oom-killer. Mark these pinned pages as unevictable to speed up vmscan. Add check_move_lru_page() to move page to appropriate lru list. This patch was inspired by Chris Wilson's change [1]. [1]: https://patchwork.kernel.org/patch/9768741/ Cc: Chris Wilson Cc: Michal Hocko Cc: Joonas Lahtinen Cc: Peter Zijlstra Cc: Andrew Morton Cc: Dave Hansen Signed-off-by: Kuo-Hsin Yang --- The previous mapping_set_unevictable patch is worse on gem_syslatency because it defers to vmscan to move these pages to the unevictable list and the test measures latency to allocate 2MiB pages. This performance impact can be solved by explicit moving pages to the unevictable list in the i915 function. Chris, can you help to run the "igt/benchmarks/gem_syslatency -t 120 -b -m" test with this patch on your testing machine? I tried to run the test on a Celeron N4000, 4GB Ram machine. The mean value with this patch is similar to that with the mlock patch. x tip-mean.txt # current stock i915 + lock_vma-mean.txt # the old mlock patch * mapping-mean.txt # this patch N Min Max Median Avg Stddev x 60 548.898 2563.653 2149.573 1999.273 480.837 + 60 479.049 2119.902 1964.399 1893.226 314.736 * 60 455.358 3212.368 1991.308 1903.686 411.448 Changes for v3: Use check_move_lru_page instead of shmem_unlock_mapping to move pages to appropriate lru lists. Changes for v2: Squashed the two patches. Documentation/vm/unevictable-lru.rst | 4 +++- drivers/gpu/drm/i915/i915_gem.c | 20 +++++++++++++++++++- include/linux/swap.h | 1 + mm/vmscan.c | 20 +++++++++++++++++--- 4 files changed, 40 insertions(+), 5 deletions(-) diff --git a/Documentation/vm/unevictable-lru.rst b/Documentation/vm/unevictable-lru.rst index fdd84cb8d511..a812fb55136d 100644 --- a/Documentation/vm/unevictable-lru.rst +++ b/Documentation/vm/unevictable-lru.rst @@ -143,7 +143,7 @@ using a number of wrapper functions: Query the address space, and return true if it is completely unevictable. -These are currently used in two places in the kernel: +These are currently used in three places in the kernel: (1) By ramfs to mark the address spaces of its inodes when they are created, and this mark remains for the life of the inode. @@ -154,6 +154,8 @@ These are currently used in two places in the kernel: swapped out; the application must touch the pages manually if it wants to ensure they're in memory. + (3) By the i915 driver to mark pinned address space until it's unpinned. + Detecting Unevictable Pages --------------------------- diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c index 0c8aa57ce83b..6dc3ecef67e4 100644 --- a/drivers/gpu/drm/i915/i915_gem.c +++ b/drivers/gpu/drm/i915/i915_gem.c @@ -2387,6 +2387,7 @@ i915_gem_object_put_pages_gtt(struct drm_i915_gem_object *obj, { struct sgt_iter sgt_iter; struct page *page; + struct address_space *mapping; __i915_gem_object_release_shmem(obj, pages, true); @@ -2395,6 +2396,9 @@ i915_gem_object_put_pages_gtt(struct drm_i915_gem_object *obj, if (i915_gem_object_needs_bit17_swizzle(obj)) i915_gem_object_save_bit_17_swizzle(obj, pages); + mapping = file_inode(obj->base.filp)->i_mapping; + mapping_clear_unevictable(mapping); + for_each_sgt_page(page, sgt_iter, pages) { if (obj->mm.dirty) set_page_dirty(page); @@ -2402,6 +2406,10 @@ i915_gem_object_put_pages_gtt(struct drm_i915_gem_object *obj, if (obj->mm.madv == I915_MADV_WILLNEED) mark_page_accessed(page); + lock_page(page); + check_move_lru_page(page); + unlock_page(page); + put_page(page); } obj->mm.dirty = false; @@ -2559,6 +2567,7 @@ static int i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) * Fail silently without starting the shrinker */ mapping = obj->base.filp->f_mapping; + mapping_set_unevictable(mapping); noreclaim = mapping_gfp_constraint(mapping, ~__GFP_RECLAIM); noreclaim |= __GFP_NORETRY | __GFP_NOWARN; @@ -2630,6 +2639,10 @@ static int i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) } last_pfn = page_to_pfn(page); + lock_page(page); + check_move_lru_page(page); + unlock_page(page); + /* Check that the i965g/gm workaround works. */ WARN_ON((gfp & __GFP_DMA32) && (last_pfn >= 0x00100000UL)); } @@ -2673,8 +2686,13 @@ static int i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) err_sg: sg_mark_end(sg); err_pages: - for_each_sgt_page(page, sgt_iter, st) + mapping_clear_unevictable(mapping); + for_each_sgt_page(page, sgt_iter, st) { + lock_page(page); + check_move_lru_page(page); + unlock_page(page); put_page(page); + } sg_free_table(st); kfree(st); diff --git a/include/linux/swap.h b/include/linux/swap.h index d8a07a4f171d..a812f24d69f2 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -370,6 +370,7 @@ static inline int node_reclaim(struct pglist_data *pgdat, gfp_t mask, extern int page_evictable(struct page *page); extern void check_move_unevictable_pages(struct page **, int nr_pages); +extern void check_move_lru_page(struct page *page); extern int kswapd_run(int nid); extern void kswapd_stop(int nid); diff --git a/mm/vmscan.c b/mm/vmscan.c index 62ac0c488624..2399ccaa15e7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4184,12 +4184,11 @@ int page_evictable(struct page *page) #ifdef CONFIG_SHMEM /** - * check_move_unevictable_pages - check pages for evictability and move to appropriate zone lru list + * check_move_unevictable_pages - move evictable pages to appropriate evictable + * lru lists * @pages: array of pages to check * @nr_pages: number of pages to check * - * Checks pages for evictability and moves them to the appropriate lru list. - * * This function is only used for SysV IPC SHM_UNLOCK. */ void check_move_unevictable_pages(struct page **pages, int nr_pages) @@ -4234,3 +4233,18 @@ void check_move_unevictable_pages(struct page **pages, int nr_pages) } } #endif /* CONFIG_SHMEM */ + +/** + * check_move_lru_page - check page for evictability and move it to + * appropriate zone lru list + * @page: page to be move to appropriate lru list + * + * If this function fails to isolate an unevictable page, vmscan will handle it + * when it attempts to reclaim the page. + */ +void check_move_lru_page(struct page *page) +{ + if (!isolate_lru_page(page)) + putback_lru_page(page); +} +EXPORT_SYMBOL(check_move_lru_page);