From patchwork Thu Dec 10 00:43:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 980E6C4361B for ; Thu, 10 Dec 2020 00:43:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1656323CD4 for ; Thu, 10 Dec 2020 00:43:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1656323CD4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6CC816B005D; Wed, 9 Dec 2020 19:43:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67E946B0068; Wed, 9 Dec 2020 19:43:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AC6A6B006C; Wed, 9 Dec 2020 19:43:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0167.hostedemail.com [216.40.44.167]) by kanga.kvack.org (Postfix) with ESMTP id 2CA696B005D for ; Wed, 9 Dec 2020 19:43:41 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E87948249980 for ; Thu, 10 Dec 2020 00:43:40 +0000 (UTC) X-FDA: 77575524600.23.hill03_5e14469273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id C683F37606 for ; Thu, 10 Dec 2020 00:43:40 +0000 (UTC) X-HE-Tag: hill03_5e14469273f4 X-Filterd-Recvd-Size: 4426 Received: from mail-qv1-f67.google.com (mail-qv1-f67.google.com [209.85.219.67]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:40 +0000 (UTC) Received: by mail-qv1-f67.google.com with SMTP id s6so1623670qvn.6 for ; Wed, 09 Dec 2020 16:43:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=YDDAi2s6I9irj1eFCp+Xpz5CYfQrtxxsgY8jH6RM458=; b=Nk+AuTMfxQ6BbAHmfpjqdXSFnw1CX0GCnpt5Rtr+S5ZZHzRoxhUkXzjSaLBFo9FKnP +IGv/oMQZ60Z5oFcu+WcBKLRJVlJIYOStwwQhx8A63/pkfbIk01AMMQrVYLFm122JdCB ZhbhX33sTQJUJplzJA2KCaOHlmGycGbsVHpDDZmMcTiFO6zgEcZ5bn3oKR/nhS7NmGXE lSUNz94k7ztCGv2c6TPO+oYZAaXQIIpJzaTSorVPYYAS5rRfsh2MWxhI1wbtSM8fm6SD ty0uVYq1b+GXDXOHEDRKOXvQB09Qf6VhbIcqJueEsD6VjWHJFlCxwCjTIUSuBPWyseQi q3JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YDDAi2s6I9irj1eFCp+Xpz5CYfQrtxxsgY8jH6RM458=; b=mGloXlfBvxYZe8kVcbjwttOVZ98jPXull8kmjT3WJHp3TBgRrWbzTcPsaft11+dWjG Fyt4awHiJo5IdurrxSg1fyre74Svp0ddH28Ta3tXN1PKHy3bOj2nBPwqlDKOJ4kE9CAB DHZ8GUEr3xS7wRSq100cSgKese/tOV89f/MgDPSv0XPOip5ohFb5tcZFImqghMV5Z0tF mZ2s+ET3REgivlJlTOzsfs/4Kc64lOpj2lWhmKGpzhBw8Pn106UUSS82QkB4+DKlTh20 mpacZ7fcgeMt50mAomaaG/q5zPF11T0NMuLVy8POQdil+OiNlM26m7vk15GHDTwHK4fZ RJXA== X-Gm-Message-State: AOAM533V0u+gb5GNQdPezAV8OXqbZZFIk8btVg6WZ1M0TrB+1EZtHUaJ pM2vXfc/o8+EEFKg82WGtilEAw== X-Google-Smtp-Source: ABdhPJzAa7q5sUByKWOSFZ0Py2q3Uc376O6i1oNIXW+AdlNdeOTECHVZXZpiUJYIRGDv8y9xQIyh+Q== X-Received: by 2002:ad4:4051:: with SMTP id r17mr5889435qvp.39.1607561019788; Wed, 09 Dec 2020 16:43:39 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:39 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 1/8] mm/gup: perform check_dax_vmas only when FS_DAX is enabled Date: Wed, 9 Dec 2020 19:43:28 -0500 Message-Id: <20201210004335.64634-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is no need to check_dax_vmas() and run through the npage loop of pinned pages if FS_DAX is not enabled. Add a stub check_dax_vmas() function for no-FS_DAX case. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard Reviewed-by: Pankaj Gupta --- mm/gup.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 98eb8e6d2609..cdb8b9eeb016 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1568,6 +1568,7 @@ struct page *get_dump_page(unsigned long addr) #endif /* CONFIG_ELF_CORE */ #if defined(CONFIG_FS_DAX) || defined (CONFIG_CMA) +#ifdef CONFIG_FS_DAX static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) { long i; @@ -1586,6 +1587,12 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) } return false; } +#else +static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) +{ + return false; +} +#endif #ifdef CONFIG_CMA static long check_and_migrate_cma_pages(struct mm_struct *mm, From patchwork Thu Dec 10 00:43:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94877C4167B for ; Thu, 10 Dec 2020 00:43:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2A84823CD1 for ; Thu, 10 Dec 2020 00:43:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A84823CD1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2FCF56B0068; Wed, 9 Dec 2020 19:43:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B1626B006C; Wed, 9 Dec 2020 19:43:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1795D6B006E; Wed, 9 Dec 2020 19:43:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id D56806B0068 for ; Wed, 9 Dec 2020 19:43:42 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A21D71EE6 for ; Thu, 10 Dec 2020 00:43:42 +0000 (UTC) X-FDA: 77575524684.01.jail86_5d09ebd273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 828C410045EE1 for ; Thu, 10 Dec 2020 00:43:42 +0000 (UTC) X-HE-Tag: jail86_5d09ebd273f4 X-Filterd-Recvd-Size: 4288 Received: from mail-qv1-f68.google.com (mail-qv1-f68.google.com [209.85.219.68]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:42 +0000 (UTC) Received: by mail-qv1-f68.google.com with SMTP id u16so1624241qvl.7 for ; Wed, 09 Dec 2020 16:43:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Vo/yLKAiBS5h7nMLASyDzuT56nkaTKWz46PdLnDqyVo=; b=lWcq4VQzxkaWPyNizEDNGYhGjugYHnHlB7gIkV8AVNth2RU1O8bPJqDyqXkjGelUBb DtuSfCf2vU3iQ2Aw2aHIB/OUhtzdIKvp9GPqGanu6vYYeLYd8uHYXfCfkRfHwbkwK0nn AA+lWcTiJyWmv/jcnvBHdVb8hyCNhk7oArTBfqp+Rx1gSVn6hSBDo+x8ejNat3qPP7Pw I0ouWxKweeSLCAuynp4EjKkntCkncxKCEJmdCf3dlExR9b+ZDCgZWGnvRPaP8NUKW7G2 V244hehbzwIJzbQTSVR+BTFmASHF1nXUTfYoDWSs1CLrVlB3p++uRUQ/RlMTreKpyCvF RO7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Vo/yLKAiBS5h7nMLASyDzuT56nkaTKWz46PdLnDqyVo=; b=ueBAZGSe2zCp+2unzoLPaFdsYH9CstSq7ZonTNp/Q5WB0u0DJQdHNbYQXjaFJ4bqP9 GOjTgWu0cMMJIpqT+k4QRMnwnNoD5sDAu4Vz8g4sRSdxDwo0h+kNMyiru/ImnIMqHM3J ant4CA60wFJZvMbWt4r+bBg7s4CeV6wLsomlRmgxEq2PzB16O8J6sqebGdNToPGEO0DK 053rovyi0eNQWrDDaqJl829u6G8z8rO5iR6UWj6n+p4JCjsqDKJVn4cjKmcVakb4aUWL neZnI1EWp2PLtGqYUZdrBV7pmj8/3ZudDn83kc4hXJxH8rX5cv3KYMVnoTpUJygUSE/R WBUg== X-Gm-Message-State: AOAM533+9wJjY8jSPjjz10VLSnhFwbujeOB8uNoXwojdq23PR1Dps+Pe OwqGLOJNxuFwmT99qtljgkaUTw== X-Google-Smtp-Source: ABdhPJydALvSGS7GDNYtKUswYkrylNumV2tLiFG9B8JTmcFkB18LgsEdm1vhHem+5qCNXzlbBRkswg== X-Received: by 2002:a0c:e6e7:: with SMTP id m7mr6461346qvn.11.1607561021480; Wed, 09 Dec 2020 16:43:41 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:40 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 2/8] mm/gup: don't pin migrated cma pages in movable zone Date: Wed, 9 Dec 2020 19:43:29 -0500 Message-Id: <20201210004335.64634-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order not to fragment CMA the pinned pages are migrated. However, they are migrated to ZONE_MOVABLE, which also should not have pinned pages. Remove __GFP_MOVABLE, so pages can be migrated to zones where pinning is allowed. Signed-off-by: Pavel Tatashin Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Acked-by: Michal Hocko --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index cdb8b9eeb016..3a76c005a3e2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1610,7 +1610,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, - .gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_NOWARN, + .gfp_mask = GFP_USER | __GFP_NOWARN, }; check_again: From patchwork Thu Dec 10 00:43:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B7A5C433FE for ; Thu, 10 Dec 2020 00:43:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2358223B98 for ; Thu, 10 Dec 2020 00:43:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2358223B98 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E9AB56B006C; Wed, 9 Dec 2020 19:43:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D64576B006E; Wed, 9 Dec 2020 19:43:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFB9A6B0070; Wed, 9 Dec 2020 19:43:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id A78B56B006C for ; Wed, 9 Dec 2020 19:43:44 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 77C698249980 for ; Thu, 10 Dec 2020 00:43:44 +0000 (UTC) X-FDA: 77575524768.07.plot91_5c108c7273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 5ED6F1803F9AF for ; Thu, 10 Dec 2020 00:43:44 +0000 (UTC) X-HE-Tag: plot91_5c108c7273f4 X-Filterd-Recvd-Size: 4757 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:43 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id 1so3324061qka.0 for ; Wed, 09 Dec 2020 16:43:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=wKF1vMMrBlL3Nf1z4M/UQUVU8LetyRUPTbIEfnAIYlQ=; b=gqbd9DoMaRvvzvvMIqaS5F41tXsvEg/1IPa7tX68ji4V8ZFM0lAhGyUZ1CmiVZXVSh biDr/Ej0RPp58RM/ATaFiF2WLnZCp9mVyqJVF8saPqoaMIdnbxPdA9IzLH1s2jY1UK4Y H2lHueaV79j1DW3rTXM50YMPOTq29GTP2KnfVIJxkLxzhdbc+fjGIE2Eq7Ue0jnYz0A/ cW2to2nIGGjeyaZYDoshUaphaprRyBhQwGVqtTeymXkz9SP2YoPNh3LxiaImeY19SgbI w5Q+FiKl6hlqVYo0yVw5f3cEDEQ9eba8GOePohl4kJKP3tZk/EQx4jdfOaSRMeVu6A09 tSPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wKF1vMMrBlL3Nf1z4M/UQUVU8LetyRUPTbIEfnAIYlQ=; b=pzp9bJUgixc8xHcoLKDZKpDl8/VOHI7i52rFKIl5JsbFxfIJwWslQLronVt0SsUUV4 EdgnqRf6mwBB2d+MCeJqD1VMPfEohA1paXrzm7OgxNjJfNeq2sEAyqBEKjne03dzz44r Xr4D09kKwv0KsklXki1IQ5QC3YEeKDwjuf2hZMZiO1lKS0ocBa+31KtdLc5S9zYBPx0C XUl6/5ihglM8ro2pClWZDGw+7134OWW/FGbS4eluDlZbcLLLg/q8J0URqltv6I7MZJmg 1DdGRMZkVB9mtHjxFfNkakNSe4d9Nhhbv0xD8MSv8GNOPvOR05JXP8DMHo8/F8DwppJa vpPA== X-Gm-Message-State: AOAM533yklU9dSBDCT/v/ZUkn/Nb1iQD7+d6B+I1e6y5BM5Uo0Gwxdij gsc1AHCYphRKrJVktsw9Rf2IKA== X-Google-Smtp-Source: ABdhPJwVjQblG8lz4nHLZuO6EXQnNDMXKQRm0jhOeWrGTKeTLfE1Ij+ZT0B/O2DFXL+mzXgo5kPtBQ== X-Received: by 2002:a37:8384:: with SMTP id f126mr5922627qkd.500.1607561023216; Wed, 09 Dec 2020 16:43:43 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:42 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 3/8] mm/gup: make __gup_longterm_locked common Date: Wed, 9 Dec 2020 19:43:30 -0500 Message-Id: <20201210004335.64634-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __gup_longterm_locked() has CMA || FS_DAX version and a common stub version. In the preparation of prohibiting longterm pinning of pages from movable zone make the CMA || FS_DAX version common, and delete the stub version. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard Reviewed-by: Ira Weiny --- mm/gup.c | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 3a76c005a3e2..0e2de888a8b0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1567,7 +1567,6 @@ struct page *get_dump_page(unsigned long addr) } #endif /* CONFIG_ELF_CORE */ -#if defined(CONFIG_FS_DAX) || defined (CONFIG_CMA) #ifdef CONFIG_FS_DAX static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) { @@ -1757,18 +1756,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, kfree(vmas_tmp); return rc; } -#else /* !CONFIG_FS_DAX && !CONFIG_CMA */ -static __always_inline long __gup_longterm_locked(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int flags) -{ - return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, - NULL, flags); -} -#endif /* CONFIG_FS_DAX || CONFIG_CMA */ static bool is_valid_gup_flags(unsigned int gup_flags) { From patchwork Thu Dec 10 00:43:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 419EEC4167B for ; Thu, 10 Dec 2020 00:43:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0211E22D5B for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0211E22D5B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A03776B006E; Wed, 9 Dec 2020 19:43:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DD146B0070; Wed, 9 Dec 2020 19:43:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 719F96B0071; Wed, 9 Dec 2020 19:43:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0042.hostedemail.com [216.40.44.42]) by kanga.kvack.org (Postfix) with ESMTP id 4FDE36B006E for ; Wed, 9 Dec 2020 19:43:46 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1A184181AEF23 for ; Thu, 10 Dec 2020 00:43:46 +0000 (UTC) X-FDA: 77575524852.08.nest41_08058ca273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id F0E3F1819E769 for ; Thu, 10 Dec 2020 00:43:45 +0000 (UTC) X-HE-Tag: nest41_08058ca273f4 X-Filterd-Recvd-Size: 8394 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:45 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id w79so3309528qkb.5 for ; Wed, 09 Dec 2020 16:43:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=58dOOX3CT4RgbZN4Iw/Ztczf8OW6HingBFIIO3HxvKE=; b=QEV+Vv5BZix6tq+rJigcucRn7NtniibHkkvq84g/WGOiHZ+79JPaLLH0XQIJF1YpBn jyk4ywwH8Wd4bXSoWUbrxq22Zdp348uDTDbWjKxjrAP4qqQ7uQkJUsgSxIE2dw2Z6kGy uNK1FSp+nRR4R17lIUddn8W6RFqwj+d9uTiFbnbDGawrK+6TuQDkbA8uEFNCrkRX99O9 Hx6YLXr5cvmMQbFxTVby4sz/s8H1f0tIPwltNr+VkP4l39RTg70mZVTc3GwG1/SKvgDg 59cDntVv6qlbFWagb0yrhikE0LwGD+hvICZVBhETHT09wrSKOZSdz2PGBrAxpfKxOLAC j0ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=58dOOX3CT4RgbZN4Iw/Ztczf8OW6HingBFIIO3HxvKE=; b=gUlmxMCeZVzLp+vDB2txgw8upm6yB69wXpblXvS/JciB/Jgcf7f7NF5CcuzY5L3xGB AOZXcUlpazHhGj4qNja8SwcEQeX6e6mbIvbPOJioUTEDPmOtAL6+mUuGLBnJBLv9yX3r joe+38ga7MZ90MoT8wa4rGNgrb4E6NKbvww6QpgjJN0wyhTl78BhdBJZr+lncIqXMGZs oKJ0lRg3GwM+3LTlsAuaQm7+T5X1DHkCemFXysujywFV66RyrI1btEDbykFxp/6TZiLP 6HCaacBSJbTpfJiMOxpfy8IQgIibHy84W8s9hQhDYnpiDZBfMdk3WNKRb4dWUGppVwEM 3QkA== X-Gm-Message-State: AOAM530Y+PJLoE/9mc6AzFxhdvAWMgaEXPiWWhUyqtrDcXm7fJ8TIe9L QTXMzpkUjUE/ydv/zQsEfYnkYQ== X-Google-Smtp-Source: ABdhPJxwNDyBJwYij2uBaeo20myfhyZEJi/8ui6ywvr6y8tyrE6/82bqHD4mZ/G5y/YfBxOZbPzsJw== X-Received: by 2002:a37:a890:: with SMTP id r138mr6153843qke.164.1607561024825; Wed, 09 Dec 2020 16:43:44 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:44 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 4/8] mm cma: rename PF_MEMALLOC_NOCMA to PF_MEMALLOC_PIN Date: Wed, 9 Dec 2020 19:43:31 -0500 Message-Id: <20201210004335.64634-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PF_MEMALLOC_NOCMA is used ot guarantee that the allocator will not return pages that might belong to CMA region. This is currently used for long term gup to make sure that such pins are not going to be done on any CMA pages. When PF_MEMALLOC_NOCMA has been introduced we haven't realized that it is focusing on CMA pages too much and that there is larger class of pages that need the same treatment. MOVABLE zone cannot contain any long term pins as well so it makes sense to reuse and redefine this flag for that usecase as well. Rename the flag to PF_MEMALLOC_PIN which defines an allocation context which can only get pages suitable for long-term pins. Also re-name: memalloc_nocma_save()/memalloc_nocma_restore to memalloc_pin_save()/memalloc_pin_restore() and make the new functions common. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- include/linux/sched.h | 2 +- include/linux/sched/mm.h | 21 +++++---------------- mm/gup.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/page_alloc.c | 4 ++-- 5 files changed, 12 insertions(+), 23 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 76cd21fa5501..5c4bd5e1cbd8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1548,7 +1548,7 @@ extern struct pid *cad_pid; #define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */ #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ -#define PF_MEMALLOC_NOCMA 0x10000000 /* All allocation request will have _GFP_MOVABLE cleared */ +#define PF_MEMALLOC_PIN 0x10000000 /* All allocation request will have _GFP_MOVABLE cleared */ #define PF_FREEZER_SKIP 0x40000000 /* Freezer should not count it as freezable */ #define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index d5ece7a9a403..a4b5da13d2c6 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -254,29 +254,18 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC) | flags; } -#ifdef CONFIG_CMA -static inline unsigned int memalloc_nocma_save(void) +static inline unsigned int memalloc_pin_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_NOCMA; + unsigned int flags = current->flags & PF_MEMALLOC_PIN; - current->flags |= PF_MEMALLOC_NOCMA; + current->flags |= PF_MEMALLOC_PIN; return flags; } -static inline void memalloc_nocma_restore(unsigned int flags) +static inline void memalloc_pin_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_NOCMA) | flags; + current->flags = (current->flags & ~PF_MEMALLOC_PIN) | flags; } -#else -static inline unsigned int memalloc_nocma_save(void) -{ - return 0; -} - -static inline void memalloc_nocma_restore(unsigned int flags) -{ -} -#endif #ifdef CONFIG_MEMCG DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); diff --git a/mm/gup.c b/mm/gup.c index 0e2de888a8b0..0eb8a85fb704 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1726,7 +1726,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (!vmas_tmp) return -ENOMEM; } - flags = memalloc_nocma_save(); + flags = memalloc_pin_save(); } rc = __get_user_pages_locked(mm, start, nr_pages, pages, @@ -1749,7 +1749,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, rc = check_and_migrate_cma_pages(mm, start, rc, pages, vmas_tmp, gup_flags); out: - memalloc_nocma_restore(flags); + memalloc_pin_restore(flags); } if (vmas_tmp != vmas) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 37f15c3c24dc..e797b41998ec 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1033,10 +1033,10 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) { struct page *page; - bool nocma = !!(current->flags & PF_MEMALLOC_NOCMA); + bool pin = !!(current->flags & PF_MEMALLOC_PIN); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (nocma && is_migrate_cma_page(page)) + if (pin && is_migrate_cma_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index eaa227a479e4..2dea5600f308 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3772,8 +3772,8 @@ static inline unsigned int current_alloc_flags(gfp_t gfp_mask, #ifdef CONFIG_CMA unsigned int pflags = current->flags; - if (!(pflags & PF_MEMALLOC_NOCMA) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (!(pflags & PF_MEMALLOC_PIN) && + gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif From patchwork Thu Dec 10 00:43:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BFB4C433FE for ; Thu, 10 Dec 2020 00:43:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DED6122D5B for ; Thu, 10 Dec 2020 00:43:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DED6122D5B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 528966B0070; Wed, 9 Dec 2020 19:43:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F07F6B0071; Wed, 9 Dec 2020 19:43:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 215176B0072; Wed, 9 Dec 2020 19:43:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0005.hostedemail.com [216.40.44.5]) by kanga.kvack.org (Postfix) with ESMTP id F0A4C6B0070 for ; Wed, 9 Dec 2020 19:43:47 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C24828249980 for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) X-FDA: 77575524894.19.men92_0f01d21273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 9DCD51AD31E for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) X-HE-Tag: men92_0f01d21273f4 X-Filterd-Recvd-Size: 7266 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id y18so3284208qki.11 for ; Wed, 09 Dec 2020 16:43:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=sUpjC2JGVFFWHT2eOX4v5Pd/VbxmqCdGzRg2mz4PxH0=; b=cZikR+BeHtWn6M7s+IlCMr3/UuAQ5Ar0APOR0SvjzWfOTd5DE7tvyvarLLSgwkqSis XbrAoIWDR3Fac/+0JXDiXLCy/u95KDigXA3b1hdYsBMGjeokv5Qyw/h9fa+CoWJKfG43 WnF5pg8pyyJzwJLzfwD2HMuKz5i0JvcZnMhyDndY81imB9SVwQR5xQyN02diPodndXsY S/CXCR9ezkPpyFFpMRxlr1DMUKt2qrw1tM4Z0lAtiNR8lLorsKDZF3zx94i2YJtL+LXm LAkbOfFiz5A/XiffhzSLpvBTddVDN+cv2HALO5EazslsZiJ/W3+omgm76d+odBso4xKN J2AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sUpjC2JGVFFWHT2eOX4v5Pd/VbxmqCdGzRg2mz4PxH0=; b=c9IrzhaGmGh0KgDCfVwhm3BKJeEPRtOPzp6KBOrDhpiKWaUqhmnAESGtb+qQZx3oQ6 pGj9+QkEMIqzFtDoA7yxKe4X6P4tPJye1C8Zz0XPmFRSY/3dafVuvyvxTCfMtwElsT9I DQDnrkvvBEB6lKgN8XLP6ffsYORePAuIpHhg9b9IrbKIA4jTFDalBSIea3koILoeQpNT tWEdV4HXefz6Qx5ikt95oH45ge1wje+/3DL2H+lFoTeao91z1oWEr8VcDUSGwj0U1hvb liYAOO9LgZxX/1hJovslcMpB+3UTWz+gbUduzpKyS+fKqHT8I6ozKygo18A9DhnREk3I UlmQ== X-Gm-Message-State: AOAM531q/bCTgFua/TTYuTf5mYLLDsl9lOm4Mm6hqir3z/YB+Bses9xg jAlJjknpBJA3KHYjY1CyZL8d1w== X-Google-Smtp-Source: ABdhPJxpG6COXfUw6XI/ibVZKCss5u6FLno1P2o5X8L0HXLNfwnptSeK3jVyPLbr4+90apRx6fDu0A== X-Received: by 2002:a37:4d58:: with SMTP id a85mr6051943qkb.324.1607561026529; Wed, 09 Dec 2020 16:43:46 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:45 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 5/8] mm: apply per-task gfp constraints in fast path Date: Wed, 9 Dec 2020 19:43:32 -0500 Message-Id: <20201210004335.64634-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Function current_gfp_context() is called after fast path. However, soon we will add more constraints which will also limit zones based on context. Move this call into fast path, and apply the correct constraints for all allocations. Also update .reclaim_idx based on value returned by current_gfp_context() because it soon will modify the allowed zones. Note: With this patch we will do one extra current->flags load during fast path, but we already load current->flags in fast-path: __alloc_pages_nodemask() prepare_alloc_pages() current_alloc_flags(gfp_mask, *alloc_flags); Later, when we add the zone constrain logic to current_gfp_context() we will be able to remove current->flags load from current_alloc_flags, and therefore return fast-path to the current performance level. Suggested-by: Michal Hocko Signed-off-by: Pavel Tatashin --- mm/page_alloc.c | 15 ++++++++------- mm/vmscan.c | 10 ++++++---- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2dea5600f308..24c99b3b12af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4932,6 +4932,13 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, } gfp_mask &= gfp_allowed_mask; + /* + * Apply scoped allocation constraints. This is mainly about GFP_NOFS + * resp. GFP_NOIO which has to be inherited for all allocation requests + * from a particular context which has been marked by + * memalloc_no{fs,io}_{save,restore}. + */ + gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags)) return NULL; @@ -4947,13 +4954,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, if (likely(page)) goto out; - /* - * Apply scoped allocation constraints. This is mainly about GFP_NOFS - * resp. GFP_NOIO which has to be inherited for all allocation requests - * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. - */ - alloc_mask = current_gfp_context(gfp_mask); + alloc_mask = gfp_mask; ac.spread_dirty_pages = false; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 7b4e31eac2cf..f51581e33fe6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3233,11 +3233,12 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist, unsigned long try_to_free_pages(struct zonelist *zonelist, int order, gfp_t gfp_mask, nodemask_t *nodemask) { + gfp_t current_gfp_mask = current_gfp_context(gfp_mask); unsigned long nr_reclaimed; struct scan_control sc = { .nr_to_reclaim = SWAP_CLUSTER_MAX, - .gfp_mask = current_gfp_context(gfp_mask), - .reclaim_idx = gfp_zone(gfp_mask), + .gfp_mask = current_gfp_mask, + .reclaim_idx = gfp_zone(current_gfp_mask), .order = order, .nodemask = nodemask, .priority = DEF_PRIORITY, @@ -4157,17 +4158,18 @@ static int __node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned in { /* Minimum pages needed in order to stay on node */ const unsigned long nr_pages = 1 << order; + gfp_t current_gfp_mask = current_gfp_context(gfp_mask); struct task_struct *p = current; unsigned int noreclaim_flag; struct scan_control sc = { .nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX), - .gfp_mask = current_gfp_context(gfp_mask), + .gfp_mask = current_gfp_mask, .order = order, .priority = NODE_RECLAIM_PRIORITY, .may_writepage = !!(node_reclaim_mode & RECLAIM_WRITE), .may_unmap = !!(node_reclaim_mode & RECLAIM_UNMAP), .may_swap = 1, - .reclaim_idx = gfp_zone(gfp_mask), + .reclaim_idx = gfp_zone(current_gfp_mask), }; trace_mm_vmscan_node_reclaim_begin(pgdat->node_id, order, From patchwork Thu Dec 10 00:43:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18096C4361B for ; Thu, 10 Dec 2020 00:43:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B86DC22D5B for ; Thu, 10 Dec 2020 00:43:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B86DC22D5B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F11496B0071; Wed, 9 Dec 2020 19:43:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E487A6B0072; Wed, 9 Dec 2020 19:43:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEC5A6B0073; Wed, 9 Dec 2020 19:43:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id B34C66B0071 for ; Wed, 9 Dec 2020 19:43:49 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 88782362B for ; Thu, 10 Dec 2020 00:43:49 +0000 (UTC) X-FDA: 77575524978.15.bee16_280cd29273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 69C581814B0C8 for ; Thu, 10 Dec 2020 00:43:49 +0000 (UTC) X-HE-Tag: bee16_280cd29273f4 X-Filterd-Recvd-Size: 8628 Received: from mail-qt1-f193.google.com (mail-qt1-f193.google.com [209.85.160.193]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:48 +0000 (UTC) Received: by mail-qt1-f193.google.com with SMTP id f14so2485466qto.12 for ; Wed, 09 Dec 2020 16:43:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=HGSArqOjWuHQduh4/5V9y89IaA5+sB598z4JFn69OP8=; b=o7oBMoPEURQxueAB/DYaG9hgqsci/S81PY/g0tU9JQ7uHPBs1l1TdiiPZ7lZWdo1jx MfMTa7UrHefkYeZsixowZ3N+3xEaq8XQn/CH+wP8cVSaa0HDusbRMavvOBGfSqKbhNEA Dk/KKqEIMSk1wyvBafZw/G3JeIq+YcCvLyDgN/aUuahGm5+KuTRpy/i4knEPd+6qyWY1 Wd7iJuRXjjTEKN4dCcUDYUb9Ppqxqy5M2NOoFpThFel693jdlmSgH84spyS2C7HQ2uWv 0tiU9dFz7n8dhB57XiqQv09AXTRixjozggqmHEwZsTZ9UDro4j+8KUDTbDRsGKq1Mwdz BnSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HGSArqOjWuHQduh4/5V9y89IaA5+sB598z4JFn69OP8=; b=KSQ0NB0wIO70J7jLxXaS2Hs6fA76kyN52ti5rkaRLIFlkNtB9honGmy9sAFeKioG8A 3/W+N6yJPASc/mFqEE2A0HoNNwITanaJBOdZPwxoJr/vLs3P2l2vym7xUYvIIEYR7AFt gCEM3dzkq05CbHcKTjRuy6vDZcgTeGjytodjT+J2VpASi7VXG9doNQb7DDhpQj+tcHco 4kk68JvJgnfNWz/CJBpWyNb5bjri6MLB2dR/7aeql55ypZSny/2RY4NOw+xkrgglr0ym 0ii25hAHZZaUknWucL1FIxI+/Tr32X7tX6bQV9Lq4RaIZ+OBO4FLWvRX15nG59h3b6B3 84qw== X-Gm-Message-State: AOAM5331ViAtQDdDrQ7uUI+oBjfWCe+qYWAji95qAfgIpYe55g2WK9sI 407/ll50uTlUQHr0+4aAd4ooPg== X-Google-Smtp-Source: ABdhPJy0FMKB/a1dDS08qr/9bboIpjUOWVwN8UwgN6wV5lFd4ss4cCgP0fB4M4q8oBJWye8XruB8Xw== X-Received: by 2002:ac8:5ccc:: with SMTP id s12mr5995034qta.364.1607561028139; Wed, 09 Dec 2020 16:43:48 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:47 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 6/8] mm: honor PF_MEMALLOC_PIN for all movable pages Date: Wed, 9 Dec 2020 19:43:33 -0500 Message-Id: <20201210004335.64634-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PF_MEMALLOC_PIN is only honored for CMA pages, extend this flag to work for any allocations from ZONE_MOVABLE by removing __GFP_MOVABLE from gfp_mask when this flag is passed in the current context. Add is_pinnable_page() to return true if page is in a pinnable page. A pinnable page is not in ZONE_MOVABLE and not of MIGRATE_CMA type. Signed-off-by: Pavel Tatashin --- include/linux/mm.h | 11 +++++++++++ include/linux/sched/mm.h | 6 +++++- mm/hugetlb.c | 2 +- mm/page_alloc.c | 19 ++++++++----------- 4 files changed, 25 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index db6ae4d3fb4e..1105e4aa9472 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1100,6 +1100,17 @@ static inline bool is_zone_device_page(const struct page *page) } #endif +static inline bool is_zone_movable_page(const struct page *page) +{ + return page_zonenum(page) == ZONE_MOVABLE; +} + +/* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */ +static inline bool is_pinnable_page(struct page *page) +{ + return !is_zone_movable_page(page) && !is_migrate_cma_page(page); +} + #ifdef CONFIG_DEV_PAGEMAP_OPS void free_devmap_managed_page(struct page *page); DECLARE_STATIC_KEY_FALSE(devmap_managed_key); diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index a4b5da13d2c6..1c7ba35c405a 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -150,12 +150,13 @@ static inline bool in_vfork(struct task_struct *tsk) * Applies per-task gfp context to the given allocation flags. * PF_MEMALLOC_NOIO implies GFP_NOIO * PF_MEMALLOC_NOFS implies GFP_NOFS + * PF_MEMALLOC_PIN implies !GFP_MOVABLE */ static inline gfp_t current_gfp_context(gfp_t flags) { unsigned int pflags = READ_ONCE(current->flags); - if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS))) { + if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_PIN))) { /* * NOIO implies both NOIO and NOFS and it is a weaker context * so always make sure it makes precedence @@ -164,6 +165,9 @@ static inline gfp_t current_gfp_context(gfp_t flags) flags &= ~(__GFP_IO | __GFP_FS); else if (pflags & PF_MEMALLOC_NOFS) flags &= ~__GFP_FS; + + if (pflags & PF_MEMALLOC_PIN) + flags &= ~__GFP_MOVABLE; } return flags; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e797b41998ec..79f2643843f3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1036,7 +1036,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) bool pin = !!(current->flags & PF_MEMALLOC_PIN); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (pin && is_migrate_cma_page(page)) + if (pin && !is_pinnable_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 24c99b3b12af..c514ad058335 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3766,16 +3766,12 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) return alloc_flags; } -static inline unsigned int current_alloc_flags(gfp_t gfp_mask, - unsigned int alloc_flags) +static inline unsigned int cma_alloc_flags(gfp_t gfp_mask, + unsigned int alloc_flags) { #ifdef CONFIG_CMA - unsigned int pflags = current->flags; - - if (!(pflags & PF_MEMALLOC_PIN) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; - #endif return alloc_flags; } @@ -4423,7 +4419,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; - alloc_flags = current_alloc_flags(gfp_mask, alloc_flags); + alloc_flags = cma_alloc_flags(gfp_mask, alloc_flags); return alloc_flags; } @@ -4725,7 +4721,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) - alloc_flags = current_alloc_flags(gfp_mask, reserve_flags); + alloc_flags = cma_alloc_flags(gfp_mask, reserve_flags); /* * Reset the nodemask and zonelist iterators if memory policies can be @@ -4894,7 +4890,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, if (should_fail_alloc_page(gfp_mask, order)) return false; - *alloc_flags = current_alloc_flags(gfp_mask, *alloc_flags); + *alloc_flags = cma_alloc_flags(gfp_mask, *alloc_flags); /* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE); @@ -4936,7 +4932,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, * Apply scoped allocation constraints. This is mainly about GFP_NOFS * resp. GFP_NOIO which has to be inherited for all allocation requests * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. + * memalloc_no{fs,io}_{save,restore}. And PF_MEMALLOC_PIN which ensures + * movable zones are not used during allocation. */ gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; From patchwork Thu Dec 10 00:43:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DFDCC4167B for ; Thu, 10 Dec 2020 00:43:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B227922D5B for ; Thu, 10 Dec 2020 00:43:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B227922D5B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AC5076B0072; Wed, 9 Dec 2020 19:43:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A4F446B0073; Wed, 9 Dec 2020 19:43:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8ED576B0074; Wed, 9 Dec 2020 19:43:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 7274C6B0072 for ; Wed, 9 Dec 2020 19:43:51 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4492A180AD807 for ; Thu, 10 Dec 2020 00:43:51 +0000 (UTC) X-FDA: 77575525062.24.kiss53_020ba32273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 289D41A4A0 for ; Thu, 10 Dec 2020 00:43:51 +0000 (UTC) X-HE-Tag: kiss53_020ba32273f4 X-Filterd-Recvd-Size: 11060 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:50 +0000 (UTC) Received: by mail-qk1-f172.google.com with SMTP id z11so3303558qkj.7 for ; Wed, 09 Dec 2020 16:43:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Qk8rVh1rXtyNOJhKlNnNxraylm8G6DtD3KrN637cCCY=; b=fMdVWCNKITv0dg2NlC0bJIuTY/rS0L/auHzJODLG1gJsRMzwg1f2vhMfk8VgTPWFb1 EGP2rvW7Ddsyis7N9GW7HF9Esuunne9NnIMmaRDqXC6mpUhPIXZ+1ciCTkNDyjdLhx+y VfNtVUaQFX50fcEsJqRUZeONp2iOgImfFJ6YSI0zItK+pg6nljFaSVEl5+VMIn5Ju/yi Ef6R/VeZqKwv8OkjcphblD65i3nnkN+eM34dHKwtHrfzYg3muPKZwZvlA4vqzv/S06Gv nUFwMjeZ3/39YoDQqyquk9o1umGf+i7jw6W4Uz7/oSy7LPLzfaRkrKxMEAWQqIQc0jTU aBmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Qk8rVh1rXtyNOJhKlNnNxraylm8G6DtD3KrN637cCCY=; b=NKRkJpz0RzcKrMXeglk8mTTboEJ9f4LBSJLPhOCuwt/EmBnbm3JfBBAhdOvvM17g9b Bbim9w80T0ONFM752FWbjfm8QfNxoIJc1fgE0rxNAzPqzUvch1JBpmM+S75sbq6EXLy2 rDG9Lv1xOIfZv6f8MI8LWIY1mfdP3TVqMQUoNz+Ji+NDDOptVHDfkQN9Pp0Z8Hc53LbA ZUKez8X6fcOBqPGYBUfFaTNyy0yRKMvh3Rr6GA9Q9IrGSJdKZ374E2Nc3XPShST6NF0C WVO3hc8V/ayictf3UcOeQuzqBpuRgRmT2SzJIwpWJY+UpxOLqmrE/0WEy5mRLpBNRXKb d/Aw== X-Gm-Message-State: AOAM530nKrhRq6gL2T4nFynLAo70bDAFHYAHSK9P1/7OhbsDvo5j/852 4XhL/MI+D/HsFW7WZcGrAR17ww== X-Google-Smtp-Source: ABdhPJy/W1k/3gnadCh/Cym4o8ZSSnpU7JcnD+shVsxXMkusZ00RCHBLY90vQHfYNck7M8GFIweDew== X-Received: by 2002:a37:aa15:: with SMTP id t21mr6023243qke.86.1607561029865; Wed, 09 Dec 2020 16:43:49 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:49 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 7/8] mm/gup: migrate pinned pages out of movable zone Date: Wed, 9 Dec 2020 19:43:34 -0500 Message-Id: <20201210004335.64634-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We should not pin pages in ZONE_MOVABLE. Currently, we do not pin only movable CMA pages. Generalize the function that migrates CMA pages to migrate all movable pages. Use is_pinnable_page() to check which pages need to be migrated Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- include/linux/migrate.h | 1 + include/linux/mmzone.h | 11 ++++-- include/trace/events/migrate.h | 3 +- mm/gup.c | 65 ++++++++++++++-------------------- 4 files changed, 37 insertions(+), 43 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 0f8d1583fa8e..00bab23d1ee5 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -27,6 +27,7 @@ enum migrate_reason { MR_MEMPOLICY_MBIND, MR_NUMA_MISPLACED, MR_CONTIG_RANGE, + MR_LONGTERM_PIN, MR_TYPES }; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index fb3bf696c05e..87a7321b4252 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -405,9 +405,14 @@ enum zone_type { * likely to succeed, and to locally limit unmovable allocations - e.g., * to increase the number of THP/huge pages. Notable special cases are: * - * 1. Pinned pages: (long-term) pinning of movable pages might - * essentially turn such pages unmovable. Memory offlining might - * retry a long time. + * 1. Pinned pages: (long-term) pinning of movable pages is avoided + * when pages are pinned and faulted, but it is still possible that + * address space already has pages in ZONE_MOVABLE at the time when + * pages are pinned (i.e. user has touches that memory before + * pinning). In such case we try to migrate them to a different zone, + * but if migration fails the pages can still end-up pinned in + * ZONE_MOVABLE. In such case, memory offlining might retry a long + * time and will only succeed once user application unpins pages. * 2. memblock allocations: kernelcore/movablecore setups might create * situations where ZONE_MOVABLE contains unmovable allocations * after boot. Memory offlining and allocations fail early. diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index 4d434398d64d..363b54ce104c 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -20,7 +20,8 @@ EM( MR_SYSCALL, "syscall_or_cpuset") \ EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ EM( MR_NUMA_MISPLACED, "numa_misplaced") \ - EMe(MR_CONTIG_RANGE, "contig_range") + EM( MR_CONTIG_RANGE, "contig_range") \ + EMe(MR_LONGTERM_PIN, "longterm_pin") /* * First define the enums in the above macros to be exported to userspace diff --git a/mm/gup.c b/mm/gup.c index 0eb8a85fb704..e575237d4c67 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -88,11 +88,12 @@ static __maybe_unused struct page *try_grab_compound_head(struct page *page, int orig_refs = refs; /* - * Can't do FOLL_LONGTERM + FOLL_PIN with CMA in the gup fast - * path, so fail and let the caller fall back to the slow path. + * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a + * right zone, so fail and let the caller fall back to the slow + * path. */ - if (unlikely(flags & FOLL_LONGTERM) && - is_migrate_cma_page(page)) + if (unlikely((flags & FOLL_LONGTERM) && + !is_pinnable_page(page))) return NULL; /* @@ -1593,19 +1594,18 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) } #endif -#ifdef CONFIG_CMA -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) +static long check_and_migrate_movable_pages(struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int gup_flags) { unsigned long i; unsigned long step; bool drain_allow = true; bool migrate_allow = true; - LIST_HEAD(cma_page_list); + LIST_HEAD(movable_page_list); long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, @@ -1623,13 +1623,12 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, */ step = compound_nr(head) - (pages[i] - head); /* - * If we get a page from the CMA zone, since we are going to - * be pinning these entries, we might as well move them out - * of the CMA zone if possible. + * If we get a movable page, since we are going to be pinning + * these entries, try to move them out if possible. */ - if (is_migrate_cma_page(head)) { + if (!is_pinnable_page(head)) { if (PageHuge(head)) - isolate_huge_page(head, &cma_page_list); + isolate_huge_page(head, &movable_page_list); else { if (!PageLRU(head) && drain_allow) { lru_add_drain_all(); @@ -1637,7 +1636,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, } if (!isolate_lru_page(head)) { - list_add_tail(&head->lru, &cma_page_list); + list_add_tail(&head->lru, &movable_page_list); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), @@ -1649,7 +1648,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, i += step; } - if (!list_empty(&cma_page_list)) { + if (!list_empty(&movable_page_list)) { /* * drop the above get_user_pages reference. */ @@ -1659,7 +1658,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, for (i = 0; i < nr_pages; i++) put_page(pages[i]); - if (migrate_pages(&cma_page_list, alloc_migration_target, NULL, + if (migrate_pages(&movable_page_list, alloc_migration_target, NULL, (unsigned long)&mtc, MIGRATE_SYNC, MR_CONTIG_RANGE)) { /* * some of the pages failed migration. Do get_user_pages @@ -1667,17 +1666,16 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, */ migrate_allow = false; - if (!list_empty(&cma_page_list)) - putback_movable_pages(&cma_page_list); + if (!list_empty(&movable_page_list)) + putback_movable_pages(&movable_page_list); } /* * We did migrate all the pages, Try to get the page references - * again migrating any new CMA pages which we failed to isolate - * earlier. + * again migrating any pages which we failed to isolate earlier. */ ret = __get_user_pages_locked(mm, start, nr_pages, - pages, vmas, NULL, - gup_flags); + pages, vmas, NULL, + gup_flags); if ((ret > 0) && migrate_allow) { nr_pages = ret; @@ -1688,17 +1686,6 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, return ret; } -#else -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) -{ - return nr_pages; -} -#endif /* CONFIG_CMA */ /* * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which @@ -1746,8 +1733,8 @@ static long __gup_longterm_locked(struct mm_struct *mm, goto out; } - rc = check_and_migrate_cma_pages(mm, start, rc, pages, - vmas_tmp, gup_flags); + rc = check_and_migrate_movable_pages(mm, start, rc, pages, + vmas_tmp, gup_flags); out: memalloc_pin_restore(flags); } From patchwork Thu Dec 10 00:43:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11962973 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB88DC4361B for ; Thu, 10 Dec 2020 00:43:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9823322D5B for ; Thu, 10 Dec 2020 00:43:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9823322D5B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 327066B0073; Wed, 9 Dec 2020 19:43:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 286016B0074; Wed, 9 Dec 2020 19:43:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 126E66B0075; Wed, 9 Dec 2020 19:43:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id E98B86B0073 for ; Wed, 9 Dec 2020 19:43:52 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BAD62180AD807 for ; Thu, 10 Dec 2020 00:43:52 +0000 (UTC) X-FDA: 77575525104.16.care45_610ce49273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 9BC2D100E690B for ; Thu, 10 Dec 2020 00:43:52 +0000 (UTC) X-HE-Tag: care45_610ce49273f4 X-Filterd-Recvd-Size: 4717 Received: from mail-qt1-f194.google.com (mail-qt1-f194.google.com [209.85.160.194]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:52 +0000 (UTC) Received: by mail-qt1-f194.google.com with SMTP id y15so2491745qtv.5 for ; Wed, 09 Dec 2020 16:43:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UPZkOOUAvu9r4fFw8TJVvFyTq3W2R2uLYiLCZknh/L8=; b=gmG1PpUsZr319a18DQnsi5mdG9y7b4t3RB3P2dechoWxY0Pqz2thym4xN5wIfRn1Q4 Zw9W5/aK+5TRDcVJVOMsS3Z7f6GudDVH/eYCeIsLxu2o7Nz/4p5Jq2Xwp9inOrxmnVbj TZ5uXsD53+Jo8U+FDbuIB2tFD7HRqp6p1YLX1q9RSiMRN5Ve4xVGdG1V1CVxDuWmvHAD DKVaKsSrifnIeiLiCFQ/h9cVe692qD+IpTCv1Op5j3vHfvwaFgL79HDGLo7U9H3XQBGe Jyv8Gp8SjDOE/7SKS3q5y5zA06EMAGJWPiX/hVHPL34ak1030j7gSBQyY5EfC3ZrlbfP 8M2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UPZkOOUAvu9r4fFw8TJVvFyTq3W2R2uLYiLCZknh/L8=; b=Dd/osnx8IrSdDxRdSWE+QFCcWUrtbGTywjn+2tIUCyOUzrPUJ1tv2B6S2Lqdxx9TdV 86pY/ynF5QJwtVkr/Y/dlEhC6rIShWM+bBcCyM7ntbxa2jpjD1qUeU6hcTZ38+t5RP2O ichLmoMseeDQld6PM7K/ujP3EF0+k3iXVsGUSfIwVDD8B25Il9QHKKI5zP8tNHkcLYpW LN8e/mIhcCTTriKru56ebpL4xMTUDptXq7DqJfNTs5NtWm50fmHGAgfCAMqig5Du6eBh S19xNnPSDRANHEE8V/qu6qZwMhkDHH7gYcbSpTcirmuiMS5YEgH/5K27q5N5mAnnQq/t UpqQ== X-Gm-Message-State: AOAM531zEiRmWzHLWBm9OzXuQhQWbxk1+M/U52o0KWmP96PHM59tylUQ at+mm7WynpqSMT20akOxKr5b/Q== X-Google-Smtp-Source: ABdhPJyi0mwWQZy/L/JfYf+2tJkJILrpi7wkAfm++gQ2t5NF/5F1UStDpyKxp6YmK/amRq2h1MKphA== X-Received: by 2002:ac8:7244:: with SMTP id l4mr5870164qtp.203.1607561031637; Wed, 09 Dec 2020 16:43:51 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:51 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 8/8] memory-hotplug.rst: add a note about ZONE_MOVABLE and page pinning Date: Wed, 9 Dec 2020 19:43:35 -0500 Message-Id: <20201210004335.64634-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Document the special handling of page pinning when ZONE_MOVABLE present. Signed-off-by: Pavel Tatashin Suggested-by: David Hildenbrand --- Documentation/admin-guide/mm/memory-hotplug.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst index 5c4432c96c4b..c6618f99f765 100644 --- a/Documentation/admin-guide/mm/memory-hotplug.rst +++ b/Documentation/admin-guide/mm/memory-hotplug.rst @@ -357,6 +357,15 @@ creates ZONE_MOVABLE as following. Unfortunately, there is no information to show which memory block belongs to ZONE_MOVABLE. This is TBD. +.. note:: + Techniques that rely on long-term pinnings of memory (especially, RDMA and + vfio) are fundamentally problematic with ZONE_MOVABLE and, therefore, memory + hot remove. Pinned pages cannot reside on ZONE_MOVABLE, to guarantee that + memory can still get hot removed - be aware that pinning can fail even if + there is plenty of free memory in ZONE_MOVABLE. In addition, using + ZONE_MOVABLE might make page pinning more expensive, because pages have to be + migrated off that zone first. + .. _memory_hotplug_how_to_offline_memory: How to offline memory