From patchwork Wed Dec 2 05:23:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11944729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31EA5C64E7C for ; Wed, 2 Dec 2020 05:23:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AC64420872 for ; Wed, 2 Dec 2020 05:23:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AC64420872 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3B07C8D0006; Wed, 2 Dec 2020 00:23:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 388E88D0002; Wed, 2 Dec 2020 00:23:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 279988D0006; Wed, 2 Dec 2020 00:23:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 0FE1A8D0002 for ; Wed, 2 Dec 2020 00:23:38 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id BD59A1F08 for ; Wed, 2 Dec 2020 05:23:37 +0000 (UTC) X-FDA: 77547199674.02.spark01_4c0a14d273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 862C410097AA0 for ; Wed, 2 Dec 2020 05:23:37 +0000 (UTC) X-HE-Tag: spark01_4c0a14d273b0 X-Filterd-Recvd-Size: 4349 Received: from mail-io1-f65.google.com (mail-io1-f65.google.com [209.85.166.65]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 05:23:36 +0000 (UTC) Received: by mail-io1-f65.google.com with SMTP id n14so646234iom.10 for ; Tue, 01 Dec 2020 21:23:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=trxUS4hTsq7Vgn0OaEdGU2Oamf0jSdRBNQtCZpSo+v4=; b=dpF7Ja/wYE+ogPzGSNyV+WbT29ucR4KcbNWmSptBcV9kXTAN4dVUUACorYiyR+EX33 6J9k9N+pATTjMFVtECRNH8htHRIQFTIH7WA3hKIS+jCvV59iC6M+5cj0c6YPGnaiL0XL 7OjRq6g4vcsQFkJjtku5SlbDIMF08JSxkvPagRMv6EpnGPuN6wBWjt/8YtwzGWscTlSn HzW88rVcNP54tJxXahg8pJ2ujf/Htag82Qohw0RkcLxKCNMCc1jBLC6Tvg7LDnk1P3CL piWlohaWnneDLnt8R1fXI0iNc0N2RRoxQW5DGW055OZ/P540ruDVY0+6ehybpETzlXmV clVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=trxUS4hTsq7Vgn0OaEdGU2Oamf0jSdRBNQtCZpSo+v4=; b=mIJ+/5LqZQPG+MgDP32amenlBtQjILngq0Ah23BRJ0RyLF2j3DHpCvfK5d2wBSLYbs ULV6i6pXvT6PLnaD7LyDU76XMSYgyTAtFPwMP0usj8Fsokp6pb7TsG6mieggWnzSuoNl cv0NQz1xlULrkyXTXXlqNpseNBCZS6AX2yx7QXBRAvKbh93CZiIkVqeNz42NtZWgzN6s oqEy8qEudAFHXwk7iuFfz9JyFQv+/9vla18UIIyq4xrbRTEMhv+ZNDyDvgpdKpGAs9y4 Vqr5MxO9D3zQkiBmS+iRVScJksmBRyBjXo/YTEUSkNODpCl/l+9ZPnqVm+kRb+y9nj4f 5hhA== X-Gm-Message-State: AOAM533SBYEP0h9LYT/CLllN8DeN8WCnJj0+KxQtkQV/pDf2bT0GP6fc 00v181fvb/0/V/1/C+WBUCB+/w== X-Google-Smtp-Source: ABdhPJyiNldo5SwgJL3dxm7RfCbXMwfB+m64ZQwFTAvl7q0/UDnGvqjF07LmaClRp7jSh3OZF4a7rg== X-Received: by 2002:a6b:760e:: with SMTP id g14mr628341iom.136.1606886616474; Tue, 01 Dec 2020 21:23:36 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m19sm295938ila.81.2020.12.01.21.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 21:23:35 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com Subject: [PATCH 1/6] mm/gup: perform check_dax_vmas only when FS_DAX is enabled Date: Wed, 2 Dec 2020 00:23:25 -0500 Message-Id: <20201202052330.474592-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201202052330.474592-1-pasha.tatashin@soleen.com> References: <20201202052330.474592-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is no need to check_dax_vmas() and run through the npage loop of pinned pages if FS_DAX is not enabled. Add a stub check_dax_vmas() function for no-FS_DAX case. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- mm/gup.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 98eb8e6d2609..cdb8b9eeb016 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1568,6 +1568,7 @@ struct page *get_dump_page(unsigned long addr) #endif /* CONFIG_ELF_CORE */ #if defined(CONFIG_FS_DAX) || defined (CONFIG_CMA) +#ifdef CONFIG_FS_DAX static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) { long i; @@ -1586,6 +1587,12 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) } return false; } +#else +static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) +{ + return false; +} +#endif #ifdef CONFIG_CMA static long check_and_migrate_cma_pages(struct mm_struct *mm, From patchwork Wed Dec 2 05:23:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11944735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43F26C8300F for ; Wed, 2 Dec 2020 05:23:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 884A020872 for ; Wed, 2 Dec 2020 05:23:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 884A020872 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0C9648D0007; Wed, 2 Dec 2020 00:23:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07E958D0002; Wed, 2 Dec 2020 00:23:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAC0F8D0007; Wed, 2 Dec 2020 00:23:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id D45208D0002 for ; Wed, 2 Dec 2020 00:23:39 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9E259181AEF1A for ; Wed, 2 Dec 2020 05:23:39 +0000 (UTC) X-FDA: 77547199758.02.grain14_170d9d9273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 7EC1810097AA0 for ; Wed, 2 Dec 2020 05:23:39 +0000 (UTC) X-HE-Tag: grain14_170d9d9273b0 X-Filterd-Recvd-Size: 4124 Received: from mail-il1-f194.google.com (mail-il1-f194.google.com [209.85.166.194]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 05:23:39 +0000 (UTC) Received: by mail-il1-f194.google.com with SMTP id q1so442264ilt.6 for ; Tue, 01 Dec 2020 21:23:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=CpKoGHvxtKgwzmlV3ZPMeHJLr+EFBiNKpVwHZEuJXQ0=; b=mXaAEbMD2mJMb6+nQYBLdGk8VJKAskLxfLQhyPDM8oSqX6bfHvGVqReAC/ngwpbpm5 ICrQx0+ZhlDZLwTh8Fz4/e7TSkX9V+VBNYiUIw+0iSFrQny6wAwoHHAcLipd8g+J4lT/ 2435aDyExhY+A1YuhwxNhEWGa5qAHJiDrXM0PPEJ8dFShwp3QFl5he719duEUpTknuKS lsaZ3STSSMp791cizh8wk2tSbRcS3Q2GLYsY6j+9yVtw2QlRdmr776c1+nwQC8QOflJ6 9lLq/b4V4fzMJ3fZgfOENONWF9D1c5ysv46vjbO6r01+k55btqfrT4W5V0nltTqqDTIK v+xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CpKoGHvxtKgwzmlV3ZPMeHJLr+EFBiNKpVwHZEuJXQ0=; b=TBCOsOTLSOedn01IoqDGi1DZuEXMa3aeTx+8jshu9Tz5joTBVhZ5ryECI6ooQbfIY4 QS106QM+bigwbUaV0yvADU+eyjt0oB5QOIqgYObvuUTnQ32fW10dM0ch8sfzILWI31xP ixCEikzKaAZEsKvsmbOpDOMZK2MlOsGtt2U/4TNb/FVp8p0BEZWS0TZJxtlG2nntHiGA o3txhM5Jm0m70sg9aOlmwyYabsAFzmNWH6a9CI2tYneglf5XUpSvx4vJEIRzHngc3Vk7 ZxUmo60/A130hqaPTm46FNAwPIYFiSkVsu9XutARPmMfaH+ozPmDbto4VT9ADZFAAVQH G1Hg== X-Gm-Message-State: AOAM5300VeLVMlLz+G775fvZHXIv3LQb0A/fRWO5o8YYI8wEpHNVlCQs BzkWpn0d5wzv9t2ZNZI3f1fkKw== X-Google-Smtp-Source: ABdhPJz3xU6WUFCEx3XYhA8SLFw5JFA/5ztJvoX+p2Xi3reK3z9aXuJQcQDf3c5HDzC/jXnwJPc6uA== X-Received: by 2002:a05:6e02:1948:: with SMTP id x8mr941650ilu.225.1606886618551; Tue, 01 Dec 2020 21:23:38 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m19sm295938ila.81.2020.12.01.21.23.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 21:23:37 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com Subject: [PATCH 2/6] mm/gup: don't pin migrated cma pages in movable zone Date: Wed, 2 Dec 2020 00:23:26 -0500 Message-Id: <20201202052330.474592-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201202052330.474592-1-pasha.tatashin@soleen.com> References: <20201202052330.474592-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order not to fragment CMA the pinned pages are migrated. However, they are migrated to ZONE_MOVABLE, which also should not have pinned pages. Remove __GFP_MOVABLE, so pages can be migrated to zones where pinning is allowed. Signed-off-by: Pavel Tatashin Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Acked-by: Michal Hocko --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index cdb8b9eeb016..3a76c005a3e2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1610,7 +1610,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, - .gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_NOWARN, + .gfp_mask = GFP_USER | __GFP_NOWARN, }; check_again: From patchwork Wed Dec 2 05:23:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11944731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50577C83013 for ; Wed, 2 Dec 2020 05:23:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C6C962065C for ; Wed, 2 Dec 2020 05:23:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6C962065C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2548D8D0008; Wed, 2 Dec 2020 00:23:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B1E08D0002; Wed, 2 Dec 2020 00:23:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A40C8D0008; Wed, 2 Dec 2020 00:23:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id E9B508D0002 for ; Wed, 2 Dec 2020 00:23:41 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BA986181AEF1A for ; Wed, 2 Dec 2020 05:23:41 +0000 (UTC) X-FDA: 77547199842.01.look84_1a0c2a6273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 9E9B310046463 for ; Wed, 2 Dec 2020 05:23:41 +0000 (UTC) X-HE-Tag: look84_1a0c2a6273b0 X-Filterd-Recvd-Size: 4682 Received: from mail-il1-f195.google.com (mail-il1-f195.google.com [209.85.166.195]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 05:23:41 +0000 (UTC) Received: by mail-il1-f195.google.com with SMTP id z14so429768ilm.10 for ; Tue, 01 Dec 2020 21:23:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=vWbivB19k0h8/AlP1XM1Nmc7af9oF18AMBG0uJjqBZg=; b=g96m6GkPZHfEbSkiqZwo4JVs7jb4V5m6B/E80Zsyu24GkvrsPZp7AWEJhI//1ghTWN 5NLAv8o3wcP/KZpZQmNSbfpDFqZdUUVJjfkMerBm1X/+ACWSdy3aELusEZYv6b7yhiB7 xXf+WbsGmnygy23Xtkg0rrAJSR4ApXWWUWpH9P6mRfqmGZGMmqJZwL4rI1Dpc5Btmo2j 3QTXixFypX+bc0zrN6L05FqlHYHxxx57GyOhX0pEAvYLw28iOO7qbYYIOeFby7eDuglc oeeGldegA6vLLJ/FQvIVi3vDW1lkYcV72cVcRusrIxeiCzIDe90yPtpiRDj8k6F0UaS6 Bxag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vWbivB19k0h8/AlP1XM1Nmc7af9oF18AMBG0uJjqBZg=; b=FzjK57Tw8hRtEufMvhJFBfD+8kEekOhJ/clPUR3LEc0/gTAC+zxaua8Ehcg5sZjV3A bocWDUPckQ4GS0dSQO+AyAYin7sHLfiF4whdZvOe607gEETVmnZqD6+mU7Nku4vn+yXf GQEevulEf5eBQL9ijbGbQ56cJ8Wie6X7S7Q0mAfRsDi64gi8kPTwJeEq6Wp3yhRjKnpw szCEE7cwXXqWbJB0RQ67GuYF0vSG2WnqyVl0HyGPEdkQBf3+lZUj+ZiIn0bQiQbVbX7B hWsjCUuFwx1onjI7U6fLTcQqBI/mV6zcs4eQTgGHguD0f2D5xgSjY8umVAwwmzvwRQ4I k/KA== X-Gm-Message-State: AOAM530zs92ZzKOYD/Lg7utTSXc4BN+NLgiusWdOXetuu8i+UleCpDMX cr9iVHOrwbN4QWrkSEAq+TUH5w== X-Google-Smtp-Source: ABdhPJxgs+HxskX0uhoP9WOG8kbRPPRX02QP9Wypzqg6Mj5iS3wfXYkcKGeLH1LvMeVA4mW5Ql4yNA== X-Received: by 2002:a05:6e02:80b:: with SMTP id u11mr925843ilm.43.1606886620616; Tue, 01 Dec 2020 21:23:40 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m19sm295938ila.81.2020.12.01.21.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 21:23:40 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com Subject: [PATCH 3/6] mm/gup: make __gup_longterm_locked common Date: Wed, 2 Dec 2020 00:23:27 -0500 Message-Id: <20201202052330.474592-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201202052330.474592-1-pasha.tatashin@soleen.com> References: <20201202052330.474592-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __gup_longterm_locked() has CMA || FS_DAX version and a common stub version. In the preparation of prohibiting longterm pinning of pages from movable zone make the CMA || FS_DAX version common, and delete the stub version. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- mm/gup.c | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 3a76c005a3e2..0e2de888a8b0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1567,7 +1567,6 @@ struct page *get_dump_page(unsigned long addr) } #endif /* CONFIG_ELF_CORE */ -#if defined(CONFIG_FS_DAX) || defined (CONFIG_CMA) #ifdef CONFIG_FS_DAX static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) { @@ -1757,18 +1756,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, kfree(vmas_tmp); return rc; } -#else /* !CONFIG_FS_DAX && !CONFIG_CMA */ -static __always_inline long __gup_longterm_locked(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int flags) -{ - return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, - NULL, flags); -} -#endif /* CONFIG_FS_DAX || CONFIG_CMA */ static bool is_valid_gup_flags(unsigned int gup_flags) { From patchwork Wed Dec 2 05:23:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11944733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A483FC64E8A for ; Wed, 2 Dec 2020 05:23:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 220B42065C for ; Wed, 2 Dec 2020 05:23:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 220B42065C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5AB178D0009; Wed, 2 Dec 2020 00:23:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 55B9B8D0002; Wed, 2 Dec 2020 00:23:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D5C68D0009; Wed, 2 Dec 2020 00:23:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id 220C48D0002 for ; Wed, 2 Dec 2020 00:23:44 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E68618249980 for ; Wed, 2 Dec 2020 05:23:43 +0000 (UTC) X-FDA: 77547199926.02.paint22_040cc2f273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id C254E10097AA0 for ; Wed, 2 Dec 2020 05:23:43 +0000 (UTC) X-HE-Tag: paint22_040cc2f273b0 X-Filterd-Recvd-Size: 8061 Received: from mail-io1-f67.google.com (mail-io1-f67.google.com [209.85.166.67]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 05:23:43 +0000 (UTC) Received: by mail-io1-f67.google.com with SMTP id d8so635943ioc.13 for ; Tue, 01 Dec 2020 21:23:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=g7jlxchzUs1d/FAinzhFIBtXsEXm4/sR0Ksg9506XeU=; b=meUVIcZ7VIwMwi/cSom/+PXH6JMZqXpsyjjZQDUQ/m2V8lTEGR/Xnc9mVxGajJjhrf jgQhs6/zs4jSQ1U5xiP9cW03Sw+roeWUp0y+nkybRZWDIYj7WV5OcREAO8t/heQxUry+ T0PqrWC9kOiss9r+kGTzNB2z5IJFAt/PldzCsqwSvxYzivAxrnwNtpxZNaeaj0+82dmv uGBAO+JvNdn6SI/xfB5Zd9QY3FyAXBOdZIx2R2M47RNH/OaL4/bBOa7eaaPfiYCjjpWq Hob4bVHEvTwZCP3+PkoobH8NM5uJEJvD4eDzobDO4QspUslLUtbdKa4Brr+Z0MMFmE4h fB2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g7jlxchzUs1d/FAinzhFIBtXsEXm4/sR0Ksg9506XeU=; b=CClg6TMdwllv0ICT3cgunL4dXUCB+Uxz+7O81tEc2PcKS6OSRq57y4+dqtLvcXBFtP 3vVIBdqtG/lpD7CGmE7A4Ik55Ift+htWsCTKcK5WTzm89nVKowg2C3TTCvUdx2mXVpMp DsUC0CkO7CfF5E4LNETGvjz9zW19rhXMc4jniY2ybzfvQuR2L75ykcZIR907WHl3ZiKt 9APrDxlxabZ81xlLpWOJURmN+GemB8DvBbq/2gApgrrzEqI1e/C9HBs1IO5B2rAz/8x9 Z+l1hJfjYUJP0wmGAU9TkX2AKSYj1hqweaKr+N7DoehR4b23oqf9Dsu+R89/rjHYJNVJ hRjQ== X-Gm-Message-State: AOAM533tjlJxY1yCpydRPKNqwKz0/jrKpwPkekOaW3mYPcEsfCt8/3k+ +AdX+ywDSCPsZvFx4YTAmpCeAA== X-Google-Smtp-Source: ABdhPJw9L/9TB6Ptmk4hotjOfiKYRmPmEApH/Fplhyapd+mf1leuXaR7xRyhfGACderI/wunhOfXZg== X-Received: by 2002:a05:6638:50c:: with SMTP id i12mr742702jar.74.1606886622709; Tue, 01 Dec 2020 21:23:42 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m19sm295938ila.81.2020.12.01.21.23.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 21:23:42 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com Subject: [PATCH 4/6] mm cma: rename PF_MEMALLOC_NOCMA to PF_MEMALLOC_NOMOVABLE Date: Wed, 2 Dec 2020 00:23:28 -0500 Message-Id: <20201202052330.474592-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201202052330.474592-1-pasha.tatashin@soleen.com> References: <20201202052330.474592-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PF_MEMALLOC_NOCMA is used for longterm pinning and has an effect of clearing _GFP_MOVABLE or prohibiting allocations from ZONE_MOVABLE. We will prohibit allocating any pages that are getting longterm pinned from ZONE_MOVABLE, and we would want to unify and re-use this flag. So, rename it to generic PF_MEMALLOC_NOMOVABLE. Also re-name: memalloc_nocma_save()/memalloc_nocma_restore to memalloc_nomovable_save()/memalloc_nomovable_restore() and make the new functions common. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- include/linux/sched.h | 2 +- include/linux/sched/mm.h | 21 +++++---------------- mm/gup.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/page_alloc.c | 4 ++-- 5 files changed, 12 insertions(+), 23 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 76cd21fa5501..f1bf05f5f5fa 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1548,7 +1548,7 @@ extern struct pid *cad_pid; #define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */ #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ -#define PF_MEMALLOC_NOCMA 0x10000000 /* All allocation request will have _GFP_MOVABLE cleared */ +#define PF_MEMALLOC_NOMOVABLE 0x10000000 /* All allocation request will have _GFP_MOVABLE cleared */ #define PF_FREEZER_SKIP 0x40000000 /* Freezer should not count it as freezable */ #define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index d5ece7a9a403..5bb9a6b69479 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -254,29 +254,18 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC) | flags; } -#ifdef CONFIG_CMA -static inline unsigned int memalloc_nocma_save(void) +static inline unsigned int memalloc_nomovable_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_NOCMA; + unsigned int flags = current->flags & PF_MEMALLOC_NOMOVABLE; - current->flags |= PF_MEMALLOC_NOCMA; + current->flags |= PF_MEMALLOC_NOMOVABLE; return flags; } -static inline void memalloc_nocma_restore(unsigned int flags) +static inline void memalloc_nomovable_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_NOCMA) | flags; + current->flags = (current->flags & ~PF_MEMALLOC_NOMOVABLE) | flags; } -#else -static inline unsigned int memalloc_nocma_save(void) -{ - return 0; -} - -static inline void memalloc_nocma_restore(unsigned int flags) -{ -} -#endif #ifdef CONFIG_MEMCG DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); diff --git a/mm/gup.c b/mm/gup.c index 0e2de888a8b0..724d8a65e1df 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1726,7 +1726,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (!vmas_tmp) return -ENOMEM; } - flags = memalloc_nocma_save(); + flags = memalloc_nomovable_save(); } rc = __get_user_pages_locked(mm, start, nr_pages, pages, @@ -1749,7 +1749,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, rc = check_and_migrate_cma_pages(mm, start, rc, pages, vmas_tmp, gup_flags); out: - memalloc_nocma_restore(flags); + memalloc_nomovable_restore(flags); } if (vmas_tmp != vmas) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 37f15c3c24dc..02213c74ed6b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1033,10 +1033,10 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) { struct page *page; - bool nocma = !!(current->flags & PF_MEMALLOC_NOCMA); + bool nomovable = !!(current->flags & PF_MEMALLOC_NOMOVABLE); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (nocma && is_migrate_cma_page(page)) + if (nomovable && is_migrate_cma_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index eaa227a479e4..611799c72da5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3772,8 +3772,8 @@ static inline unsigned int current_alloc_flags(gfp_t gfp_mask, #ifdef CONFIG_CMA unsigned int pflags = current->flags; - if (!(pflags & PF_MEMALLOC_NOCMA) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (!(pflags & PF_MEMALLOC_NOMOVABLE) && + gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif From patchwork Wed Dec 2 05:23:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11944737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E213BC64E7C for ; Wed, 2 Dec 2020 05:23:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72E4C2065C for ; Wed, 2 Dec 2020 05:23:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72E4C2065C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 55EC18D000A; Wed, 2 Dec 2020 00:23:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E9508D0002; Wed, 2 Dec 2020 00:23:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 312828D000A; Wed, 2 Dec 2020 00:23:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id 1CF318D0002 for ; Wed, 2 Dec 2020 00:23:46 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E236D3636 for ; Wed, 2 Dec 2020 05:23:45 +0000 (UTC) X-FDA: 77547200010.21.mine09_31136cc273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id C4258180442C3 for ; Wed, 2 Dec 2020 05:23:45 +0000 (UTC) X-HE-Tag: mine09_31136cc273b0 X-Filterd-Recvd-Size: 6763 Received: from mail-il1-f195.google.com (mail-il1-f195.google.com [209.85.166.195]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 05:23:45 +0000 (UTC) Received: by mail-il1-f195.google.com with SMTP id x15so459299ilq.1 for ; Tue, 01 Dec 2020 21:23:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=pyAQwS4Co7I2R95ag+5lXrPe/9p6neLUf3n9lgyrkG0=; b=kRknJsSQGtDUohS7hN3L95LW6SIb5DalNHHH0d5LSFjWcttO34k/TSUtL0SfKRRaIL +adzzZ0P//8mfJ5hh7Lo8vss3XDPG9mh0MbXEluUFMMezkGfg00sb5ZqM4B+BzBbTKfT LHeGZLAlYWA+YO9yBEIdiIEeaYa9QIp9AjT+NSTBaM8V+GpFhuQMghQ2k7mmVhTS4JxH wKiDJYMXgzDASGPKAOmxSq/zDsxk6rhnu5b9wFZn9ACVOqWTj0agFa72tzERrDhIl/Qc 7LwsnYuOmJ5miF6Z48UyA9E55Ftc4dn73D1FotjYKNc2gLS0jdNS0EUHX91BN9CbKE8p Gcig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pyAQwS4Co7I2R95ag+5lXrPe/9p6neLUf3n9lgyrkG0=; b=TWKm4ApzaEAc9Wwem62oJqdOToy5xMJmRJRfRP+WGmdpAuaRsuLRZUbYEmoRL+IDLz 8fSHAAiY4tKuo33x61fq8DGeCau7PV1ntCF1RB49o5L77mm/UzgPD5ktOcUdspfVlgSe 3nP3LGs4tMFPd+o95XACf9iVu0pJoQ2BUNH2QfJDKm2aKTBuiNU4EEsmCqdgjBj0JKDj hhbcRTPhhaickaBxo61wH7AeTgS8s1/W94bX+CantARl+5y6uOWqw5yPcxYeXro0dKTD Tsx3eOtjDBNi49Fz80FP2jDmuqltxE4eDktVNnsackb24ZeZOElSK+GEzvvMnB8Ayrbq 01Zw== X-Gm-Message-State: AOAM531/zcXePVjWj3Jsa77Au7f3tePTkn7/XpiQ0ESmvWxP+YD71+hr I7c8j4vfNyEun7twpvnpXcpt5A== X-Google-Smtp-Source: ABdhPJz0k5IzB0S69VOwNzsfUHyaCVWqenfV/i20fxRSf01ht5jDP/pDg/VPs9BmYrKW9bm90bbjOQ== X-Received: by 2002:a92:850f:: with SMTP id f15mr991400ilh.286.1606886624807; Tue, 01 Dec 2020 21:23:44 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m19sm295938ila.81.2020.12.01.21.23.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 21:23:44 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com Subject: [PATCH 5/6] mm: honor PF_MEMALLOC_NOMOVABLE for all allocations Date: Wed, 2 Dec 2020 00:23:29 -0500 Message-Id: <20201202052330.474592-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201202052330.474592-1-pasha.tatashin@soleen.com> References: <20201202052330.474592-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PF_MEMALLOC_NOMOVABLE is only honored for CMA allocations, extend this flag to work for any allocations by removing __GFP_MOVABLE from gfp_mask when this flag is passed in the current context, thus prohibiting allocations from ZONE_MOVABLE. Signed-off-by: Pavel Tatashin --- mm/hugetlb.c | 2 +- mm/page_alloc.c | 26 ++++++++++++++++---------- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 02213c74ed6b..00e786201d8b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1036,7 +1036,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) bool nomovable = !!(current->flags & PF_MEMALLOC_NOMOVABLE); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (nomovable && is_migrate_cma_page(page)) + if (nomovable && is_migrate_movable(get_pageblock_migratetype(page))) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 611799c72da5..7a6d86d0bc5f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3766,20 +3766,25 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) return alloc_flags; } -static inline unsigned int current_alloc_flags(gfp_t gfp_mask, - unsigned int alloc_flags) +static inline unsigned int cma_alloc_flags(gfp_t gfp_mask, + unsigned int alloc_flags) { #ifdef CONFIG_CMA - unsigned int pflags = current->flags; - - if (!(pflags & PF_MEMALLOC_NOMOVABLE) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; - #endif return alloc_flags; } +static inline gfp_t current_gfp_checkmovable(gfp_t gfp_mask) +{ + unsigned int pflags = current->flags; + + if ((pflags & PF_MEMALLOC_NOMOVABLE)) + return gfp_mask & ~__GFP_MOVABLE; + return gfp_mask; +} + /* * get_page_from_freelist goes through the zonelist trying to allocate * a page. @@ -4423,7 +4428,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; - alloc_flags = current_alloc_flags(gfp_mask, alloc_flags); + alloc_flags = cma_alloc_flags(gfp_mask, alloc_flags); return alloc_flags; } @@ -4725,7 +4730,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) - alloc_flags = current_alloc_flags(gfp_mask, reserve_flags); + alloc_flags = cma_alloc_flags(gfp_mask, reserve_flags); /* * Reset the nodemask and zonelist iterators if memory policies can be @@ -4894,7 +4899,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, if (should_fail_alloc_page(gfp_mask, order)) return false; - *alloc_flags = current_alloc_flags(gfp_mask, *alloc_flags); + *alloc_flags = cma_alloc_flags(gfp_mask, *alloc_flags); /* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE); @@ -4932,6 +4937,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, } gfp_mask &= gfp_allowed_mask; + gfp_mask = current_gfp_checkmovable(gfp_mask); alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags)) return NULL; From patchwork Wed Dec 2 05:23:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11944739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E99CC64E90 for ; Wed, 2 Dec 2020 05:23:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A2B772065C for ; Wed, 2 Dec 2020 05:23:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A2B772065C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5D21C8D000B; Wed, 2 Dec 2020 00:23:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 583148D0002; Wed, 2 Dec 2020 00:23:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FD4A8D000B; Wed, 2 Dec 2020 00:23:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 2B8C58D0002 for ; Wed, 2 Dec 2020 00:23:48 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EDA1C180AD815 for ; Wed, 2 Dec 2020 05:23:47 +0000 (UTC) X-FDA: 77547200094.05.bird03_3a10804273b0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id D5A9F1801ABD4 for ; Wed, 2 Dec 2020 05:23:47 +0000 (UTC) X-HE-Tag: bird03_3a10804273b0 X-Filterd-Recvd-Size: 9026 Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 05:23:47 +0000 (UTC) Received: by mail-io1-f68.google.com with SMTP id y5so671254iow.5 for ; Tue, 01 Dec 2020 21:23:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ThlXfcc364FojUO3y5P/ifl4RemTQu3v9eNcTeklg7g=; b=V62z5icg0nzFchCWhHxdLHnEZw8ThRBnzwT84W37nOjJNHZaHjOMna6MYZ/5k2l8y0 iQRgQ3rKIhbf1Y4YPk8dCKgJfeDzOCFNKCLjEf3Tc91smlQ9jjhhQcQ+zZKBSWUI5my7 wszyyjhQcfu6lxWLgH6v2zovh70tmMPk2eV/kqaskr+T12Z3Ihk834pXeAP/GjoI7mFc H+0sCKJYy5tsBosNAupfcsW+nzCuZBmePwqNdzM8MjKpXsYdQWvX629BeG06u5w38qkt IQUNHWugAurAIntDBvxS1L+R/jcHhjSvH1j61JJX9UNngu6CPkHJvC7ksf7wb+y7bm5u xKqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ThlXfcc364FojUO3y5P/ifl4RemTQu3v9eNcTeklg7g=; b=ZzAYSf6aO+JzwnCz6nkqtCngme66RoFhkpSBlIfuFbCs2Lw9RMHLDOZCOhWJHOlU1v KoBy6CNl2zOaeeEMgNggNUo9TRvp2KRa8PTS/vDMC9C/yWoIYJQAEBYwJGKcNR56dcvw opLKL22D0tXdoLC35p9JZAF7Y894F4IILjc0MU78J311WudkdQwjrt/bacqRMshPHf8G ACmVELry0TVxtbrNtXHevUv2g2ZeMJCxAsJ6rgzkiyUYcRSxqw6KgWhvnbF0bYbhYy3c vthc8pmn9H16D6ibFiHdwYMADZuyAr/yW+LO+otAJQJ3k+l081C3kLju5zinZ4sR3LfX PlZQ== X-Gm-Message-State: AOAM5328TJKkNIio8Btc8+5bWjDgmGrIHA2yuEwlnpvvItW8oLw8RnRx F4jjZVEyazhh/4C9uMfytvsvtg== X-Google-Smtp-Source: ABdhPJzTLoCZReLgz51Hrk6XAF0pPmsQoEP7xNffwdXB4knjafqxqpMOhwLiaadyI7uMnqTjiR/W9g== X-Received: by 2002:a05:6602:2209:: with SMTP id n9mr605230ion.181.1606886626860; Tue, 01 Dec 2020 21:23:46 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id m19sm295938ila.81.2020.12.01.21.23.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Dec 2020 21:23:46 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com Subject: [PATCH 6/6] mm/gup: migrate pinned pages out of movable zone Date: Wed, 2 Dec 2020 00:23:30 -0500 Message-Id: <20201202052330.474592-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201202052330.474592-1-pasha.tatashin@soleen.com> References: <20201202052330.474592-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We do not allocate pin pages in ZONE_MOVABLE, but if pages were already allocated before pinning they need to migrated to a different zone. Currently, we migrate movable CMA pages only. Generalize the function that migrates CMA pages to migrate all movable pages. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- include/linux/migrate.h | 1 + include/trace/events/migrate.h | 3 +- mm/gup.c | 56 +++++++++++++--------------------- 3 files changed, 24 insertions(+), 36 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 0f8d1583fa8e..00bab23d1ee5 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -27,6 +27,7 @@ enum migrate_reason { MR_MEMPOLICY_MBIND, MR_NUMA_MISPLACED, MR_CONTIG_RANGE, + MR_LONGTERM_PIN, MR_TYPES }; diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index 4d434398d64d..363b54ce104c 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -20,7 +20,8 @@ EM( MR_SYSCALL, "syscall_or_cpuset") \ EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ EM( MR_NUMA_MISPLACED, "numa_misplaced") \ - EMe(MR_CONTIG_RANGE, "contig_range") + EM( MR_CONTIG_RANGE, "contig_range") \ + EMe(MR_LONGTERM_PIN, "longterm_pin") /* * First define the enums in the above macros to be exported to userspace diff --git a/mm/gup.c b/mm/gup.c index 724d8a65e1df..1d511f65f8a7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1593,19 +1593,18 @@ static bool check_dax_vmas(struct vm_area_struct **vmas, long nr_pages) } #endif -#ifdef CONFIG_CMA -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) +static long check_and_migrate_movable_pages(struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int gup_flags) { unsigned long i; unsigned long step; bool drain_allow = true; bool migrate_allow = true; - LIST_HEAD(cma_page_list); + LIST_HEAD(page_list); long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, @@ -1623,13 +1622,12 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, */ step = compound_nr(head) - (pages[i] - head); /* - * If we get a page from the CMA zone, since we are going to - * be pinning these entries, we might as well move them out - * of the CMA zone if possible. + * If we get a movable page, since we are going to be pinning + * these entries, try to move them out if possible. */ - if (is_migrate_cma_page(head)) { + if (is_migrate_movable(get_pageblock_migratetype(head))) { if (PageHuge(head)) - isolate_huge_page(head, &cma_page_list); + isolate_huge_page(head, &page_list); else { if (!PageLRU(head) && drain_allow) { lru_add_drain_all(); @@ -1637,7 +1635,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, } if (!isolate_lru_page(head)) { - list_add_tail(&head->lru, &cma_page_list); + list_add_tail(&head->lru, &page_list); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), @@ -1649,7 +1647,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, i += step; } - if (!list_empty(&cma_page_list)) { + if (!list_empty(&page_list)) { /* * drop the above get_user_pages reference. */ @@ -1659,7 +1657,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, for (i = 0; i < nr_pages; i++) put_page(pages[i]); - if (migrate_pages(&cma_page_list, alloc_migration_target, NULL, + if (migrate_pages(&page_list, alloc_migration_target, NULL, (unsigned long)&mtc, MIGRATE_SYNC, MR_CONTIG_RANGE)) { /* * some of the pages failed migration. Do get_user_pages @@ -1667,17 +1665,16 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, */ migrate_allow = false; - if (!list_empty(&cma_page_list)) - putback_movable_pages(&cma_page_list); + if (!list_empty(&page_list)) + putback_movable_pages(&page_list); } /* * We did migrate all the pages, Try to get the page references - * again migrating any new CMA pages which we failed to isolate - * earlier. + * again migrating any pages which we failed to isolate earlier. */ ret = __get_user_pages_locked(mm, start, nr_pages, - pages, vmas, NULL, - gup_flags); + pages, vmas, NULL, + gup_flags); if ((ret > 0) && migrate_allow) { nr_pages = ret; @@ -1688,17 +1685,6 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, return ret; } -#else -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) -{ - return nr_pages; -} -#endif /* CONFIG_CMA */ /* * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which @@ -1746,8 +1732,8 @@ static long __gup_longterm_locked(struct mm_struct *mm, goto out; } - rc = check_and_migrate_cma_pages(mm, start, rc, pages, - vmas_tmp, gup_flags); + rc = check_and_migrate_movable_pages(mm, start, rc, pages, + vmas_tmp, gup_flags); out: memalloc_nomovable_restore(flags); }