From patchwork Mon Jan 25 19:47:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A411C433E0 for ; Mon, 25 Jan 2021 19:47:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A6CE32251F for ; Mon, 25 Jan 2021 19:47:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A6CE32251F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1967B8D0026; Mon, 25 Jan 2021 14:47:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 11E438D0023; Mon, 25 Jan 2021 14:47:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB3808D0026; Mon, 25 Jan 2021 14:47:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id C50028D0023 for ; Mon, 25 Jan 2021 14:47:56 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 826FC181AEF32 for ; Mon, 25 Jan 2021 19:47:56 +0000 (UTC) X-FDA: 77745332952.21.drop72_120c51427588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 54278180442CB for ; Mon, 25 Jan 2021 19:47:56 +0000 (UTC) X-HE-Tag: drop72_120c51427588 X-Filterd-Recvd-Size: 4383 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:47:55 +0000 (UTC) Received: by mail-qt1-f182.google.com with SMTP id d15so10566119qtw.12 for ; Mon, 25 Jan 2021 11:47:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uI6NShNpRZxuWFJgcFGIsf9y7F3FhEyQzIGIbDpR8GM=; b=APd7o0SfBkNxtMpH7XgAZalnVlax3qUjjcZ+T+cdkl721zjTqiyrNgXce5aI1JJhVI bNjSKFbZ1wyY6khN4tVflZG3pBI0NyM2Pn5NrhEvsFSUoi1iFeBR0qBTvqdDK2YqWTol HRwdRsuAbAocWhIzZA35IFuhtxT8eF3U+58pIGQioZzxndFcdjcQ6Emht1tGSp1l+rE1 kWHtBDtJIHZ3ynh8lE5qFiJaIg3upEOlt3xN0Q5EXPmSjxt3CHFa8QBb1U8FuKGt/KY5 5LkyG1H1xtQv/uujJpKHSKwMbDKLK8S3mJSx/veQ+M8RVAWGWdY7TtO2zjzMb8ITmdw3 OzuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uI6NShNpRZxuWFJgcFGIsf9y7F3FhEyQzIGIbDpR8GM=; b=J/B5mJXQCrnyQQWeNSHKIbxgvis95aWd7xamwT7ugPbgWHy5II1B9bnLFj9yETLBap Kecim1NexUZ44BWdUzEQNzNirHow6nC1xsKGXGjU1cSpZWcjdP8GzN9tGVtEDCY2uRU3 1WI7GRqOVpf5anXyCh1AeXOyuQuSWJv8sSJg/PFDkR4KEb1veQpHXJUXFNee93FMI7A1 Dos0cZehr7UgVKmpqA/Fl4y+Z923QAzLwLFCSfC+XQLkrlK3VhyLhQpAagqloTEgHwuN C1DuDgMdxhvePxhO2WrZzxeGXI8KHWoE2LrbBum0IiFzwELJux/lzPbtA6ZhMQL681UP KAzw== X-Gm-Message-State: AOAM531XyStUC0eVGxNmv9OlVxYTOC1B6FLsFzcjGPNwJEUPFejMRGLq dNgMZdCOqs/Yv5x+OtVM2t39Aw== X-Google-Smtp-Source: ABdhPJw5iCQaRdjeqKnek/6/7TMDHdOTBAJYl2HAxX/BZQACvfjwwxQckKi3CGyggwDyfJkTML5qaw== X-Received: by 2002:ac8:e82:: with SMTP id v2mr2047681qti.164.1611604075264; Mon, 25 Jan 2021 11:47:55 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.47.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:47:54 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 01/14] mm/gup: don't pin migrated cma pages in movable zone Date: Mon, 25 Jan 2021 14:47:38 -0500 Message-Id: <20210125194751.1275316-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order not to fragment CMA the pinned pages are migrated. However, they are migrated to ZONE_MOVABLE, which also should not have pinned pages. Remove __GFP_MOVABLE, so pages can be migrated to zones where pinning is allowed. Signed-off-by: Pavel Tatashin Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Acked-by: Michal Hocko --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index 3e086b073624..24f25b1e9103 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1563,7 +1563,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, long ret = nr_pages; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, - .gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_NOWARN, + .gfp_mask = GFP_USER | __GFP_NOWARN, }; check_again: From patchwork Mon Jan 25 19:47:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACA54C433E6 for ; Mon, 25 Jan 2021 19:47:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 415D522525 for ; Mon, 25 Jan 2021 19:47:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 415D522525 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CB62A8D002E; Mon, 25 Jan 2021 14:47:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C41858D0023; Mon, 25 Jan 2021 14:47:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A461B8D002E; Mon, 25 Jan 2021 14:47:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id 7D8188D0023 for ; Mon, 25 Jan 2021 14:47:58 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 3E8248249980 for ; Mon, 25 Jan 2021 19:47:58 +0000 (UTC) X-FDA: 77745333036.13.flame09_390160227588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 1DD4818140B70 for ; Mon, 25 Jan 2021 19:47:58 +0000 (UTC) X-HE-Tag: flame09_390160227588 X-Filterd-Recvd-Size: 5772 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:47:57 +0000 (UTC) Received: by mail-qk1-f175.google.com with SMTP id 19so13732493qkh.3 for ; Mon, 25 Jan 2021 11:47:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=/BFgBC6QAZDKdqJwclIw/UFhagKoK9nwCvAO/0D2vSo=; b=CmdxkXZULQNoAHH3ACXza+mPWBw4TkEUp5gdwo9v367G5ivVreRo2jfDKVnAfklKCZ lj/PfZNwM9XMAgMXUnM98CEkHtoOECGI2Y0fu0q1xxQ0NEIxll/TMuaZo65GQpW2XHyU rIrv4ioZ4SPXFX6hExoeutaJaFN6Zu0xoA3DImi6eVCWWNdMmGSBi/p3ICAR+yl+WaeJ R3+LncgUE2Ls+wUl0QzLu91lk0bsnXlR1jdnsIdi+99JwLhK0a83VrLiaiiVKZqdSbG3 1xmIOxMMElDpTxuj70785Z55JdnBzrgS5LSZRWHM+jZ0kX/7mCQz37JfMpZxQ6ziTejt Y3tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/BFgBC6QAZDKdqJwclIw/UFhagKoK9nwCvAO/0D2vSo=; b=tSKVaEMdbqcHITqEdbHN4elcIThnPHS/co3ySojA775lwgge68bHqiMfDtIsvaHtm2 V9E8jUCGF8/+g8QhU5j9OU01mvb8A7/XX00TLDjA57xnWmPtWcSxVXM9s3pqJLzTg9Jk bMJ6gtTiugZ6pL/uq8pwLQ/Ce9kFT7fSR/faf0HqQTu+EH0h6ugfpvoRSubR7SQqgKKe LLf22pcj4PmAVnTBln31Yha3jm965M68z7tdzJXianBRmJkHfvPWVPMkqVHvEwsId9k0 rYKCoGGoPbWIcY9MqV6kLQ3PI949mE0/88WfwVGG4MWlNQdYsddq1cHJTOhBMLsMkQj+ rI2w== X-Gm-Message-State: AOAM532CvEznQQCbmSnj0yriQlekxpPdc3/pWhBoswh+b4v1tY0AIVoU RAU5zp3RZpRtYvmoeLTLGlA6Uw== X-Google-Smtp-Source: ABdhPJz+Q8gL0GZ9W1lsgErCnpY5WXm31yZTxCvb9A79URsfZ1Q6xwrI2e5Q2K0QFJueffLs1BietQ== X-Received: by 2002:a37:d58:: with SMTP id 85mr2382064qkn.280.1611604076921; Mon, 25 Jan 2021 11:47:56 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.47.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:47:56 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 02/14] mm/gup: check every subpage of a compound page during isolation Date: Mon, 25 Jan 2021 14:47:39 -0500 Message-Id: <20210125194751.1275316-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When pages are isolated in check_and_migrate_movable_pages() we skip compound number of pages at a time. However, as Jason noted, it is not necessary correct that pages[i] corresponds to the pages that we skipped. This is because it is possible that the addresses in this range had split_huge_pmd()/split_huge_pud(), and these functions do not update the compound page metadata. The problem can be reproduced if something like this occurs: 1. User faulted huge pages. 2. split_huge_pmd() was called for some reason 3. User has unmapped some sub-pages in the range 4. User tries to longterm pin the addresses. The resulting pages[i] might end-up having pages which are not compound size page aligned. Fixes: aa712399c1e8 ("mm/gup: speed up check_and_migrate_cma_pages() on huge page") Reported-by: Jason Gunthorpe Signed-off-by: Pavel Tatashin Reviewed-by: Jason Gunthorpe --- mm/gup.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 24f25b1e9103..16f10d5a9eb6 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1556,26 +1556,23 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, unsigned int gup_flags) { unsigned long i; - unsigned long step; bool drain_allow = true; bool migrate_allow = true; LIST_HEAD(cma_page_list); long ret = nr_pages; + struct page *prev_head, *head; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, .gfp_mask = GFP_USER | __GFP_NOWARN, }; check_again: - for (i = 0; i < nr_pages;) { - - struct page *head = compound_head(pages[i]); - - /* - * gup may start from a tail page. Advance step by the left - * part. - */ - step = compound_nr(head) - (pages[i] - head); + prev_head = NULL; + for (i = 0; i < nr_pages; i++) { + head = compound_head(pages[i]); + if (head == prev_head) + continue; + prev_head = head; /* * If we get a page from the CMA zone, since we are going to * be pinning these entries, we might as well move them out @@ -1599,8 +1596,6 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, } } } - - i += step; } if (!list_empty(&cma_page_list)) { From patchwork Mon Jan 25 19:47:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8AE5C433E0 for ; Mon, 25 Jan 2021 19:48:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6639D22525 for ; Mon, 25 Jan 2021 19:48:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6639D22525 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A475F8D002F; Mon, 25 Jan 2021 14:48:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D0098D0023; Mon, 25 Jan 2021 14:48:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8959B8D002F; Mon, 25 Jan 2021 14:48:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id 5FB328D0023 for ; Mon, 25 Jan 2021 14:48:00 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CF1D03649 for ; Mon, 25 Jan 2021 19:47:59 +0000 (UTC) X-FDA: 77745333078.17.pail91_5c01a1e27588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id ACA51180D0184 for ; Mon, 25 Jan 2021 19:47:59 +0000 (UTC) X-HE-Tag: pail91_5c01a1e27588 X-Filterd-Recvd-Size: 5430 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:47:59 +0000 (UTC) Received: by mail-qk1-f170.google.com with SMTP id a12so3796215qkh.10 for ; Mon, 25 Jan 2021 11:47:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=gHgQfTRvar1lIU9rNG4Q491ak9Bif38ih9mdt5Czeeg=; b=c7sG1z0zx4IBwrpL9ujY99TSc8IAU20MGw/TaPpyT+YoiodG5nGIzUR/NTujP0u1kd Rn3irH637aV9Tmqlkhd1csC9dMvLFqE/l4J73pOWLHBaE3rnkGsjFAYCS75cZmh9X/Lt 1TOlZeUR85/P6119/G1x0xgTxAB4RbFCyKLmQv712qp8Sftqrk4WSgdOJZ6OAba0d3o1 ThrDl65vkoeoFfWW3zq4GYnGO2B/aFe1mVY6tV2v2QtEZtBJOicRcxRAsE6gZMKuRVhD YBiPHBxO83ut5cd4fxWVQPDYCMuccwOB2WH2d9CFF/jxKSQJLwOgPwjtb3L85cI7WFie N16g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gHgQfTRvar1lIU9rNG4Q491ak9Bif38ih9mdt5Czeeg=; b=SFZg9LVTEUa0G6cnTJhbvhYxTfl5cLtavm0soW/ATJ/E1gvhaK9gb5e6ZImSawBnJ7 DLf4upYh9tTtY1/FE/Pvmfr7o5ygGGF03FV7GAfy/BsiRHE5YZVU4bKUZqCPaEhMr4FQ 6HYMFscQtfczaeLQ3+AHXtHUFh/dfLKspnrVDyJuylvQ0iWpoOJfOz0ejxlUPwxK8RH+ ZLgJMhAuWq1HYjrpW/21rfhLcdTIFxyZuqTOZyaCpm6x/Cy3Xav3OSdj+7XDMouVya7Z ylKNG0VLJa0wKmqpwp+yyPHB4cmn/AKCP5wf8j/HJ+1rXAJ8/lI5BC0Ocyxcu0n4mZ6b Jrdg== X-Gm-Message-State: AOAM531/6QjOMCPS5Srmg0veILmYdwQdl8mhLnmo5j9gjOtlQqNv+TzI C5CPC3hRmBLO+XOx4Y3B/6mrHw== X-Google-Smtp-Source: ABdhPJz3Za/JbaLs+6ZSMuUuylAr92cCaBZaN+5myWkrAXEtJPMKFBUlfdcIgUMufV3m8N/ZNiYoqg== X-Received: by 2002:a37:8703:: with SMTP id j3mr2422378qkd.455.1611604078581; Mon, 25 Jan 2021 11:47:58 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.47.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:47:58 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 03/14] mm/gup: return an error on migration failure Date: Mon, 25 Jan 2021 14:47:40 -0500 Message-Id: <20210125194751.1275316-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When migration failure occurs, we still pin pages, which means that we may pin CMA movable pages which should never be the case. Instead return an error without pinning pages when migration failure happens. No need to retry migrating, because migrate_pages() already retries 10 times. Signed-off-by: Pavel Tatashin Reviewed-by: Jason Gunthorpe --- mm/gup.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 16f10d5a9eb6..88ce41f41543 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1557,7 +1557,6 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, { unsigned long i; bool drain_allow = true; - bool migrate_allow = true; LIST_HEAD(cma_page_list); long ret = nr_pages; struct page *prev_head, *head; @@ -1608,17 +1607,15 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, for (i = 0; i < nr_pages; i++) put_page(pages[i]); - if (migrate_pages(&cma_page_list, alloc_migration_target, NULL, - (unsigned long)&mtc, MIGRATE_SYNC, MR_CONTIG_RANGE)) { - /* - * some of the pages failed migration. Do get_user_pages - * without migration. - */ - migrate_allow = false; - + ret = migrate_pages(&cma_page_list, alloc_migration_target, + NULL, (unsigned long)&mtc, MIGRATE_SYNC, + MR_CONTIG_RANGE); + if (ret) { if (!list_empty(&cma_page_list)) putback_movable_pages(&cma_page_list); + return ret > 0 ? -ENOMEM : ret; } + /* * We did migrate all the pages, Try to get the page references * again migrating any new CMA pages which we failed to isolate @@ -1628,7 +1625,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, pages, vmas, NULL, gup_flags); - if ((ret > 0) && migrate_allow) { + if (ret > 0) { nr_pages = ret; drain_allow = true; goto check_again; From patchwork Mon Jan 25 19:47:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE239C433E6 for ; Mon, 25 Jan 2021 19:48:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4786E22525 for ; Mon, 25 Jan 2021 19:48:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4786E22525 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D7CE38D0030; Mon, 25 Jan 2021 14:48:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8DBB8D0023; Mon, 25 Jan 2021 14:48:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2D878D0030; Mon, 25 Jan 2021 14:48:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id 9D0F08D0023 for ; Mon, 25 Jan 2021 14:48:01 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 64EE1180AE7F3 for ; Mon, 25 Jan 2021 19:48:01 +0000 (UTC) X-FDA: 77745333162.25.bikes84_3a13bb027588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 369AE1804E3A0 for ; Mon, 25 Jan 2021 19:48:01 +0000 (UTC) X-HE-Tag: bikes84_3a13bb027588 X-Filterd-Recvd-Size: 7167 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:00 +0000 (UTC) Received: by mail-qk1-f173.google.com with SMTP id r77so13526569qka.12 for ; Mon, 25 Jan 2021 11:48:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=g2firNhV/7kB7YU7cH9BNSgXaQWIxvW5HlS98FNnYEs=; b=AMPWTTnTQtQrH0f812HX8J/XenqFD0LhTJ4uvra0Yc82gbHE9JUqY698LNtS80BE7R HR+xMh6u82Ttl2HRET2KJJgXpfY0HAW1XAkqnXoYkorW9DdsIXlDDaKOJsRmrtQ2/nEG Dmb4GCZxUK9zEwlcH3WDx/URO0SVMy0zcD9v2FKoTwgARO1MZ/OlQM2BSz/juCLIjm3+ Ea0mQylNJ+kUPUNcC5yvUomLFphaeUQkqGKv5Qe3PZZaI9d8ehH1Z1VgrSynAksIDQ8Q 1KdzNIxtcFbSYYmlQkrBgCfYAOIWMyB8X7sV+6aCuasMoW4NW0MbgSIrAIc2CBltCiiY S4Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g2firNhV/7kB7YU7cH9BNSgXaQWIxvW5HlS98FNnYEs=; b=QTmKP1VNumhVBu0LR9xtdySWwtatw3pqsFarHbARC8QQWuiDCSPnw7ypmhPs3068ht 7//HUX62As3kR9Y+PMQAN6t6O1vWnGT4VRMOmKAkHE26GSs3MYZQB3hXS6EHuUh9nB3L RVR+6qAbMs4z7/3Mfv1VE5DNjgPtUxhz3Nl8n7dxviwhxPIgACw3PEzyt9+Hd7/eYLjN 9EGVbvGA87VNG+de4Pc4dEZT9wb7hju+JLGwY2go6zFXQM6QXW9ju+foIfMTyJy1K63s /lacSjKFFE2y7YWYvdZSiedseDdl9oIUec/lBbmTdmOJSiVq6m1FzxGKs0VinoT29cQZ GjUA== X-Gm-Message-State: AOAM533A+Y3uY3LughH8F8ceb+ZKyT32/TJGZ6ow0uTjDJGyFVD7iUQ/ wS766u9uTe60D39SbEeGqrXcLA== X-Google-Smtp-Source: ABdhPJwYEL3DoI+pABS4RBJJVNs455EMYUvP8bvQI2v1Jkb8103rBVGefY9aunaJRQVj9v3yEHgwng== X-Received: by 2002:a37:4554:: with SMTP id s81mr2369392qka.323.1611604080206; Mon, 25 Jan 2021 11:48:00 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.47.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:47:59 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 04/14] mm/gup: check for isolation errors Date: Mon, 25 Jan 2021 14:47:41 -0500 Message-Id: <20210125194751.1275316-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is still possible that we pin movable CMA pages if there are isolation errors and cma_page_list stays empty when we check again. Check for isolation errors, and return success only when there are no isolation errors, and cma_page_list is empty after checking. Because isolation errors are transient, we retry indefinitely. Fixes: 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages allocated from CMA region") Signed-off-by: Pavel Tatashin Reviewed-by: Jason Gunthorpe --- mm/gup.c | 60 ++++++++++++++++++++++++++++++++------------------------ 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 88ce41f41543..7ecca2d66dff 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1555,8 +1555,8 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, struct vm_area_struct **vmas, unsigned int gup_flags) { - unsigned long i; - bool drain_allow = true; + unsigned long i, isolation_error_count; + bool drain_allow; LIST_HEAD(cma_page_list); long ret = nr_pages; struct page *prev_head, *head; @@ -1567,6 +1567,8 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, check_again: prev_head = NULL; + isolation_error_count = 0; + drain_allow = true; for (i = 0; i < nr_pages; i++) { head = compound_head(pages[i]); if (head == prev_head) @@ -1578,25 +1580,35 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, * of the CMA zone if possible. */ if (is_migrate_cma_page(head)) { - if (PageHuge(head)) - isolate_huge_page(head, &cma_page_list); - else { + if (PageHuge(head)) { + if (!isolate_huge_page(head, &cma_page_list)) + isolation_error_count++; + } else { if (!PageLRU(head) && drain_allow) { lru_add_drain_all(); drain_allow = false; } - if (!isolate_lru_page(head)) { - list_add_tail(&head->lru, &cma_page_list); - mod_node_page_state(page_pgdat(head), - NR_ISOLATED_ANON + - page_is_file_lru(head), - thp_nr_pages(head)); + if (isolate_lru_page(head)) { + isolation_error_count++; + continue; } + list_add_tail(&head->lru, &cma_page_list); + mod_node_page_state(page_pgdat(head), + NR_ISOLATED_ANON + + page_is_file_lru(head), + thp_nr_pages(head)); } } } + /* + * If list is empty, and no isolation errors, means that all pages are + * in the correct zone. + */ + if (list_empty(&cma_page_list) && !isolation_error_count) + return ret; + if (!list_empty(&cma_page_list)) { /* * drop the above get_user_pages reference. @@ -1616,23 +1628,19 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, return ret > 0 ? -ENOMEM : ret; } - /* - * We did migrate all the pages, Try to get the page references - * again migrating any new CMA pages which we failed to isolate - * earlier. - */ - ret = __get_user_pages_locked(mm, start, nr_pages, - pages, vmas, NULL, - gup_flags); - - if (ret > 0) { - nr_pages = ret; - drain_allow = true; - goto check_again; - } + /* We unpinned pages before migration, pin them again */ + ret = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + NULL, gup_flags); + if (ret <= 0) + return ret; + nr_pages = ret; } - return ret; + /* + * check again because pages were unpinned, and we also might have + * had isolation errors and need more pages to migrate. + */ + goto check_again; } #else static long check_and_migrate_cma_pages(struct mm_struct *mm, From patchwork Mon Jan 25 19:47:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3521C433E0 for ; Mon, 25 Jan 2021 19:48:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 685132251F for ; Mon, 25 Jan 2021 19:48:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 685132251F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A4C518D0031; Mon, 25 Jan 2021 14:48:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95D338D0023; Mon, 25 Jan 2021 14:48:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D2128D0031; Mon, 25 Jan 2021 14:48:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 59E018D0023 for ; Mon, 25 Jan 2021 14:48:03 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1CF161EE6 for ; Mon, 25 Jan 2021 19:48:03 +0000 (UTC) X-FDA: 77745333246.30.field58_2716a5927588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id EFE24180B31BF for ; Mon, 25 Jan 2021 19:48:02 +0000 (UTC) X-HE-Tag: field58_2716a5927588 X-Filterd-Recvd-Size: 8571 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:02 +0000 (UTC) Received: by mail-qt1-f175.google.com with SMTP id d15so10566345qtw.12 for ; Mon, 25 Jan 2021 11:48:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=erDKiXAtt/ZfDNNFCFxfMe9eST86O7VxkIZqpHHCdMo=; b=gXzBEYM45h44JmTh+SOS1KqJ1znNfWJdkHIooP65qs8VDI0rCPGi5a6NwsG4j/pCNh /2/dc+XTJVBSZrZCNDbuwaGzREMMmuImcnaeIEbB+Q02AmkENedC7dJkYmFjnqVp+Myv DHV05lsECVIBoPoj5Vso7Cfy4F9qn/Jl9/+Kfo/Td4bLGSjaGbivdep/zGt7PiizKxbR gstHnSIHGaFghCA2orssPKJIZ4UdhRzPKHjSS85B3h5LpTQ/Fj1sFh2m0eCI1sdpOOJG BeeMpZ92z2aU8mtDFN3zjna36X77bHpBNPnq1mLUSe0IcOM7v7LCaye3DaZPNqFvcs2y G0ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=erDKiXAtt/ZfDNNFCFxfMe9eST86O7VxkIZqpHHCdMo=; b=fWHhn/rYG40SnSp65WlBVzkNxDn2fJcGyl5WET6Lgp+768K0lLLZklWXhOEfzXR8a+ JkZyS9d7n0ifZfWclYjYqAEvtkKRDPyJiloBp0QmkfwjbC19HISINwCzwaWuPiW4gBNB N73rwHPdmZOofbOm89oHwbudd2NYCtPk9S5P0Ma29aqOwXCtq/xvoSU/gIeFd40BAuCz +yTlwmaQtC2KDZwcugpt1o1eoEArrHkZ6FrF8K5MZY2Nz0MnOMbCCOahn6RI27uCvW9d 4CNZt/y6b9pMcb7RlNU5NwRL/Hm9CJfAr/RXK1aIlVoXwxu1JVX5BDLw+dcCdPhfvR5/ x8iA== X-Gm-Message-State: AOAM533v1lXAe7stwAHBWXf9RqIssxLN3ao00tOKEfuYegbvLUqJnbuz FX9ERG/z9pnRP3LbvWt9HzZaIQ== X-Google-Smtp-Source: ABdhPJwbOUzBr/56M0RmyAQ+ZsDl4fcJihGJYA2TsdytVsQeYtM4XvDXE+uslGEnRTypRVtMn1fUjw== X-Received: by 2002:aed:3661:: with SMTP id e88mr2018577qtb.243.1611604081880; Mon, 25 Jan 2021 11:48:01 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:01 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 05/14] mm cma: rename PF_MEMALLOC_NOCMA to PF_MEMALLOC_PIN Date: Mon, 25 Jan 2021 14:47:42 -0500 Message-Id: <20210125194751.1275316-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PF_MEMALLOC_NOCMA is used ot guarantee that the allocator will not return pages that might belong to CMA region. This is currently used for long term gup to make sure that such pins are not going to be done on any CMA pages. When PF_MEMALLOC_NOCMA has been introduced we haven't realized that it is focusing on CMA pages too much and that there is larger class of pages that need the same treatment. MOVABLE zone cannot contain any long term pins as well so it makes sense to reuse and redefine this flag for that usecase as well. Rename the flag to PF_MEMALLOC_PIN which defines an allocation context which can only get pages suitable for long-term pins. Also re-name: memalloc_nocma_save()/memalloc_nocma_restore to memalloc_pin_save()/memalloc_pin_restore() and make the new functions common. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard Acked-by: Michal Hocko --- include/linux/sched.h | 2 +- include/linux/sched/mm.h | 21 +++++---------------- mm/gup.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/page_alloc.c | 4 ++-- 5 files changed, 12 insertions(+), 23 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index e47a3823f408..f90a27611926 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1576,7 +1576,7 @@ extern struct pid *cad_pid; #define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */ #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ #define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */ -#define PF_MEMALLOC_NOCMA 0x10000000 /* All allocation request will have _GFP_MOVABLE cleared */ +#define PF_MEMALLOC_PIN 0x10000000 /* Allocation context constrained to zones which allow long term pinning. */ #define PF_FREEZER_SKIP 0x40000000 /* Freezer should not count it as freezable */ #define PF_SUSPEND_TASK 0x80000000 /* This thread called freeze_processes() and should not be frozen */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 1ae08b8462a4..5f4dd3274734 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -270,29 +270,18 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) current->flags = (current->flags & ~PF_MEMALLOC) | flags; } -#ifdef CONFIG_CMA -static inline unsigned int memalloc_nocma_save(void) +static inline unsigned int memalloc_pin_save(void) { - unsigned int flags = current->flags & PF_MEMALLOC_NOCMA; + unsigned int flags = current->flags & PF_MEMALLOC_PIN; - current->flags |= PF_MEMALLOC_NOCMA; + current->flags |= PF_MEMALLOC_PIN; return flags; } -static inline void memalloc_nocma_restore(unsigned int flags) +static inline void memalloc_pin_restore(unsigned int flags) { - current->flags = (current->flags & ~PF_MEMALLOC_NOCMA) | flags; + current->flags = (current->flags & ~PF_MEMALLOC_PIN) | flags; } -#else -static inline unsigned int memalloc_nocma_save(void) -{ - return 0; -} - -static inline void memalloc_nocma_restore(unsigned int flags) -{ -} -#endif #ifdef CONFIG_MEMCG DECLARE_PER_CPU(struct mem_cgroup *, int_active_memcg); diff --git a/mm/gup.c b/mm/gup.c index 7ecca2d66dff..857b273e32ac 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1669,7 +1669,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, long rc; if (gup_flags & FOLL_LONGTERM) - flags = memalloc_nocma_save(); + flags = memalloc_pin_save(); rc = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, NULL, gup_flags); @@ -1678,7 +1678,7 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (rc > 0) rc = check_and_migrate_cma_pages(mm, start, rc, pages, vmas, gup_flags); - memalloc_nocma_restore(flags); + memalloc_pin_restore(flags); } return rc; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a6bad1f686c5..2d79e515a7a3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1049,10 +1049,10 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) { struct page *page; - bool nocma = !!(current->flags & PF_MEMALLOC_NOCMA); + bool pin = !!(current->flags & PF_MEMALLOC_PIN); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (nocma && is_migrate_cma_page(page)) + if (pin && is_migrate_cma_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b031a5ae0bd5..f92d7c810953 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3813,8 +3813,8 @@ static inline unsigned int current_alloc_flags(gfp_t gfp_mask, #ifdef CONFIG_CMA unsigned int pflags = current->flags; - if (!(pflags & PF_MEMALLOC_NOCMA) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (!(pflags & PF_MEMALLOC_PIN) && + gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; #endif From patchwork Mon Jan 25 19:47:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E9A7C433E6 for ; Mon, 25 Jan 2021 19:48:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9E5612251F for ; Mon, 25 Jan 2021 19:48:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E5612251F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 58AF78D0032; Mon, 25 Jan 2021 14:48:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 53C8C8D0023; Mon, 25 Jan 2021 14:48:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DAB88D0032; Mon, 25 Jan 2021 14:48:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0045.hostedemail.com [216.40.44.45]) by kanga.kvack.org (Postfix) with ESMTP id 257AB8D0023 for ; Mon, 25 Jan 2021 14:48:05 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DAF981EE6 for ; Mon, 25 Jan 2021 19:48:04 +0000 (UTC) X-FDA: 77745333288.27.pear30_2111f6927588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id B8ED53D66D for ; Mon, 25 Jan 2021 19:48:04 +0000 (UTC) X-HE-Tag: pear30_2111f6927588 X-Filterd-Recvd-Size: 5754 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:04 +0000 (UTC) Received: by mail-qv1-f50.google.com with SMTP id dj6so6777729qvb.1 for ; Mon, 25 Jan 2021 11:48:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=BhWP1P8Q7i9lD0EGpT6UypXh912bl/nwbH5yu3+aADs=; b=R90VZ3vGcrRYvZbgV7azLdKs09oOeO1Lk8KPeEtI0CYo+B4Bw6mPEKxqL5eg56jNc9 ZDrbez70bMDtoC6Od7HL0ftNoMEQ3peg7mRtVWsgG325a5PlRlbmKT7RHgvkhtOhId4d 7dYEeW1LvOtO3oZjif6bi5MbAEz9No5RYl4KG4ubdRNqV0/tlyNZMriVg+YWst3j+WTZ PjO8GSotud/xnvOzQkGUVD0179zez9cw1bfDMJSQuTSACrGxlrSUyp4iYu0Ps2hw+Zuv dQq7Ilb7K8Fa+eF8jxWDHJeEvOyhu0eMMUe1CmjKngGzlok/qZkPzNDWJsDq0udh+T7i vr8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BhWP1P8Q7i9lD0EGpT6UypXh912bl/nwbH5yu3+aADs=; b=o4bPBpxiN1Q5IYIwqA/UVhy5ucEk7GmWd7tTB3vZ96pTDXgMKqZROIFmgEaOzc2B4m lAv4SpH6eqnUzxQ8ZgaK4eTP3H+KwKUfszpnZqVVotOldUzqj39xIfmmOm+2QbG/uqOe w8K+p4dwjDoUN79qoJbeWzCKAP58Cv/leuuj3o+86T0NGlSSAsWsIkNg4grOq5/y8HFj RQkn2mX1XY8PdTXVADF5rfwA5STTsDemEEzj3DSy/GLo5e3jXE2FP6FJWf/KSLOsEVfV lsHmepRXA9PnkkBK4O9PaFSE77KkFsdiKXxWruoWp9zl2x4U8vNKcXhBbtzE8PQnYJOH LHzA== X-Gm-Message-State: AOAM531kVAbmmJw42VxN/BRqYmAAvua+l+sZQtBKoq5SkAiO6OFeGURF Wq9JVR4h4a41KvnMQaH3/2rp7Q== X-Google-Smtp-Source: ABdhPJy3QWGRlhCdTJ6t8Bl2AWaYHX6rqsFzWzOohroOr2+KwZLFlfWsdO5+EVfO+/raYzXEIL5oIA== X-Received: by 2002:a0c:c688:: with SMTP id d8mr2432312qvj.8.1611604083512; Mon, 25 Jan 2021 11:48:03 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:02 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 06/14] mm: apply per-task gfp constraints in fast path Date: Mon, 25 Jan 2021 14:47:43 -0500 Message-Id: <20210125194751.1275316-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Function current_gfp_context() is called after fast path. However, soon we will add more constraints which will also limit zones based on context. Move this call into fast path, and apply the correct constraints for all allocations. Also update .reclaim_idx based on value returned by current_gfp_context() because it soon will modify the allowed zones. Note: With this patch we will do one extra current->flags load during fast path, but we already load current->flags in fast-path: __alloc_pages_nodemask() prepare_alloc_pages() current_alloc_flags(gfp_mask, *alloc_flags); Later, when we add the zone constrain logic to current_gfp_context() we will be able to remove current->flags load from current_alloc_flags, and therefore return fast-path to the current performance level. Suggested-by: Michal Hocko Signed-off-by: Pavel Tatashin Acked-by: Michal Hocko --- mm/page_alloc.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f92d7c810953..c93e801a45e9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4981,6 +4981,13 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, } gfp_mask &= gfp_allowed_mask; + /* + * Apply scoped allocation constraints. This is mainly about GFP_NOFS + * resp. GFP_NOIO which has to be inherited for all allocation requests + * from a particular context which has been marked by + * memalloc_no{fs,io}_{save,restore}. + */ + gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags)) return NULL; @@ -4996,13 +5003,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, if (likely(page)) goto out; - /* - * Apply scoped allocation constraints. This is mainly about GFP_NOFS - * resp. GFP_NOIO which has to be inherited for all allocation requests - * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. - */ - alloc_mask = current_gfp_context(gfp_mask); + alloc_mask = gfp_mask; ac.spread_dirty_pages = false; /* From patchwork Mon Jan 25 19:47:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EE01C433DB for ; Mon, 25 Jan 2021 19:48:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC6B320708 for ; Mon, 25 Jan 2021 19:48:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC6B320708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DF4838D0033; Mon, 25 Jan 2021 14:48:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE03C8D0023; Mon, 25 Jan 2021 14:48:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7E858D0033; Mon, 25 Jan 2021 14:48:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0220.hostedemail.com [216.40.44.220]) by kanga.kvack.org (Postfix) with ESMTP id 9DA588D0023 for ; Mon, 25 Jan 2021 14:48:06 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 66A128249980 for ; Mon, 25 Jan 2021 19:48:06 +0000 (UTC) X-FDA: 77745333372.17.jail09_3f06d6727588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 4AD0C180D018B for ; Mon, 25 Jan 2021 19:48:06 +0000 (UTC) X-HE-Tag: jail09_3f06d6727588 X-Filterd-Recvd-Size: 8845 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:05 +0000 (UTC) Received: by mail-qv1-f44.google.com with SMTP id dj6so6777761qvb.1 for ; Mon, 25 Jan 2021 11:48:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=LhppBrlO+4i70tHv2AjpL8r+Cfd3uOXgnDMjQBn3cUs=; b=eq8XSyDB/19ggwJb7BW13KrHkAWCSll+WyrZFUDeOw/o2sm0UeR2miK+j8aTAvhuuY RnyosPmaS+XntefiXQtKI/yVVLNFXu6F4FUZqkNZBVh6DewuKlnFD79rFz63RUBhQhWO dPtOwSUWOZaoYSBXAPs3EA5pN88gc35ZlfcnypX+jkGk3w8usmrs2tbzX4Dhh8RitgqX f/UPa9fZFg0E4Rlrg3Cpe2cVrcokmAgU3uyzKRDCJFuuK46kDLL02B0fQt14tdH49YUY h+GRKsfSAcm/spPnokkjhCrH+jWitWiNEwyzAt96NiAuoBnAJW/fYh3BCOuidDPRj8gt Pjwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LhppBrlO+4i70tHv2AjpL8r+Cfd3uOXgnDMjQBn3cUs=; b=DRsnvrxrGUstQMwysEt8yByPelA7IM79JZZzcmYbhIRb7FGZ9Hove+yvIvFznae1Dv LGwAayua6iSSpBDs5wEZ4fuAXptRNF3vXvFdjvVncPCKfnCGcd9AMua1SPWI5drc8kEv a51AtgDeFzWG8JSzY+VfOpy3z1lWLVAklJgh6FgCxa+CX/i1O5L0+thM4lB7Vk/g4mei hUMypjadyigpKJ3C0dDUutExqoz5wQ3iJ55TD8Yw3GcsO0goN6G2qGqGrO2lvHQrdQdg 92FphZEQJRcXzchLeE6lJBr8cozziBlunXDoT+sZSAxjggDcei+eDyY/PS6DWUR76fqf JkDw== X-Gm-Message-State: AOAM532CrHqxrNG2oJ+UW08i4mNyJm12e4At1cfx80qahiZNq0rxOT3X KG52PwiHypRY2xVbXPFj5LVmww== X-Google-Smtp-Source: ABdhPJy01D2w7lqV6ZgpFHOjOk5EmSTTBYXu3WNtFZmoEH7y9aid9Z3Wq4+CvltSeaSOm8p4T0ZNCA== X-Received: by 2002:ad4:4f83:: with SMTP id em3mr953591qvb.28.1611604085092; Mon, 25 Jan 2021 11:48:05 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:04 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 07/14] mm: honor PF_MEMALLOC_PIN for all movable pages Date: Mon, 25 Jan 2021 14:47:44 -0500 Message-Id: <20210125194751.1275316-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: PF_MEMALLOC_PIN is only honored for CMA pages, extend this flag to work for any allocations from ZONE_MOVABLE by removing __GFP_MOVABLE from gfp_mask when this flag is passed in the current context. Add is_pinnable_page() to return true if page is in a pinnable page. A pinnable page is not in ZONE_MOVABLE and not of MIGRATE_CMA type. Signed-off-by: Pavel Tatashin Acked-by: Michal Hocko --- include/linux/mm.h | 11 +++++++++++ include/linux/sched/mm.h | 6 +++++- mm/hugetlb.c | 2 +- mm/page_alloc.c | 20 +++++++++----------- 4 files changed, 26 insertions(+), 13 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a5d618d08506..0990a76d5e6f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1117,6 +1117,17 @@ static inline bool is_zone_device_page(const struct page *page) } #endif +static inline bool is_zone_movable_page(const struct page *page) +{ + return page_zonenum(page) == ZONE_MOVABLE; +} + +/* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */ +static inline bool is_pinnable_page(struct page *page) +{ + return !is_zone_movable_page(page) && !is_migrate_cma_page(page); +} + #ifdef CONFIG_DEV_PAGEMAP_OPS void free_devmap_managed_page(struct page *page); DECLARE_STATIC_KEY_FALSE(devmap_managed_key); diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 5f4dd3274734..a55277b0d475 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -150,12 +150,13 @@ static inline bool in_vfork(struct task_struct *tsk) * Applies per-task gfp context to the given allocation flags. * PF_MEMALLOC_NOIO implies GFP_NOIO * PF_MEMALLOC_NOFS implies GFP_NOFS + * PF_MEMALLOC_PIN implies !GFP_MOVABLE */ static inline gfp_t current_gfp_context(gfp_t flags) { unsigned int pflags = READ_ONCE(current->flags); - if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS))) { + if (unlikely(pflags & (PF_MEMALLOC_NOIO | PF_MEMALLOC_NOFS | PF_MEMALLOC_PIN))) { /* * NOIO implies both NOIO and NOFS and it is a weaker context * so always make sure it makes precedence @@ -164,6 +165,9 @@ static inline gfp_t current_gfp_context(gfp_t flags) flags &= ~(__GFP_IO | __GFP_FS); else if (pflags & PF_MEMALLOC_NOFS) flags &= ~__GFP_FS; + + if (pflags & PF_MEMALLOC_PIN) + flags &= ~__GFP_MOVABLE; } return flags; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2d79e515a7a3..77098492a2fd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1052,7 +1052,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) bool pin = !!(current->flags & PF_MEMALLOC_PIN); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { - if (pin && is_migrate_cma_page(page)) + if (pin && !is_pinnable_page(page)) continue; if (PageHWPoison(page)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c93e801a45e9..3f17c73ad582 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3807,16 +3807,13 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask) return alloc_flags; } -static inline unsigned int current_alloc_flags(gfp_t gfp_mask, - unsigned int alloc_flags) +/* Must be called after current_gfp_context() which can change gfp_mask */ +static inline unsigned int gpf_to_alloc_flags(gfp_t gfp_mask, + unsigned int alloc_flags) { #ifdef CONFIG_CMA - unsigned int pflags = current->flags; - - if (!(pflags & PF_MEMALLOC_PIN) && - gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) alloc_flags |= ALLOC_CMA; - #endif return alloc_flags; } @@ -4472,7 +4469,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) } else if (unlikely(rt_task(current)) && !in_interrupt()) alloc_flags |= ALLOC_HARDER; - alloc_flags = current_alloc_flags(gfp_mask, alloc_flags); + alloc_flags = gpf_to_alloc_flags(gfp_mask, alloc_flags); return alloc_flags; } @@ -4774,7 +4771,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, reserve_flags = __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) - alloc_flags = current_alloc_flags(gfp_mask, reserve_flags); + alloc_flags = gpf_to_alloc_flags(gfp_mask, reserve_flags); /* * Reset the nodemask and zonelist iterators if memory policies can be @@ -4943,7 +4940,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, if (should_fail_alloc_page(gfp_mask, order)) return false; - *alloc_flags = current_alloc_flags(gfp_mask, *alloc_flags); + *alloc_flags = gpf_to_alloc_flags(gfp_mask, *alloc_flags); /* Dirty zone balancing only done in the fast path */ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE); @@ -4985,7 +4982,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, * Apply scoped allocation constraints. This is mainly about GFP_NOFS * resp. GFP_NOIO which has to be inherited for all allocation requests * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. + * memalloc_no{fs,io}_{save,restore}. And PF_MEMALLOC_PIN which ensures + * movable zones are not used during allocation. */ gfp_mask = current_gfp_context(gfp_mask); alloc_mask = gfp_mask; From patchwork Mon Jan 25 19:47:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52507C433E0 for ; Mon, 25 Jan 2021 19:48:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D956E2072C for ; Mon, 25 Jan 2021 19:48:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D956E2072C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8BDF58D0034; Mon, 25 Jan 2021 14:48:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 86EC48D0023; Mon, 25 Jan 2021 14:48:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C24E8D0034; Mon, 25 Jan 2021 14:48:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0194.hostedemail.com [216.40.44.194]) by kanga.kvack.org (Postfix) with ESMTP id 529F58D0023 for ; Mon, 25 Jan 2021 14:48:08 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1ABE28249980 for ; Mon, 25 Jan 2021 19:48:08 +0000 (UTC) X-FDA: 77745333456.06.chalk25_360abe227588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id E6DD31003CCEE for ; Mon, 25 Jan 2021 19:48:07 +0000 (UTC) X-HE-Tag: chalk25_360abe227588 X-Filterd-Recvd-Size: 6343 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:07 +0000 (UTC) Received: by mail-qv1-f45.google.com with SMTP id u16so4346404qvo.9 for ; Mon, 25 Jan 2021 11:48:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=W7ovzQ9DvHfyNSsR/jfLnSWR/h2NFzL1g+VyieOVil4=; b=UADMLlrIegNcGWIZww7E6o5AILPiEpQ+5GaHphe+4E8b2+SQFfZPbwUoVh2BV4aH3T H3zPy4Zk1mFIELuNlzORQ3hEUXyqOwWvQYdUUIsWVw+1q9nr4k1u4SnDWyLTLcaOoOM6 bGCvr0fgKVMxlTHmmaN7i8nGHw2cSNs7BJsKD6BYubZ5XrUjpVcIE/Kvbaoyb/ckQZQI W68bT5W+mia8ZNUjTEcEN/sT6LmSe8tJ2xU8hRTaeoGpQGct9aUwuVxo9Md5obaSb8et GiACS9VOC//9aGFRx++DunIreXKx8XvzMAWhCtholjcW5qZiMYqI5ybHRYA4SVGJArMd 1S+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W7ovzQ9DvHfyNSsR/jfLnSWR/h2NFzL1g+VyieOVil4=; b=rtl5h5wXa5ln01wetgkX21w0FJXSbIng0yGo2zH3bnpssJZonbeKU/UUfNBcya4OcK U/UGh4V0rZpCpM+1sL0Lz+u5o0MeKjNDGVi7+gZj8ZfMxHJMiK5uQIB6xUiHKM56x5Aw OI9iPSVGsv5lP5hJra1yM3JC1cvZ0qelzSvfltwAHFmdhkL0OiTWJyAVt6msXgVpz96w nyMmiMPYoPdE2oTeYnlJiMbj6mnnEG72igNxBFJQTHxrO59J33+2e8dPEM/84pbpYX3Q R//fBXbVjdHDllxj2wdlXn9Mr05WeY8WFU+9P2vZb5Ms6WvXA3RqK9isQSG3cALMbktR JZdA== X-Gm-Message-State: AOAM532yqAviyYGQ/qn+erIHhpW0DQzRezbEgH//V6OgV+B7xj9HlZwt k57onYkMeiHS4F6r8xa4ib4SXg== X-Google-Smtp-Source: ABdhPJzyqOavXJVtf9luNFZH6jMghueOYrdSnD4a7cqs/Vq/eAP0dRviO09WN0H4kKvXTXL5LyWi7A== X-Received: by 2002:a0c:a8c7:: with SMTP id h7mr2337061qvc.23.1611604086739; Mon, 25 Jan 2021 11:48:06 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:06 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 08/14] mm/gup: do not migrate zero page Date: Mon, 25 Jan 2021 14:47:45 -0500 Message-Id: <20210125194751.1275316-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On some platforms ZERO_PAGE(0) might end-up in a movable zone. Do not migrate zero page in gup during longterm pinning as migration of zero page is not allowed. For example, in x86 QEMU with 16G of memory and kernelcore=5G parameter, I see the following: Boot#1: zero_pfn 0x48a8d zero_pfn zone: ZONE_DMA32 Boot#2: zero_pfn 0x20168d zero_pfn zone: ZONE_MOVABLE On x86, empty_zero_page is declared in .bss and depending on the loader may end up in different physical locations during boots. Also, move is_zero_pfn() my_zero_pfn() functions under CONFIG_MMU, because zero_pfn that they are using is declared in memory.c which is compiled with CONFIG_MMU. Signed-off-by: Pavel Tatashin --- include/linux/mm.h | 3 ++- include/linux/mmzone.h | 4 ++++ include/linux/pgtable.h | 3 +-- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0990a76d5e6f..0295e074cd5d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1125,7 +1125,8 @@ static inline bool is_zone_movable_page(const struct page *page) /* MIGRATE_CMA and ZONE_MOVABLE do not allow pin pages */ static inline bool is_pinnable_page(struct page *page) { - return !is_zone_movable_page(page) && !is_migrate_cma_page(page); + return !(is_zone_movable_page(page) || is_migrate_cma_page(page)) + || is_zero_pfn(page_to_pfn(page)); } #ifdef CONFIG_DEV_PAGEMAP_OPS diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index ae588b2f87ef..72b0b6eba854 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -427,6 +427,10 @@ enum zone_type { * techniques might use alloc_contig_range() to hide previously * exposed pages from the buddy again (e.g., to implement some sort * of memory unplug in virtio-mem). + * 6. ZERO_PAGE(0), kernelcore/movablecore setups might create + * situations where ZERO_PAGE(0) which is allocated differently + * on different platforms may end up in a movable zone. ZERO_PAGE(0) + * cannot be migrated. * * In general, no unmovable allocations that degrade memory offlining * should end up in ZONE_MOVABLE. Allocators (like alloc_contig_range()) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index ea5c4102c23e..54a740602618 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1118,6 +1118,7 @@ extern void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, extern void untrack_pfn_moved(struct vm_area_struct *vma); #endif +#ifdef CONFIG_MMU #ifdef __HAVE_COLOR_ZERO_PAGE static inline int is_zero_pfn(unsigned long pfn) { @@ -1142,8 +1143,6 @@ static inline unsigned long my_zero_pfn(unsigned long addr) } #endif -#ifdef CONFIG_MMU - #ifndef CONFIG_TRANSPARENT_HUGEPAGE static inline int pmd_trans_huge(pmd_t pmd) { From patchwork Mon Jan 25 19:47:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24E2DC433DB for ; Mon, 25 Jan 2021 19:48:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A2BC7206D4 for ; Mon, 25 Jan 2021 19:48:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A2BC7206D4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7ABD48D0035; Mon, 25 Jan 2021 14:48:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73B738D0023; Mon, 25 Jan 2021 14:48:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58C338D0035; Mon, 25 Jan 2021 14:48:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0075.hostedemail.com [216.40.44.75]) by kanga.kvack.org (Postfix) with ESMTP id 3E5E58D0023 for ; Mon, 25 Jan 2021 14:48:10 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 00F63181AEF32 for ; Mon, 25 Jan 2021 19:48:10 +0000 (UTC) X-FDA: 77745333540.27.ocean25_031572d27588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id CF6B53D668 for ; Mon, 25 Jan 2021 19:48:09 +0000 (UTC) X-HE-Tag: ocean25_031572d27588 X-Filterd-Recvd-Size: 10991 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:09 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id z6so10630243qtn.0 for ; Mon, 25 Jan 2021 11:48:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bRb14bTDsR904QOnrllhSqwBPW3GMdN/JMUfm/Chs0c=; b=Yce291vTXFuFg6HWK5DarOVUy6VNEXMKKHc2/PWyfgcgBGHMArKI34zEdgbwX37TUk 4xsLIJ/cC3d4mCy2VB0kpIAfkmUYIm+0oJKuKM3F3/+pIIaSS87ulRYfQXS2oZpzadY9 U9IGU4bwhpmlcONxVTv9xmlCNONipQENehKmQr/s0/XqpWA5E3LdkepM3z6wX++r1z+K GqqKHEDee8X4bQka6QWROdGW6+BKhyFWzdMwUnNRRMR6Ck0GprvcjqdKvnMqNaM43B1Z 7cUjqpxN1FHUKMbWkgSt+8hhYD450JXF8PHABF772KbPluHkeVEQyMgPyAMAhn4710fA EqXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bRb14bTDsR904QOnrllhSqwBPW3GMdN/JMUfm/Chs0c=; b=J3cBPyQpGpGwgoJbLlLWa/kyEpE2VTdOfXHKstKs0B1nR7UG7TJsvU8SGKHgnoQKUn o1SsT+DhiRn65FBNhWWHRQUuum4ie4NneJdimjz3Nqx06ahoHgfrq9zFcyvbs03daoZW oMxsu6ov+sCoUCl07vsL5MZ8a11OSnPEnhVRP34UCn/Q6RlWoGqMVuoTeM8YMe51e+bU NijvXo206j19VIisjW0xnkkp1gITXIYP9321i8rFUT0XogGBDvZqHxA5C90qJscSOQj1 mA0aMJTYZ0LvrHm3IQKuh+Z07Np3cs5r9+IYbknOrohdz7DRV9tqSKqtwCdbGdrthJM9 XgJw== X-Gm-Message-State: AOAM531Eqoib+mRv+Xcs5nPugp8apsmRHni0johCgqq8vJB8XTprK6z2 ESe/uGPX5lOY2o3DUTc5q16JtQ== X-Google-Smtp-Source: ABdhPJzLax7absKUXwWSnNAImqALXsM7+m78dFnmv3NFCy3/ObclAzmJhIB1kY5AYoANPa+AwMgjxA== X-Received: by 2002:ac8:5c92:: with SMTP id r18mr1994203qta.27.1611604088519; Mon, 25 Jan 2021 11:48:08 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:07 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 09/14] mm/gup: migrate pinned pages out of movable zone Date: Mon, 25 Jan 2021 14:47:46 -0500 Message-Id: <20210125194751.1275316-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We should not pin pages in ZONE_MOVABLE. Currently, we do not pin only movable CMA pages. Generalize the function that migrates CMA pages to migrate all movable pages. Use is_pinnable_page() to check which pages need to be migrated Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- include/linux/migrate.h | 1 + include/linux/mmzone.h | 9 ++++- include/trace/events/migrate.h | 3 +- mm/gup.c | 67 +++++++++++++++++----------------- 4 files changed, 44 insertions(+), 36 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3a389633b68f..fdf65f23acec 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -27,6 +27,7 @@ enum migrate_reason { MR_MEMPOLICY_MBIND, MR_NUMA_MISPLACED, MR_CONTIG_RANGE, + MR_LONGTERM_PIN, MR_TYPES }; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 72b0b6eba854..300eb31439cb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -407,8 +407,13 @@ enum zone_type { * to increase the number of THP/huge pages. Notable special cases are: * * 1. Pinned pages: (long-term) pinning of movable pages might - * essentially turn such pages unmovable. Memory offlining might - * retry a long time. + * essentially turn such pages unmovable. Therefore, we do not allow + * pinning long-term pages in ZONE_MOVABLE. When pages are pinned and + * faulted, they come from the right zone right away. However, it is + * still possible that address space already has pages in + * ZONE_MOVABLE at the time when pages are pinned (i.e. user has + * touches that memory before pinning). In such case we migrate them + * to a different zone. When migration fails - pinning fails. * 2. memblock allocations: kernelcore/movablecore setups might create * situations where ZONE_MOVABLE contains unmovable allocations * after boot. Memory offlining and allocations fail early. diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h index 4d434398d64d..363b54ce104c 100644 --- a/include/trace/events/migrate.h +++ b/include/trace/events/migrate.h @@ -20,7 +20,8 @@ EM( MR_SYSCALL, "syscall_or_cpuset") \ EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ EM( MR_NUMA_MISPLACED, "numa_misplaced") \ - EMe(MR_CONTIG_RANGE, "contig_range") + EM( MR_CONTIG_RANGE, "contig_range") \ + EMe(MR_LONGTERM_PIN, "longterm_pin") /* * First define the enums in the above macros to be exported to userspace diff --git a/mm/gup.c b/mm/gup.c index 857b273e32ac..df29825305f8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -89,11 +89,12 @@ static __maybe_unused struct page *try_grab_compound_head(struct page *page, int orig_refs = refs; /* - * Can't do FOLL_LONGTERM + FOLL_PIN with CMA in the gup fast - * path, so fail and let the caller fall back to the slow path. + * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a + * right zone, so fail and let the caller fall back to the slow + * path. */ - if (unlikely(flags & FOLL_LONGTERM) && - is_migrate_cma_page(page)) + if (unlikely((flags & FOLL_LONGTERM) && + !is_pinnable_page(page))) return NULL; /* @@ -1547,17 +1548,17 @@ struct page *get_dump_page(unsigned long addr) } #endif /* CONFIG_ELF_CORE */ -#ifdef CONFIG_CMA -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) +#ifdef CONFIG_MIGRATION +static long check_and_migrate_movable_pages(struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int gup_flags) { unsigned long i, isolation_error_count; bool drain_allow; - LIST_HEAD(cma_page_list); + LIST_HEAD(movable_page_list); long ret = nr_pages; struct page *prev_head, *head; struct migration_target_control mtc = { @@ -1575,13 +1576,12 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, continue; prev_head = head; /* - * If we get a page from the CMA zone, since we are going to - * be pinning these entries, we might as well move them out - * of the CMA zone if possible. + * If we get a movable page, since we are going to be pinning + * these entries, try to move them out if possible. */ - if (is_migrate_cma_page(head)) { + if (!is_pinnable_page(head)) { if (PageHuge(head)) { - if (!isolate_huge_page(head, &cma_page_list)) + if (!isolate_huge_page(head, &movable_page_list)) isolation_error_count++; } else { if (!PageLRU(head) && drain_allow) { @@ -1593,7 +1593,7 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, isolation_error_count++; continue; } - list_add_tail(&head->lru, &cma_page_list); + list_add_tail(&head->lru, &movable_page_list); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), @@ -1606,10 +1606,10 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, * If list is empty, and no isolation errors, means that all pages are * in the correct zone. */ - if (list_empty(&cma_page_list) && !isolation_error_count) + if (list_empty(&movable_page_list) && !isolation_error_count) return ret; - if (!list_empty(&cma_page_list)) { + if (!list_empty(&movable_page_list)) { /* * drop the above get_user_pages reference. */ @@ -1619,12 +1619,12 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, for (i = 0; i < nr_pages; i++) put_page(pages[i]); - ret = migrate_pages(&cma_page_list, alloc_migration_target, + ret = migrate_pages(&movable_page_list, alloc_migration_target, NULL, (unsigned long)&mtc, MIGRATE_SYNC, - MR_CONTIG_RANGE); + MR_LONGTERM_PIN); if (ret) { - if (!list_empty(&cma_page_list)) - putback_movable_pages(&cma_page_list); + if (!list_empty(&movable_page_list)) + putback_movable_pages(&movable_page_list); return ret > 0 ? -ENOMEM : ret; } @@ -1643,16 +1643,16 @@ static long check_and_migrate_cma_pages(struct mm_struct *mm, goto check_again; } #else -static long check_and_migrate_cma_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, - struct page **pages, - struct vm_area_struct **vmas, - unsigned int gup_flags) +static long check_and_migrate_movable_pages(struct mm_struct *mm, + unsigned long start, + unsigned long nr_pages, + struct page **pages, + struct vm_area_struct **vmas, + unsigned int gup_flags) { return nr_pages; } -#endif /* CONFIG_CMA */ +#endif /* CONFIG_MIGRATION */ /* * __gup_longterm_locked() is a wrapper for __get_user_pages_locked which @@ -1676,8 +1676,9 @@ static long __gup_longterm_locked(struct mm_struct *mm, if (gup_flags & FOLL_LONGTERM) { if (rc > 0) - rc = check_and_migrate_cma_pages(mm, start, rc, pages, - vmas, gup_flags); + rc = check_and_migrate_movable_pages(mm, start, rc, + pages, vmas, + gup_flags); memalloc_pin_restore(flags); } return rc; From patchwork Mon Jan 25 19:47:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E668CC433E6 for ; Mon, 25 Jan 2021 19:48:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 86F17206D4 for ; Mon, 25 Jan 2021 19:48:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 86F17206D4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5FF048D0036; Mon, 25 Jan 2021 14:48:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 58F348D0023; Mon, 25 Jan 2021 14:48:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 148268D0036; Mon, 25 Jan 2021 14:48:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id ECF878D0023 for ; Mon, 25 Jan 2021 14:48:11 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B0094362D for ; Mon, 25 Jan 2021 19:48:11 +0000 (UTC) X-FDA: 77745333582.28.land57_1a0a0fd27588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 76D176C1A for ; Mon, 25 Jan 2021 19:48:11 +0000 (UTC) X-HE-Tag: land57_1a0a0fd27588 X-Filterd-Recvd-Size: 4856 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:10 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id r77so13527077qka.12 for ; Mon, 25 Jan 2021 11:48:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=z/iYHqVeCyWybAIwBASO/JRh5H6y4Wucaqn9XTIErdQ=; b=ggxWFJx1XNRIPR+VPqKxFscBPx27Vie4AlDDaPRS0mTDqH54+IBnC1q4/B/aQgVuDi bQy/G19+TUuwriBvOR3qKylUfMo8aEA/sssA638vsok6rLz29eNdH0tzpRF9FZDTm6kX ZsdNcrXEc2WtBcqRjG7i1UuM/BBdeNi5zMLDut663oqjTUQgWKlhgDWc+pPfNnMxcFXm MjAdHpC+E6DmvxGDfoGxz8OL0j4Me4jtxqBMlcaTW4Qf/OzbN0ovx1DD0h+fCBpfgNHe BIg25RJ3G4qs8V+VRvGN/b3dhnUFlfr+IUa9j+QLE+WIUZR8QUyTTEGUz3p7BPxl+vnk HlFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z/iYHqVeCyWybAIwBASO/JRh5H6y4Wucaqn9XTIErdQ=; b=VT/A4uo5b6z5CRgp2r6iNY7j/zzRuc9LH8cg5YQ9RprQ8ajKxLQI2Eqk/CsXW3vCQF NiUxaF9lbv8z8jZnP1r+eMPwfxrBT2jMqfNj8JUMAuqpX5ggb1cUpvRB/ncfoFQRGnPr ML1Gc6dBJTtAxO/K3/ZcF0E0Dqh/PFZD3RrtmYk4nLZQ33MkAjKvuEFTf721YE1bc/21 lPkKGw9uLiZcuwUlz9xZ4y5nlpKMU3OYu/gAsAN1G2QbdWw8pJnkoyaCCb2u6yTQg8w4 tKzur16XKw142a+JejOs8nv1LAlymep6gBmLtIhcNT+cTVY72gIvHJSoK4yXbA78aClr xDcQ== X-Gm-Message-State: AOAM5330OiKlzcxlYwSvpaPV464eYu6fzCl+lNJ3WKB84on6OJ2O/NUa OTCHHb3J7vwqv1L9JFw5dtm0Dg== X-Google-Smtp-Source: ABdhPJwHb2z3O87rhiHdk/7Xsu4L+A54QJcA/Onlzvq9WKtDY2/7cQjQseTRCIMa3SSoFW5xmC91oA== X-Received: by 2002:a05:620a:13ae:: with SMTP id m14mr2317266qki.35.1611604090257; Mon, 25 Jan 2021 11:48:10 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:09 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 10/14] memory-hotplug.rst: add a note about ZONE_MOVABLE and page pinning Date: Mon, 25 Jan 2021 14:47:47 -0500 Message-Id: <20210125194751.1275316-11-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Document the special handling of page pinning when ZONE_MOVABLE present. Signed-off-by: Pavel Tatashin Suggested-by: David Hildenbrand Acked-by: Michal Hocko --- Documentation/admin-guide/mm/memory-hotplug.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst index 5c4432c96c4b..c6618f99f765 100644 --- a/Documentation/admin-guide/mm/memory-hotplug.rst +++ b/Documentation/admin-guide/mm/memory-hotplug.rst @@ -357,6 +357,15 @@ creates ZONE_MOVABLE as following. Unfortunately, there is no information to show which memory block belongs to ZONE_MOVABLE. This is TBD. +.. note:: + Techniques that rely on long-term pinnings of memory (especially, RDMA and + vfio) are fundamentally problematic with ZONE_MOVABLE and, therefore, memory + hot remove. Pinned pages cannot reside on ZONE_MOVABLE, to guarantee that + memory can still get hot removed - be aware that pinning can fail even if + there is plenty of free memory in ZONE_MOVABLE. In addition, using + ZONE_MOVABLE might make page pinning more expensive, because pages have to be + migrated off that zone first. + .. _memory_hotplug_how_to_offline_memory: How to offline memory From patchwork Mon Jan 25 19:47:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F02DFC433E0 for ; Mon, 25 Jan 2021 19:48:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D5A120708 for ; Mon, 25 Jan 2021 19:48:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D5A120708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 938D08D0037; Mon, 25 Jan 2021 14:48:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BC3D8D0023; Mon, 25 Jan 2021 14:48:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EDA18D0037; Mon, 25 Jan 2021 14:48:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0225.hostedemail.com [216.40.44.225]) by kanga.kvack.org (Postfix) with ESMTP id 4A33B8D0023 for ; Mon, 25 Jan 2021 14:48:14 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0AFA61EE6 for ; Mon, 25 Jan 2021 19:48:14 +0000 (UTC) X-FDA: 77745333708.16.slip29_5a12c9f27588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id D76FD100E6932 for ; Mon, 25 Jan 2021 19:48:12 +0000 (UTC) X-HE-Tag: slip29_5a12c9f27588 X-Filterd-Recvd-Size: 4282 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:12 +0000 (UTC) Received: by mail-qt1-f181.google.com with SMTP id t14so6974380qto.8 for ; Mon, 25 Jan 2021 11:48:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=mjWbyN3QlCSRxacF3EdFD9Q6Qqvve11XewDvx8+hMpo=; b=QyqCNtzOZjM+ofJjxKN6Zxgsv1aotNfM8QD9SAxjhV1OH52yIpl6YtH/AqGv+QUzGh LbI7sONzqoiweiAtLaCZJ2f3bi2JeBjwuemwTo+nxdrLnJgeJftszsKtGaaVvnvOmZq/ Pf6Smrxlzj3DHcnp9U+v4hb5LnxPVq1o8gPuiZl8h15rkU5kyPWYd1kgA3hKdi1qxrRY 54IAZGiME/nPd2KM6TzR8sC9B+rRdaWe0pHCxG5CVbXJN0cA0ROp3dQCrIX0pEfhTO1U EANQarLiJ/JrKdo+vZGT8pU5m8TS/BBpC7lOvL/B7bw2ovLArcwG8gDmZpAhNNKiAYNn 4KCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mjWbyN3QlCSRxacF3EdFD9Q6Qqvve11XewDvx8+hMpo=; b=mVe9OWmrRDAqrb0hfIFp3gvgvGytvXQW0YEsFCDCzBPtbqiFWCGZY56QgX6FTPCQ8C V2NOobHgH+D31pN0TMj43m9pMVa1EtXFcerP+Kg8ODx4QdbkCCRkXgwWVuuPTgpgLEBv yzLN5u3TnE71CjH/Bb541j5xcsSDhCk9Cqx//Z2KLQF4pAT7DTT3ta/FXOEvtGMmt2S0 71mhWiTF23J5l5ZazNyhMW4DKZD2RhlEVNFGUiteECVBxzALkJgeEQKggQX56mQWbkMI y2MzxtgKvoC/OtQGfEpDhojvBXHrCdKB4DINUj4QID3x/fASsOIgOtAuDSAjsdB9cUlm o3lg== X-Gm-Message-State: AOAM532fbOD2Z124BL+DVCR/hdX24iEaQOVpjr6qoGPEx3TzRE491azx Xk/Vx57xNNLkYHDdhHquEBpBvQ== X-Google-Smtp-Source: ABdhPJyeXbEGvlROSYzc12iqp04Fh8xESSSyL4/yMP4LQTF+v2sZ3EQkzQM1HBNIDwwwu17OzD/NHA== X-Received: by 2002:a05:622a:102:: with SMTP id u2mr2061298qtw.37.1611604091925; Mon, 25 Jan 2021 11:48:11 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:11 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 11/14] mm/gup: change index type to long as it counts pages Date: Mon, 25 Jan 2021 14:47:48 -0500 Message-Id: <20210125194751.1275316-12-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In __get_user_pages_locked() i counts number of pages which should be long, as long is used in all other places to contain number of pages, and 32-bit becomes increasingly small for handling page count proportional values. Signed-off-by: Pavel Tatashin Acked-by: Michal Hocko --- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index df29825305f8..f98af75dab0f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1479,7 +1479,7 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, { struct vm_area_struct *vma; unsigned long vm_flags; - int i; + long i; /* calculate required read or write permissions. * If FOLL_FORCE is set, we only require the "MAY" flags. From patchwork Mon Jan 25 19:47:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 250ECC433DB for ; Mon, 25 Jan 2021 19:48:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 83598206D4 for ; Mon, 25 Jan 2021 19:48:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 83598206D4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 624D18D0038; Mon, 25 Jan 2021 14:48:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 589F28D0023; Mon, 25 Jan 2021 14:48:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 401F68D0038; Mon, 25 Jan 2021 14:48:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0065.hostedemail.com [216.40.44.65]) by kanga.kvack.org (Postfix) with ESMTP id 2425A8D0023 for ; Mon, 25 Jan 2021 14:48:15 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D14798249980 for ; Mon, 25 Jan 2021 19:48:14 +0000 (UTC) X-FDA: 77745333708.15.able33_400f4fd27588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id AB55D1814B0CC for ; Mon, 25 Jan 2021 19:48:14 +0000 (UTC) X-HE-Tag: able33_400f4fd27588 X-Filterd-Recvd-Size: 8267 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:14 +0000 (UTC) Received: by mail-qk1-f173.google.com with SMTP id a12so3796959qkh.10 for ; Mon, 25 Jan 2021 11:48:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=IY6D8/ZMyGIUDKqllv86b5qOcyIRLy633LCHCtaaquY=; b=B2YCL2rAyFoZqE4WT/4QUlTLv0sbMnioKPrp1NGkhpM4TfBDRU0nz+ueHoSh2qqzzH Y3ws3+Lgh9rdkYaAlQ0tux2bwasCXVlUL3V6AoyITO940H+MX59ufzy3ves/bKkzEpvP dJ4LorhjJ+Tiu2GN+FnEl5uMAkdvdCygd1cYuM9fDokHwsb3q1/9yKgL9Xcc8Ec8YcLh 5bmOyEKXP0SCtISkhaByU7tvRgoJVo1Re+PRCO8ka11ZRNARhNU+xkumGzdAt6VKyJgt dPfcQqFEQF6Q9xhV87BVtimH/vmyY9cumqyNJZz5/5Roj8V4Q7adw/cXIEwnqMCwdzM1 9x3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IY6D8/ZMyGIUDKqllv86b5qOcyIRLy633LCHCtaaquY=; b=Trt3ouHHfK2l0EMNU+7DgNryKoGtp4I4/Vc4YjsGqFIHtfsHBm1fMXP5y2bLCtG0j7 IpFFYeiY7XjN/MEQNHFdcc73IXFNVLACvuyorOrX4+jbP/eMmvhUqp4+/zVEM/Z2PJFL o7JNTy4Isex9rQBOGJouK1nEExg/KYiBpI2DOHbHQjd8d3jVcVs4IHKnC2PCNUW06tc2 i981DZLnMVm3qM6KIeYt5eGOputDbs7q7w3l9w4ABGsD2daA7KGvsiwhKq2w170vevAL zkqMDOY+4X8UDdjQnHqO6Gg4ERU3eW7XMFPirk7bU3GiWS7KjIFQCskSgSPFdPbG+Dj+ 3AFg== X-Gm-Message-State: AOAM532QNj7yktPTdzh1jZDQj6DIH5TJ7ohakFF10CUH4XMlLe/QSDY8 Ky1l+nNRJ77vq7unBuqdKJT+dw== X-Google-Smtp-Source: ABdhPJytYypa/BCyR6Vkx3Q9a3yFIqDeeC4LQvsoMCNCusgTFdSKwLYE8QhUdAdv2IOy5rLuLeHXUQ== X-Received: by 2002:a37:a7d1:: with SMTP id q200mr2444716qke.365.1611604093523; Mon, 25 Jan 2021 11:48:13 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:13 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 12/14] mm/gup: longterm pin migration cleanup Date: Mon, 25 Jan 2021 14:47:49 -0500 Message-Id: <20210125194751.1275316-13-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When pages are longterm pinned, we must migrated them out of movable zone. The function that migrates them has a hidden loop with goto. The loop is to retry on isolation failures, and after successful migration. Make this code better by moving this loop to the caller. Signed-off-by: Pavel Tatashin Reviewed-by: Jason Gunthorpe --- mm/gup.c | 93 ++++++++++++++++++++++---------------------------------- 1 file changed, 37 insertions(+), 56 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index f98af75dab0f..fabfe2a5c627 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1549,27 +1549,28 @@ struct page *get_dump_page(unsigned long addr) #endif /* CONFIG_ELF_CORE */ #ifdef CONFIG_MIGRATION -static long check_and_migrate_movable_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, +/* + * Check whether all pages are pinnable, if so return number of pages. If some + * pages are not pinnable, migrate them, and unpin all pages. Return zero if + * pages were migrated, or if some pages were not successfully isolated. + * Return negative error if migration fails. + */ +static long check_and_migrate_movable_pages(unsigned long nr_pages, struct page **pages, - struct vm_area_struct **vmas, unsigned int gup_flags) { - unsigned long i, isolation_error_count; - bool drain_allow; + unsigned long i; + unsigned long isolation_error_count = 0; + bool drain_allow = true; LIST_HEAD(movable_page_list); - long ret = nr_pages; - struct page *prev_head, *head; + long ret = 0; + struct page *prev_head = NULL; + struct page *head; struct migration_target_control mtc = { .nid = NUMA_NO_NODE, .gfp_mask = GFP_USER | __GFP_NOWARN, }; -check_again: - prev_head = NULL; - isolation_error_count = 0; - drain_allow = true; for (i = 0; i < nr_pages; i++) { head = compound_head(pages[i]); if (head == prev_head) @@ -1607,47 +1608,27 @@ static long check_and_migrate_movable_pages(struct mm_struct *mm, * in the correct zone. */ if (list_empty(&movable_page_list) && !isolation_error_count) - return ret; + return nr_pages; + if (gup_flags & FOLL_PIN) { + unpin_user_pages(pages, nr_pages); + } else { + for (i = 0; i < nr_pages; i++) + put_page(pages[i]); + } if (!list_empty(&movable_page_list)) { - /* - * drop the above get_user_pages reference. - */ - if (gup_flags & FOLL_PIN) - unpin_user_pages(pages, nr_pages); - else - for (i = 0; i < nr_pages; i++) - put_page(pages[i]); - ret = migrate_pages(&movable_page_list, alloc_migration_target, NULL, (unsigned long)&mtc, MIGRATE_SYNC, MR_LONGTERM_PIN); - if (ret) { - if (!list_empty(&movable_page_list)) - putback_movable_pages(&movable_page_list); - return ret > 0 ? -ENOMEM : ret; - } - - /* We unpinned pages before migration, pin them again */ - ret = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, - NULL, gup_flags); - if (ret <= 0) - return ret; - nr_pages = ret; + if (ret && !list_empty(&movable_page_list)) + putback_movable_pages(&movable_page_list); } - /* - * check again because pages were unpinned, and we also might have - * had isolation errors and need more pages to migrate. - */ - goto check_again; + return ret > 0 ? -ENOMEM : ret; } #else -static long check_and_migrate_movable_pages(struct mm_struct *mm, - unsigned long start, - unsigned long nr_pages, +static long check_and_migrate_movable_pages(unsigned long nr_pages, struct page **pages, - struct vm_area_struct **vmas, unsigned int gup_flags) { return nr_pages; @@ -1665,22 +1646,22 @@ static long __gup_longterm_locked(struct mm_struct *mm, struct vm_area_struct **vmas, unsigned int gup_flags) { - unsigned long flags = 0; + unsigned int flags; long rc; - if (gup_flags & FOLL_LONGTERM) - flags = memalloc_pin_save(); - - rc = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, NULL, - gup_flags); + if (!(gup_flags & FOLL_LONGTERM)) + return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + NULL, gup_flags); + flags = memalloc_pin_save(); + do { + rc = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + NULL, gup_flags); + if (rc <= 0) + break; + rc = check_and_migrate_movable_pages(rc, pages, gup_flags); + } while (!rc); + memalloc_pin_restore(flags); - if (gup_flags & FOLL_LONGTERM) { - if (rc > 0) - rc = check_and_migrate_movable_pages(mm, start, rc, - pages, vmas, - gup_flags); - memalloc_pin_restore(flags); - } return rc; } From patchwork Mon Jan 25 19:47:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D08C2C433E0 for ; Mon, 25 Jan 2021 19:48:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 51C4320708 for ; Mon, 25 Jan 2021 19:48:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 51C4320708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E43D18D003B; Mon, 25 Jan 2021 14:48:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE0178D0023; Mon, 25 Jan 2021 14:48:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA5B88D003B; Mon, 25 Jan 2021 14:48:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id A4A4F8D0023 for ; Mon, 25 Jan 2021 14:48:16 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 64899180AD807 for ; Mon, 25 Jan 2021 19:48:16 +0000 (UTC) X-FDA: 77745333792.06.pies83_210527727588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 3AD071003CCF7 for ; Mon, 25 Jan 2021 19:48:16 +0000 (UTC) X-HE-Tag: pies83_210527727588 X-Filterd-Recvd-Size: 10213 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:15 +0000 (UTC) Received: by mail-qk1-f174.google.com with SMTP id n15so9049276qkh.8 for ; Mon, 25 Jan 2021 11:48:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Ca95TMJT4ShVUsz7drjIPNjU47Yiarr1AByU+uGNnGI=; b=i1J3Ss8NvulKFTVXkwnvE/0omlR6s+5Yp2oyhLYvWL1INoRa37fVAIs1AMaQKDsfeo PdOXcP0OmEJk5OEzyJ4GEECDUpV9nkoJx2Lq1cPqNB/ujBg7sIZe2Vvbm4rBxrTwlqO4 NS5GnxoxFJCKgp1JOCIEZYrPXWpwSFM51aML5h4z3blrZC4uO+biRbYG77176iqVVQZ2 xocUTbIap3U5TJsxD1GtaNlXIlrNlb78bxuTmryelGz3or93XhHLY7jGVL2GWYvC+48v rBjYYe6Lfx9+WDb32fmcQq8FTzvKFTjozj7lUND+NYf2RdIU/2uEuMg6ihchuUBzJwmU 7ghA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ca95TMJT4ShVUsz7drjIPNjU47Yiarr1AByU+uGNnGI=; b=HQ0QzeBfBvkph6dBViIkfGX4T+31FZGGneyATFq7DESt00MLDNSs0ORtD8llr9eblN pW4TvlPzJeYU4iQiyEMrpy9AowPkOEnKn8al60iPKYqfLhfvNKqADqkfgWIIv5nilIFq tAiBIkBZJO0nuA+bcfvv0zxCg8W/Veg0X9r7X5DVrM9JWciDAQmJymkBAqDfIxHGq98x fuoyRS/IFQ9tbJc+t+EJ9UCIpEu9hoTjh+f6ZidXU7h4uTgQGqFALP7sMZmS4J+kgyi2 eMvcyiRCRY07Xw9q6AckyDq8LFbMXafOco/gH8j+w0UelKtA8oCGwIARalR7ZFsZQpe8 xOXg== X-Gm-Message-State: AOAM531GWKGvPaznRHP02BE8qZudw1KtOWGLaLhJktDSlCvQlG8sn4pQ haPCFe9zMVvKagCwlgyWrK76sA== X-Google-Smtp-Source: ABdhPJwhUd5EAmSYbcYIodK1SuK4RYyZuXxxxO7P4gD+3kK2D9MfxRx4+Ernng3+tzKIsO21XppT0w== X-Received: by 2002:a05:620a:b03:: with SMTP id t3mr2437218qkg.459.1611604095164; Mon, 25 Jan 2021 11:48:15 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:14 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 13/14] selftests/vm: gup_test: fix test flag Date: Mon, 25 Jan 2021 14:47:50 -0500 Message-Id: <20210125194751.1275316-14-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In gup_test both gup_flags and test_flags use the same flags field. This is broken. Farther, in the actual gup_test.c all the passed gup_flags are erased and unconditionally replaced with FOLL_WRITE. Which means that test_flags are ignored, and code like this always performs pin dump test: 155 if (gup->flags & GUP_TEST_FLAG_DUMP_PAGES_USE_PIN) 156 nr = pin_user_pages(addr, nr, gup->flags, 157 pages + i, NULL); 158 else 159 nr = get_user_pages(addr, nr, gup->flags, 160 pages + i, NULL); 161 break; Add a new test_flags field, to allow raw gup_flags to work. Add a new subcommand for DUMP_USER_PAGES_TEST to specify that pin test should be performed. Remove unconditional overwriting of gup_flags via FOLL_WRITE. But, preserve the previous behaviour where FOLL_WRITE was the default flag, and add a new option "-W" to unset FOLL_WRITE. Rename flags with gup_flags. With the fix, dump works like this: root@virtme:/# gup_test -c ---- page #0, starting from user virt addr: 0x7f8acb9e4000 page:00000000d3d2ee27 refcount:2 mapcount:1 mapping:0000000000000000 index:0x0 pfn:0x100bcf anon flags: 0x300000000080016(referenced|uptodate|lru|swapbacked) raw: 0300000000080016 ffffd0e204021608 ffffd0e208df2e88 ffff8ea04243ec61 raw: 0000000000000000 0000000000000000 0000000200000000 0000000000000000 page dumped because: gup_test: dump_pages() test DUMP_USER_PAGES_TEST: done root@virtme:/# gup_test -c -p ---- page #0, starting from user virt addr: 0x7fd19701b000 page:00000000baed3c7d refcount:1025 mapcount:1 mapping:0000000000000000 index:0x0 pfn:0x108008 anon flags: 0x300000000080014(uptodate|lru|swapbacked) raw: 0300000000080014 ffffd0e204200188 ffffd0e205e09088 ffff8ea04243ee71 raw: 0000000000000000 0000000000000000 0000040100000000 0000000000000000 page dumped because: gup_test: dump_pages() test DUMP_USER_PAGES_TEST: done Refcount shows the difference between pin vs no-pin case. Also change type of nr from int to long, as it counts number of pages. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- mm/gup_test.c | 23 ++++++++++------------- mm/gup_test.h | 3 ++- tools/testing/selftests/vm/gup_test.c | 15 +++++++++++---- 3 files changed, 23 insertions(+), 18 deletions(-) diff --git a/mm/gup_test.c b/mm/gup_test.c index e3cf78e5873e..a6ed1c877679 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -94,7 +94,7 @@ static int __gup_test_ioctl(unsigned int cmd, { ktime_t start_time, end_time; unsigned long i, nr_pages, addr, next; - int nr; + long nr; struct page **pages; int ret = 0; bool needs_mmap_lock = @@ -126,37 +126,34 @@ static int __gup_test_ioctl(unsigned int cmd, nr = (next - addr) / PAGE_SIZE; } - /* Filter out most gup flags: only allow a tiny subset here: */ - gup->flags &= FOLL_WRITE; - switch (cmd) { case GUP_FAST_BENCHMARK: - nr = get_user_pages_fast(addr, nr, gup->flags, + nr = get_user_pages_fast(addr, nr, gup->gup_flags, pages + i); break; case GUP_BASIC_TEST: - nr = get_user_pages(addr, nr, gup->flags, pages + i, + nr = get_user_pages(addr, nr, gup->gup_flags, pages + i, NULL); break; case PIN_FAST_BENCHMARK: - nr = pin_user_pages_fast(addr, nr, gup->flags, + nr = pin_user_pages_fast(addr, nr, gup->gup_flags, pages + i); break; case PIN_BASIC_TEST: - nr = pin_user_pages(addr, nr, gup->flags, pages + i, + nr = pin_user_pages(addr, nr, gup->gup_flags, pages + i, NULL); break; case PIN_LONGTERM_BENCHMARK: nr = pin_user_pages(addr, nr, - gup->flags | FOLL_LONGTERM, + gup->gup_flags | FOLL_LONGTERM, pages + i, NULL); break; case DUMP_USER_PAGES_TEST: - if (gup->flags & GUP_TEST_FLAG_DUMP_PAGES_USE_PIN) - nr = pin_user_pages(addr, nr, gup->flags, + if (gup->test_flags & GUP_TEST_FLAG_DUMP_PAGES_USE_PIN) + nr = pin_user_pages(addr, nr, gup->gup_flags, pages + i, NULL); else - nr = get_user_pages(addr, nr, gup->flags, + nr = get_user_pages(addr, nr, gup->gup_flags, pages + i, NULL); break; default: @@ -187,7 +184,7 @@ static int __gup_test_ioctl(unsigned int cmd, start_time = ktime_get(); - put_back_pages(cmd, pages, nr_pages, gup->flags); + put_back_pages(cmd, pages, nr_pages, gup->test_flags); end_time = ktime_get(); gup->put_delta_usec = ktime_us_delta(end_time, start_time); diff --git a/mm/gup_test.h b/mm/gup_test.h index 90a6713d50eb..887ac1d5f5bc 100644 --- a/mm/gup_test.h +++ b/mm/gup_test.h @@ -21,7 +21,8 @@ struct gup_test { __u64 addr; __u64 size; __u32 nr_pages_per_call; - __u32 flags; + __u32 gup_flags; + __u32 test_flags; /* * Each non-zero entry is the number of the page (1-based: first page is * page 1, so that zero entries mean "do nothing") from the .addr base. diff --git a/tools/testing/selftests/vm/gup_test.c b/tools/testing/selftests/vm/gup_test.c index 6c6336dd3b7f..943cc2608dc2 100644 --- a/tools/testing/selftests/vm/gup_test.c +++ b/tools/testing/selftests/vm/gup_test.c @@ -37,13 +37,13 @@ int main(int argc, char **argv) { struct gup_test gup = { 0 }; unsigned long size = 128 * MB; - int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 0; + int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 1; unsigned long cmd = GUP_FAST_BENCHMARK; int flags = MAP_PRIVATE; char *file = "/dev/zero"; char *p; - while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwSH")) != -1) { + while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHp")) != -1) { switch (opt) { case 'a': cmd = PIN_FAST_BENCHMARK; @@ -65,9 +65,13 @@ int main(int argc, char **argv) */ gup.which_pages[0] = 1; break; + case 'p': + /* works only with DUMP_USER_PAGES_TEST */ + gup.test_flags |= GUP_TEST_FLAG_DUMP_PAGES_USE_PIN; + break; case 'F': /* strtol, so you can pass flags in hex form */ - gup.flags = strtol(optarg, 0, 0); + gup.gup_flags = strtol(optarg, 0, 0); break; case 'm': size = atoi(optarg) * MB; @@ -93,6 +97,9 @@ int main(int argc, char **argv) case 'w': write = 1; break; + case 'W': + write = 0; + break; case 'f': file = optarg; break; @@ -140,7 +147,7 @@ int main(int argc, char **argv) gup.nr_pages_per_call = nr_pages; if (write) - gup.flags |= FOLL_WRITE; + gup.gup_flags |= FOLL_WRITE; fd = open("/sys/kernel/debug/gup_test", O_RDWR); if (fd == -1) { From patchwork Mon Jan 25 19:47:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12044257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44112C433DB for ; Mon, 25 Jan 2021 19:48:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D3E5620708 for ; Mon, 25 Jan 2021 19:48:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3E5620708 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9F5D88D0040; Mon, 25 Jan 2021 14:48:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A7C68D0023; Mon, 25 Jan 2021 14:48:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7872D8D0040; Mon, 25 Jan 2021 14:48:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0031.hostedemail.com [216.40.44.31]) by kanga.kvack.org (Postfix) with ESMTP id 5DE008D0023 for ; Mon, 25 Jan 2021 14:48:18 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 285D68249980 for ; Mon, 25 Jan 2021 19:48:18 +0000 (UTC) X-FDA: 77745333876.11.bite78_5d0475527588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id DCD91180F8B92 for ; Mon, 25 Jan 2021 19:48:17 +0000 (UTC) X-HE-Tag: bite78_5d0475527588 X-Filterd-Recvd-Size: 7099 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:48:17 +0000 (UTC) Received: by mail-qk1-f174.google.com with SMTP id n7so2411126qkc.4 for ; Mon, 25 Jan 2021 11:48:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=9bnQ1cYpKSogma2d9jSskcM/kWG+ieYizroT/wf+fYw=; b=SbFXkwHCcoJpnp3pSh+U+6dAEB5fNp4W6yYJzWpDQ2OxM4IV0JCl+/ZnQ3BxeLrKH7 dgSLcul/DjwmdcLD2+cBU2tjGAHgVmGjD7gTz2lEXgnZuAg3VfjHI3ymQ0780T3H9i8C MvZSPDRaNt/VBDO7iKzpiVlorxOyMkS/LQUFkidHyCvLA6jtmtCK+svAOB2xKkgqqQck Pgy4QOY3TY8GmkX4/f+cAwFmOtL0ElcJEBWKNMNCLvSqVEeDkvcz4yYez4nxTtPKzJ7f YM6MyXdx9c1/4uihA1f0UOlOf6gDbN9KrKRxoHAr91DQOlPpppUCc65/yMCMEFRBCTpT I1oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9bnQ1cYpKSogma2d9jSskcM/kWG+ieYizroT/wf+fYw=; b=YOY48j2fFjtw1tZf4GnFidRHpckS6ADjnUyQBegg8K0Z6uKaFj40LX0fqSWh7+0+a2 Zmx9Y2CBz0/8NJ+gy8MLZx1/EOcuNkEqpHlZk8I3rtwNRStFLhpjkNw0v+kzrhIOEZuR 4DQAPDPcuWt0cLb7GzmaGc0UFZunXJyFOvIbgD9WL9tULMOfISMM6GlU8tBD2UIBTPMr Zw9aN8yoMlzaZXir3AoeS6yQy2+aNBQ/VtoQ7tGGh48RZ7EEVcEV2XX7xFT0GU6wQ4Wh AvMauA2OhRhmIa2Z1IMny5AaWdxld2V/KTKYwrU0XhRYbSIfwnpgVQkYL2ECsxxAYckB /oWA== X-Gm-Message-State: AOAM533e0XGq6b+di64MPi7qguh+zIp9LE0onyeKixunodyHqqfmgXyu bGqx03n/f4Jy1nbXCChNTXB3alacQ4VRbQ== X-Google-Smtp-Source: ABdhPJxfjv7ObuguLsaXbUEv43CLPW8iNfijHOlMaUxjKGwAIxhp+wxrf0HI6Y+/uMKENQ0ADNF3oA== X-Received: by 2002:a37:5d07:: with SMTP id r7mr2402284qkb.335.1611604096795; Mon, 25 Jan 2021 11:48:16 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id c12sm12121569qtq.76.2021.01.25.11.48.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:48:16 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org, ira.weiny@intel.com, linux-kselftest@vger.kernel.org, jmorris@namei.org Subject: [PATCH v8 14/14] selftests/vm: gup_test: test faulting in kernel, and verify pinnable pages Date: Mon, 25 Jan 2021 14:47:51 -0500 Message-Id: <20210125194751.1275316-15-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125194751.1275316-1-pasha.tatashin@soleen.com> References: <20210125194751.1275316-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When pages are pinned they can be faulted in userland and migrated, and they can be faulted right in kernel without migration. In either case, the pinned pages must end-up being pinnable (not movable). Add a new test to gup_test, to help verify that the gup/pup (get_user_pages() / pin_user_pages()) behavior with respect to pinnable and movable pages is reasonable and correct. Specifically, provide a way to: 1) Verify that only "pinnable" pages are pinned. This is checked automatically for you. 2) Verify that gup/pup performance is reasonable. This requires comparing benchmarks between doing gup/pup on pages that have been pre-faulted in from user space, vs. doing gup/pup on pages that are not faulted in until gup/pup time (via FOLL_TOUCH). This decision is controlled with the new -z command line option. Signed-off-by: Pavel Tatashin Reviewed-by: John Hubbard --- mm/gup_test.c | 6 ++++++ tools/testing/selftests/vm/gup_test.c | 23 +++++++++++++++++++---- 2 files changed, 25 insertions(+), 4 deletions(-) diff --git a/mm/gup_test.c b/mm/gup_test.c index a6ed1c877679..d974dec19e1c 100644 --- a/mm/gup_test.c +++ b/mm/gup_test.c @@ -52,6 +52,12 @@ static void verify_dma_pinned(unsigned int cmd, struct page **pages, dump_page(page, "gup_test failure"); break; + } else if (cmd == PIN_LONGTERM_BENCHMARK && + WARN(!is_pinnable_page(page), + "pages[%lu] is NOT pinnable but pinned\n", + i)) { + dump_page(page, "gup_test failure"); + break; } } break; diff --git a/tools/testing/selftests/vm/gup_test.c b/tools/testing/selftests/vm/gup_test.c index 943cc2608dc2..1e662d59c502 100644 --- a/tools/testing/selftests/vm/gup_test.c +++ b/tools/testing/selftests/vm/gup_test.c @@ -13,6 +13,7 @@ /* Just the flags we need, copied from mm.h: */ #define FOLL_WRITE 0x01 /* check pte is writable */ +#define FOLL_TOUCH 0x02 /* mark page accessed */ static char *cmd_to_str(unsigned long cmd) { @@ -39,11 +40,11 @@ int main(int argc, char **argv) unsigned long size = 128 * MB; int i, fd, filed, opt, nr_pages = 1, thp = -1, repeats = 1, write = 1; unsigned long cmd = GUP_FAST_BENCHMARK; - int flags = MAP_PRIVATE; + int flags = MAP_PRIVATE, touch = 0; char *file = "/dev/zero"; char *p; - while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHp")) != -1) { + while ((opt = getopt(argc, argv, "m:r:n:F:f:abctTLUuwWSHpz")) != -1) { switch (opt) { case 'a': cmd = PIN_FAST_BENCHMARK; @@ -110,6 +111,10 @@ int main(int argc, char **argv) case 'H': flags |= (MAP_HUGETLB | MAP_ANONYMOUS); break; + case 'z': + /* fault pages in gup, do not fault in userland */ + touch = 1; + break; default: return -1; } @@ -167,8 +172,18 @@ int main(int argc, char **argv) else if (thp == 0) madvise(p, size, MADV_NOHUGEPAGE); - for (; (unsigned long)p < gup.addr + size; p += PAGE_SIZE) - p[0] = 0; + /* + * FOLL_TOUCH, in gup_test, is used as an either/or case: either + * fault pages in from the kernel via FOLL_TOUCH, or fault them + * in here, from user space. This allows comparison of performance + * between those two cases. + */ + if (touch) { + gup.gup_flags |= FOLL_TOUCH; + } else { + for (; (unsigned long)p < gup.addr + size; p += PAGE_SIZE) + p[0] = 0; + } /* Only report timing information on the *_BENCHMARK commands: */ if ((cmd == PIN_FAST_BENCHMARK) || (cmd == GUP_FAST_BENCHMARK) ||