From patchwork Sun Sep 26 16:12:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80B71C433FE for ; Sun, 26 Sep 2021 23:43:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E646D6115B for ; Sun, 26 Sep 2021 23:43:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E646D6115B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 75CD16B0074; Sun, 26 Sep 2021 19:43:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70D546B0075; Sun, 26 Sep 2021 19:43:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56180900002; Sun, 26 Sep 2021 19:43:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id 44EEA6B0074 for ; Sun, 26 Sep 2021 19:43:57 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id F163F1803E8EC for ; Sun, 26 Sep 2021 23:43:56 +0000 (UTC) X-FDA: 78631354872.01.A34FFAC Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf23.hostedemail.com (Postfix) with ESMTP id B413A90000A9 for ; Sun, 26 Sep 2021 23:43:56 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id bj3-20020a17090b088300b0019e6603fe89so10420112pjb.4 for ; Sun, 26 Sep 2021 16:43:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=u81btMS2EcfvD4xOl+t93Wlb5o4OTa3kZuiyGFUFTF8=; b=qlSgKJdiolKmSwykOFnZi1/XJzd8aFrG+DROd106Q+ECW3QKD8Nq+JIZs8Kg3z48Mr IbhFU/F89Xivj9Bx6lJAEbcB3aT43JnnsQ77qFheBfb327hb/4+YfPwwoLkE9k6uO6Yy MPmRD7YGyItvx1v3mFfgOwrhqpQZz29UX8QzsgwyXn4LjLMd09NaIqZMjpvqIjxI2PWJ ggPv9KW7lLO5emIWBrnA72c3UMPZI8FfSgv0wnfj9/Es0Eq2YXSyx6jHHYacjXMc4W9m IdNYXLqyHdM/EVBPqxKwJA7VKxShkNFbjn2WegVHGy43UoLkXNjzxjX4aYt3wMBTIlgu jO0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=u81btMS2EcfvD4xOl+t93Wlb5o4OTa3kZuiyGFUFTF8=; b=bnGh1FfkjoW6pXBM3F+SK9uBiS1vHQMW6TMZumFFhq3ft+gpahpbTRvSz5JNN6QY6+ E0xpdF+M6L3YgKOfZ3Fk+haIqN30Dq74/ZhKG9WqGrKCA5DQwyN/pqzIkf7S5a2f66Yy 9AlCe9BKdd3kvw8ztYyNgd8OVLJm1fi0rcVSHimKsMHaSEgBzRsHlhgU/j+8gUpmR9lO dPaSs3wRCidqrUzprEveO0m3NRkqbq1AX478q1LvIwTsUChi7DiXDm7ssn8P93WS8V4X mD6dc66L0HrFDBuzfttHT40WdF1bOGzeMZzD30UaEsGoNbCAS1qiPmf7UiMMPc1uUkV0 8f1g== X-Gm-Message-State: AOAM532Bb11klhbHQZAJTQ7+uuVLD2dmdgpydci4JGwFQ4q4C+AVnl0O e8EAPULCHkXuiLpA0pZKl1U= X-Google-Smtp-Source: ABdhPJzKPnFqZTi36uFRL2WPagr7zdZSK/rhjNj1N5ymsyTwLG9xuGKRrMH+uCQAJVk6eUR4M5SOEA== X-Received: by 2002:a17:90a:e003:: with SMTP id u3mr16397472pjy.137.1632699835568; Sun, 26 Sep 2021 16:43:55 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.43.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:43:55 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 1/8] mm/madvise: propagate vma->vm_end changes Date: Sun, 26 Sep 2021 09:12:52 -0700 Message-Id: <20210926161259.238054-2-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B413A90000A9 X-Stat-Signature: zat7d4a3dd86mhy4rafcm9m3prisa6x3 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qlSgKJdi; spf=none (imf23.hostedemail.com: domain of mail-pj1-f49.google.com has no SPF policy when checking 209.85.216.49) smtp.helo=mail-pj1-f49.google.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1632699836-285171 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit The comment in madvise_dontneed_free() says that vma splits that occur while the mmap-lock is dropped, during userfaultfd_remove(), should be handled correctly, but nothing in the code indicates that it is so: prev is invalidated, and do_madvise() will therefore continue to update VMAs from the "obsolete" end (i.e., the one before the split). Propagate the changes to end from madvise_dontneed_free() back to do_madvise() and continue the updates from the new end accordingly. Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Fixes: 70ccb92fdd90 ("userfaultfd: non-cooperative: userfaultfd_remove revalidate vma in MADV_DONTNEED") Signed-off-by: Nadav Amit --- mm/madvise.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 0734db8d53a7..a2b05352ebfe 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -768,10 +768,11 @@ static long madvise_dontneed_single_vma(struct vm_area_struct *vma, static long madvise_dontneed_free(struct vm_area_struct *vma, struct vm_area_struct **prev, - unsigned long start, unsigned long end, + unsigned long start, unsigned long *pend, int behavior) { struct mm_struct *mm = vma->vm_mm; + unsigned long end = *pend; *prev = vma; if (!can_madv_lru_vma(vma)) @@ -811,6 +812,7 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, * end-vma->vm_end range, but the manager can * handle a repetition fine. */ + *pend = end; end = vma->vm_end; } VM_WARN_ON(start >= end); @@ -980,8 +982,10 @@ static int madvise_inject_error(int behavior, static long madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, - unsigned long start, unsigned long end, int behavior) + unsigned long start, unsigned long *pend, int behavior) { + unsigned long end = *pend; + switch (behavior) { case MADV_REMOVE: return madvise_remove(vma, prev, start, end); @@ -993,7 +997,7 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, return madvise_pageout(vma, prev, start, end); case MADV_FREE: case MADV_DONTNEED: - return madvise_dontneed_free(vma, prev, start, end, behavior); + return madvise_dontneed_free(vma, prev, start, pend, behavior); case MADV_POPULATE_READ: case MADV_POPULATE_WRITE: return madvise_populate(vma, prev, start, end, behavior); @@ -1199,7 +1203,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh tmp = end; /* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ - error = madvise_vma(vma, &prev, start, tmp, behavior); + error = madvise_vma(vma, &prev, start, &tmp, behavior); if (error) goto out; start = tmp; From patchwork Sun Sep 26 16:12:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FBD1C4332F for ; Sun, 26 Sep 2021 23:44:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED76761151 for ; Sun, 26 Sep 2021 23:43:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org ED76761151 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id D28036B0075; Sun, 26 Sep 2021 19:43:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB0F66B0078; Sun, 26 Sep 2021 19:43:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0507900002; Sun, 26 Sep 2021 19:43:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id 9E67B6B0075 for ; Sun, 26 Sep 2021 19:43:58 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4941B181B04B5 for ; Sun, 26 Sep 2021 23:43:58 +0000 (UTC) X-FDA: 78631354956.15.9FEBE01 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf07.hostedemail.com (Postfix) with ESMTP id ECD86100009E for ; Sun, 26 Sep 2021 23:43:57 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id rm6-20020a17090b3ec600b0019ece2bdd20so2306531pjb.1 for ; Sun, 26 Sep 2021 16:43:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0dVwIqoXrWYrBeh50L+yTf5lQAryqwYLoi0T5EzujgU=; b=H90hDEbMYLTnjkduq4ioXC3Q6x4/kREz9IaMVmY4/QEAYpTVnrObJL0XyucVJgD0aE EroOvxTubIUR87F0ETk3/AfezkybAt06QkLRvFwNPeOWs900MrsK69QBOKmj+3uAXqlu 76kIT0uPgs6y4eusRE9/4nKXyhs1PGnvHYzON1LLB6G3EKMkCz/HK3ES3vT2s8hooQ3A mPWEj+92GHVCaZme0s8KuSghaNpssd2ip2J3WBBlw0Li8DqNYFJ/qMbbVP8F9bJqPTTJ DYHtU+uKxiX7VF0Yucv9b5C2pNn8OECmb/XmfH+qEyf2ICr+/DhjNMsNc0gydewwR2Zl McOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0dVwIqoXrWYrBeh50L+yTf5lQAryqwYLoi0T5EzujgU=; b=rA+Nf6Nas3kw+qb8ahrYHyx/WZ0sGrEWYDnfwr7U/xbMOXfIXgE+M++oTP0wYzLPCX 59iwWiIEKqJwVQLQO1SWcS+MgWeAKLAyzenwjU+1NyGwRJ8uVTrcm/QABxKxtjxyi5FJ LmKw2BjYiy8Y0unFsypA+3VqmB5GDNgy6J+8ZfxVp30csw5WRScmSbLygWhKswbcuCYK TrBXGBn3gRwsxZnqfkIIENoh+CiM/GTdW9/TJYuZNKMaB1vA0AGn+N+JcsnBhBdjaDYC ITTluojHAGTMVFz7MhKkC9+zIT1RJdncoVUQ1zVAzPoMeF9JkJNQV+0oDT5pZFEK9OiW Vssw== X-Gm-Message-State: AOAM530zDBi3p097OP8N+KenFt8bzwfhQItpGj6F47gQ23vqNUEly1Ko 1JXewuOuPA6vEH9r27GADMg= X-Google-Smtp-Source: ABdhPJyRIFjbZLc5BHV5apUrQUvzT/e1HBNNhmP+BETtZps1zARdLAa0dGFFMUZLxRSGC2M1VfACxw== X-Received: by 2002:a17:90b:1bd1:: with SMTP id oa17mr16386548pjb.82.1632699836897; Sun, 26 Sep 2021 16:43:56 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.43.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:43:56 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 2/8] mm/madvise: remove unnecessary check on madvise_dontneed_free() Date: Sun, 26 Sep 2021 09:12:53 -0700 Message-Id: <20210926161259.238054-3-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: ECD86100009E X-Stat-Signature: kcdea571s47gwofap61ihn8xunypptdh Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=H90hDEbM; spf=none (imf07.hostedemail.com: domain of mail-pj1-f43.google.com has no SPF policy when checking 209.85.216.43) smtp.helo=mail-pj1-f43.google.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1632699837-811065 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit madvise_dontneed_free() is called only from madvise_vma() and the behavior is always either MADV_FREE or MADV_DONTNEED. There is no need to check again in madvise_dontneed_free() if the behavior is any different. Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/madvise.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index a2b05352ebfe..fe843513a4e8 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -820,10 +820,8 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, if (behavior == MADV_DONTNEED) return madvise_dontneed_single_vma(vma, start, end); - else if (behavior == MADV_FREE) + else /* behavior == MADV_FREE */ return madvise_free_single_vma(vma, start, end); - else - return -EINVAL; } static long madvise_populate(struct vm_area_struct *vma, From patchwork Sun Sep 26 16:12:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BAFDC433EF for ; Sun, 26 Sep 2021 23:44:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 95BF2610CA for ; Sun, 26 Sep 2021 23:44:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 95BF2610CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id EDD116B0078; Sun, 26 Sep 2021 19:43:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DEB12900002; Sun, 26 Sep 2021 19:43:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B79CB6B007D; Sun, 26 Sep 2021 19:43:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id A27BE6B0078 for ; Sun, 26 Sep 2021 19:43:59 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 6073B8249980 for ; Sun, 26 Sep 2021 23:43:59 +0000 (UTC) X-FDA: 78631354998.28.4FC97E3 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf04.hostedemail.com (Postfix) with ESMTP id 20B7150000AD for ; Sun, 26 Sep 2021 23:43:59 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id n2so10481713plk.12 for ; Sun, 26 Sep 2021 16:43:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NySU333JXW/fyiS9W1CxTkc8pnmhF1ZxQI/NACcKQmE=; b=S2Xi3H0DaUdTWODZoF6hKlWXPNbgIRsxr+vIrSIhNnJVNSOFJz/CL1weP2sL5o/aC8 paGG1RCdehDqjnIGdEuBezmF2cfX5r7V75kFFEx7yR73kkvUzjUDdqBrf9BrrJiKM7cC pmHlhh1Gf4IALFaw9lBJKJyZGODDCdcbHsiGf02WdaRxzCVPDs9+25P5/bKOhiBdBLyd mh9KXGUJt/7fOUwc5kdkLGLTnwnpf1a7p8cEQv4wSc338zrdXcD2vj/04po7T4y1o7Su zyW4dnzYq4bN0B9fxef857ggnUOZiN+emo0zn3toHpLbDVvRFLCWrmqfGJq0VaK6THsq FuBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NySU333JXW/fyiS9W1CxTkc8pnmhF1ZxQI/NACcKQmE=; b=ZWz807We9HgyH6S0e5NHEc0doDW1o47tRq+CmCOzpt7sH9EmPVNZ1lZ1/i+6kmcFIN P2mJFw9Ab1efg65zuCNSnyqRDpY6tqXomgzA+j5DdGWj0uDmO/lzidQ0NCh0hehRAYop jRrhPxkzb/weaPfj7e6VAeDCeBmSdI1Tx/Xzm29C4RXwFp2g5GTunzjRiFcXmxNrGu+X Koo9x+JaUWzAbMu4Cr27cCnS7nh1Eh+urV61NgnSfTJbKoWBd7THYqN75KvMk7BCblRI +JyHwJR+4M4K8MwVPt2Ho5bncq5Sbv4IHnKmcJnhfT+ZDgXigIwdZUKvMFceK326Ufqy YseQ== X-Gm-Message-State: AOAM530/h6wh1+Tf40RCtPCxwcZ0deIMYva3yFgbaQcsp2FBAl0bTpa9 tFNseZHIsNRmz9zq34MrqSI= X-Google-Smtp-Source: ABdhPJylHV5/+/gjFjZAYFu4kVShQNgLaLHnT25MFpUQmwpdWFzGophkZ/jj2j+GSp2ZtFbgIvdljg== X-Received: by 2002:a17:902:9684:b0:13e:32cb:c95d with SMTP id n4-20020a170902968400b0013e32cbc95dmr383828plp.31.1632699838216; Sun, 26 Sep 2021 16:43:58 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.43.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:43:57 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 3/8] mm/madvise: remove unnecessary checks on madvise_free_single_vma() Date: Sun, 26 Sep 2021 09:12:54 -0700 Message-Id: <20210926161259.238054-4-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 20B7150000AD X-Stat-Signature: swt7yau3p1j5zharnmza9i65of4kz5ep Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=S2Xi3H0D; spf=none (imf04.hostedemail.com: domain of mail-pl1-f174.google.com has no SPF policy when checking 209.85.214.174) smtp.helo=mail-pl1-f174.google.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1632699839-691690 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit madvise_free_single_vma() currently rechecks that the range fits within the VMA, adapts it accordingly, and returns -EINVAL if the range is entirely outside of the VMA. The error-code of -EINVAL is incorrect according to the man pages (as it should have been -ENOMEM), but anyhow the range that is provided to madvise_free_single_vma() should always be valid. It is set correctly in do_madvise() and then rechecked in madvise_dontneed_free() is the mmap-lock is dropped. Remove this check. Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/madvise.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index fe843513a4e8..17e39c70704b 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -716,14 +716,8 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, if (!vma_is_anonymous(vma)) return -EINVAL; - range.start = max(vma->vm_start, start_addr); - if (range.start >= vma->vm_end) - return -EINVAL; - range.end = min(vma->vm_end, end_addr); - if (range.end <= vma->vm_start) - return -EINVAL; mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, - range.start, range.end); + start_addr, end_addr); lru_add_drain(); tlb_gather_mmu(&tlb, mm); From patchwork Sun Sep 26 16:12:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEBFBC433EF for ; Sun, 26 Sep 2021 23:44:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6C0A5610CA for ; Sun, 26 Sep 2021 23:44:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6C0A5610CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7D3166B007B; Sun, 26 Sep 2021 19:44:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BD806B007D; Sun, 26 Sep 2021 19:44:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53A2C6B007E; Sun, 26 Sep 2021 19:44:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0203.hostedemail.com [216.40.44.203]) by kanga.kvack.org (Postfix) with ESMTP id 367156B007B for ; Sun, 26 Sep 2021 19:44:01 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E2EB21802DF6A for ; Sun, 26 Sep 2021 23:44:00 +0000 (UTC) X-FDA: 78631355040.01.D05F26E Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf23.hostedemail.com (Postfix) with ESMTP id 9FBE890000A6 for ; Sun, 26 Sep 2021 23:44:00 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id h3so16045723pgb.7 for ; Sun, 26 Sep 2021 16:44:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7IgX6dL5NOEQt+/OKQBPTYq3DM8yqU22ikiRTuWFEQs=; b=iCi1KOKi0WklCCvp2FWY4GYlC2rLLIejsnGEbQv645XvjEvRLC03v/zhGtg9n7XhI5 B7SDPa7KX8ZnvOjkUBe0CCIoVYh6gTmIbz3aOJUartJx+s2v8diqqYBRmDpD9sv8nrbf 2YajJ6pOYEjoSYxPtatkIp0SsNnV5CUJlljPKsQyaMLQmnrh1OZfp3saxNV9wGOGBbb4 dkeHjdYZk8bjL3Q1v88Q7+WNv2SCd8BuohBIQP14f4TeMZc1Rek2pse0wAsz3XH+FZjZ RvGNfN2EEGMdoANBp+ep4zUR9Il8wWpwBvyFfyFk1K502jardGs8i2c2SXldKHtFjhLP FTDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7IgX6dL5NOEQt+/OKQBPTYq3DM8yqU22ikiRTuWFEQs=; b=NQLVxiZSMPho88boBTwDKHPO68MiVsUZyH+WjkDqZLBMmM9F8GVkbWQMXeHj9SzBf6 yGHN5ICGYGrzgrO4u+6ZHVet03O7BCMm/C8HhETo4yBCz1/8v8RHrfVLhbtKw+lXCimM 9c1PtkM9JQjs40kQHlI2LgZlqPcEySiJ00tZeXODGQU+Jo00qnZxr9SV6sSGfhYXr2In vpAWK6gM+oegM1gA9dyEEKyRVxzYstZ0L13tdf7nOm0Pnz6K9iZMDcHygYqkSWeOOUo6 9Qp62j2ijFjBsDdhf+SLqkNFkBttF1QH3bgjXJI3BDbrC1EsGF/6Im0OEmMS48L14Grm Gv2g== X-Gm-Message-State: AOAM530HUXgyL84XHTbWIM8XolGT+sLCcw/bsNRsCGScP1R1gi2XFFep tjAqbqYtvWlBw6BJUu8GV6s= X-Google-Smtp-Source: ABdhPJx0ScK7+0SaPr9aLkqxWDiNKDgH66MEuAVqy1Czm2rGkTHqNSM7usqARFL9qBBrsg6IAd+PBQ== X-Received: by 2002:a63:8543:: with SMTP id u64mr13991077pgd.220.1632699839511; Sun, 26 Sep 2021 16:43:59 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.43.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:43:59 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 4/8] mm/madvise: define madvise behavior in a struct Date: Sun, 26 Sep 2021 09:12:55 -0700 Message-Id: <20210926161259.238054-5-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iCi1KOKi; spf=none (imf23.hostedemail.com: domain of mail-pg1-f175.google.com has no SPF policy when checking 209.85.215.175) smtp.helo=mail-pg1-f175.google.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9FBE890000A6 X-Stat-Signature: aaryye4ofbedig9goq95c4bwdry7k11b X-HE-Tag: 1632699840-809159 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit The different behaviors of madvise are different in several ways, which are distributed across several functions. Use the design pattern from iouring in order to define the actions that are required for each behavior. The next patches will get rid of old helper functions that are modified in this patch and the redundant use of array_index_nospec(). The next patches will add more actions for each leaf into the new struct. No functional change is intended. Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/madvise.c | 168 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 109 insertions(+), 59 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 17e39c70704b..127507c71ba9 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -29,6 +29,7 @@ #include #include #include +#include #include @@ -39,6 +40,101 @@ struct madvise_walk_private { bool pageout; }; +struct madvise_info { + u8 behavior_valid: 1; + u8 process_behavior_valid: 1; + u8 need_mmap_read_only: 1; +}; + +static const struct madvise_info madvise_info[MADV_SOFT_OFFLINE+1] = { + [MADV_DOFORK] = { + .behavior_valid = 1, + }, + [MADV_DONTFORK] = { + .behavior_valid = 1, + }, + [MADV_NORMAL] = { + .behavior_valid = 1, + }, + [MADV_SEQUENTIAL] = { + .behavior_valid = 1, + }, + [MADV_RANDOM] = { + .behavior_valid = 1, + }, + [MADV_REMOVE] = { + .behavior_valid = 1, + .need_mmap_read_only = 1, + }, + [MADV_WILLNEED] = { + .behavior_valid = 1, + .process_behavior_valid = 1, + .need_mmap_read_only = 1, + }, + [MADV_DONTNEED] = { + .behavior_valid = 1, + .need_mmap_read_only = 1, + }, + [MADV_FREE] = { + .behavior_valid = 1, + .need_mmap_read_only = 1, + }, + [MADV_COLD] = { + .behavior_valid = 1, + .process_behavior_valid = 1, + .need_mmap_read_only = 1, + }, + [MADV_PAGEOUT] = { + .behavior_valid = 1, + .process_behavior_valid = 1, + .need_mmap_read_only = 1, + }, +#ifdef CONFIG_KSM + [MADV_MERGEABLE] = { + .behavior_valid = 1, + }, + [MADV_UNMERGEABLE] = { + .behavior_valid = 1, + }, +#endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + [MADV_HUGEPAGE] = { + .behavior_valid = 1, + }, + [MADV_NOHUGEPAGE] = { + .behavior_valid = 1, + }, +#endif + [MADV_DONTDUMP] = { + .behavior_valid = 1, + }, + [MADV_DODUMP] = { + .behavior_valid = 1, + }, + [MADV_WIPEONFORK] = { + .behavior_valid = 1, + }, + [MADV_KEEPONFORK] = { + .behavior_valid = 1, + }, +#ifdef CONFIG_MEMORY_FAILURE + [MADV_HWPOISON] = { + .behavior_valid = 1, + }, + [MADV_SOFT_OFFLINE] = { + .behavior_valid = 1, + }, +#endif + [MADV_POPULATE_READ] = { + .behavior_valid = 1, + .need_mmap_read_only = 1, + }, + [MADV_POPULATE_WRITE] = { + .behavior_valid = 1, + .need_mmap_read_only = 1, + }, +}; + /* * Any behaviour which results in changes to the vma->vm_flags needs to * take mmap_lock for writing. Others, which simply traverse vmas, need @@ -46,20 +142,7 @@ struct madvise_walk_private { */ static int madvise_need_mmap_write(int behavior) { - switch (behavior) { - case MADV_REMOVE: - case MADV_WILLNEED: - case MADV_DONTNEED: - case MADV_COLD: - case MADV_PAGEOUT: - case MADV_FREE: - case MADV_POPULATE_READ: - case MADV_POPULATE_WRITE: - return 0; - default: - /* be safe, default to 1. list exceptions explicitly */ - return 1; - } + return !madvise_info[behavior].need_mmap_read_only; } /* @@ -999,56 +1082,23 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, } static bool -madvise_behavior_valid(int behavior) +madvise_behavior_valid(int *behavior) { - switch (behavior) { - case MADV_DOFORK: - case MADV_DONTFORK: - case MADV_NORMAL: - case MADV_SEQUENTIAL: - case MADV_RANDOM: - case MADV_REMOVE: - case MADV_WILLNEED: - case MADV_DONTNEED: - case MADV_FREE: - case MADV_COLD: - case MADV_PAGEOUT: - case MADV_POPULATE_READ: - case MADV_POPULATE_WRITE: -#ifdef CONFIG_KSM - case MADV_MERGEABLE: - case MADV_UNMERGEABLE: -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - case MADV_HUGEPAGE: - case MADV_NOHUGEPAGE: -#endif - case MADV_DONTDUMP: - case MADV_DODUMP: - case MADV_WIPEONFORK: - case MADV_KEEPONFORK: -#ifdef CONFIG_MEMORY_FAILURE - case MADV_SOFT_OFFLINE: - case MADV_HWPOISON: -#endif - return true; - - default: + if (*behavior >= ARRAY_SIZE(madvise_info)) return false; - } + + *behavior = array_index_nospec(*behavior, ARRAY_SIZE(madvise_info)); + return madvise_info[*behavior].behavior_valid; } static bool -process_madvise_behavior_valid(int behavior) +process_madvise_behavior_valid(int *behavior) { - switch (behavior) { - case MADV_COLD: - case MADV_PAGEOUT: - case MADV_WILLNEED: - return true; - default: + if (*behavior >= ARRAY_SIZE(madvise_info)) return false; - } + + *behavior = array_index_nospec(*behavior, ARRAY_SIZE(madvise_info)); + return madvise_info[*behavior].process_behavior_valid; } /* @@ -1133,7 +1183,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh start = untagged_addr(start); - if (!madvise_behavior_valid(behavior)) + if (!madvise_behavior_valid(&behavior)) return error; if (!PAGE_ALIGNED(start)) @@ -1258,7 +1308,7 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, goto put_pid; } - if (!process_madvise_behavior_valid(behavior)) { + if (!process_madvise_behavior_valid(&behavior)) { ret = -EINVAL; goto release_task; } From patchwork Sun Sep 26 16:12:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D88C0C433F5 for ; Sun, 26 Sep 2021 23:44:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6D3E161151 for ; Sun, 26 Sep 2021 23:44:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6D3E161151 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id ED8776B007D; Sun, 26 Sep 2021 19:44:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E118A6B007E; Sun, 26 Sep 2021 19:44:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C64CB6B0080; Sun, 26 Sep 2021 19:44:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A46706B007D for ; Sun, 26 Sep 2021 19:44:02 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 430D8181B04B5 for ; Sun, 26 Sep 2021 23:44:02 +0000 (UTC) X-FDA: 78631355124.25.EB219D2 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf16.hostedemail.com (Postfix) with ESMTP id EDEECF00008E for ; Sun, 26 Sep 2021 23:44:01 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id t9so2387525pju.5 for ; Sun, 26 Sep 2021 16:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2uIfgW0DLl5b1zzqh2vZMp9HHNJQ53DebXYGC7KgLB0=; b=bt8Pcnk8mKhwMewC+6gJPcZtVLn3LgH+s5vi1JOXVopT5zHaY2DkYK6+b2U8p4bIOX lAeHJbgDTLWYj+5YtJH7UfeL9CUM/d7s5NRt2GZ6JzK4POchQrFyrwNidjjpKGtI9kMO AcDzmr1YjmuVE0CAERvBrFzrobICaeOS/SqwBmNbHjYvpUBXf6rkBHfBRad1hktUlawA QjQtXJj9HGYiFzsUHakUJTydVh1i12SjfLMRtaMLKHRAUgKK4N57dXgOz1XKk7bwTYH4 DvvQaJ8d/dWE/dV9OV4mhlSR+7qz8ndm69A6LEnHTWvZWtSOpedT5vYw+NDYNZ2TnLaC mmbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2uIfgW0DLl5b1zzqh2vZMp9HHNJQ53DebXYGC7KgLB0=; b=fd6K7vVtjADZ6mr4I2Fh5idNLHjzXaTUnM/Ppo5QZr5omKdbKdMhzS+8DLYSTxqkIR WD/ryJ1HvPGi7JljhL9QUHTTQYks+rpOzOkuAADh+4dSxq8bLUhBq7SMsp3qI5OCIeo+ Vwn3WzSgLNohxPmsUmuttmH3JwskUTOravr55NXaK5KaqrFBY/AMjsXEStuBhsvzeXP6 XlStICm0vThvv0/6VEpIcxdCIyVqDd1fXkq7CJ2k2tWCqDoPzav6OdOF8NoWrfWoWPAs fPJJJH4fkzLwON3kFT+7e9PyzlvHUeaU6/s4+R+evaKx8rA9TVGYgxG+N/X1qnjcaHFi 70zA== X-Gm-Message-State: AOAM5319qcbRAsa9G48TlcZQdT8XWibs0qHoa5cHMR+43/w+JsVwbDbB FaNTEoIkcKQYJpBUqpranRg= X-Google-Smtp-Source: ABdhPJyQ474KOK4qAUHpwiptg0L4FDkBiV23b+5O5tnyftNkmc2gHVMBQqJw+JEulEWae2ZCYAkxWw== X-Received: by 2002:a17:90b:3b4c:: with SMTP id ot12mr16305275pjb.67.1632699840874; Sun, 26 Sep 2021 16:44:00 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.43.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:44:00 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 5/8] mm/madvise: perform certain operations once on process_madvise() Date: Sun, 26 Sep 2021 09:12:56 -0700 Message-Id: <20210926161259.238054-6-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EDEECF00008E X-Stat-Signature: ipk1w4pmt34ctsn814zybz5raf1h5jbh Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bt8Pcnk8; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf16.hostedemail.com: domain of mail-pj1-f50.google.com has no SPF policy when checking 209.85.216.50) smtp.helo=mail-pj1-f50.google.com X-HE-Tag: 1632699841-607300 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000381, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit There are certain operations that can be performed only once on process_madvise() instead of performing them for each IO vector. Acquiring the mmap-lock, and initializing blk_plug are specifically such operations. Collect the aforementioned operations into madvise_start() and madvise_finish(). The next patches will add additional operations into these functions. Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/madvise.c | 139 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 86 insertions(+), 53 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 127507c71ba9..84b86ae85671 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -43,6 +43,13 @@ struct madvise_walk_private { struct madvise_info { u8 behavior_valid: 1; u8 process_behavior_valid: 1; + u8 no_mmap_lock: 1; + + /* + * Any behaviour which results in changes to the vma->vm_flags needs to + * take mmap_lock for writing. Others, which simply traverse vmas, need + * to only take it for reading. + */ u8 need_mmap_read_only: 1; }; @@ -120,9 +127,11 @@ static const struct madvise_info madvise_info[MADV_SOFT_OFFLINE+1] = { #ifdef CONFIG_MEMORY_FAILURE [MADV_HWPOISON] = { .behavior_valid = 1, + .no_mmap_lock = 1, }, [MADV_SOFT_OFFLINE] = { .behavior_valid = 1, + .no_mmap_lock = 1, }, #endif [MADV_POPULATE_READ] = { @@ -135,16 +144,6 @@ static const struct madvise_info madvise_info[MADV_SOFT_OFFLINE+1] = { }, }; -/* - * Any behaviour which results in changes to the vma->vm_flags needs to - * take mmap_lock for writing. Others, which simply traverse vmas, need - * to only take it for reading. - */ -static int madvise_need_mmap_write(int behavior) -{ - return !madvise_info[behavior].need_mmap_read_only; -} - /* * We can potentially split a vm area into separate * areas, each area with its own behavior. @@ -1081,26 +1080,6 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, } } -static bool -madvise_behavior_valid(int *behavior) -{ - if (*behavior >= ARRAY_SIZE(madvise_info)) - return false; - - *behavior = array_index_nospec(*behavior, ARRAY_SIZE(madvise_info)); - return madvise_info[*behavior].behavior_valid; -} - -static bool -process_madvise_behavior_valid(int *behavior) -{ - if (*behavior >= ARRAY_SIZE(madvise_info)) - return false; - - *behavior = array_index_nospec(*behavior, ARRAY_SIZE(madvise_info)); - return madvise_info[*behavior].process_behavior_valid; -} - /* * The madvise(2) system call. * @@ -1171,21 +1150,17 @@ process_madvise_behavior_valid(int *behavior) * -EBADF - map exists, but area maps something that isn't a file. * -EAGAIN - a kernel resource was temporarily unavailable. */ -int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior) +int madvise_one_range(struct mm_struct *mm, unsigned long start, size_t len_in, + int behavior) { unsigned long end, tmp; struct vm_area_struct *vma, *prev; int unmapped_error = 0; int error = -EINVAL; - int write; size_t len; - struct blk_plug plug; start = untagged_addr(start); - if (!madvise_behavior_valid(&behavior)) - return error; - if (!PAGE_ALIGNED(start)) return error; len = PAGE_ALIGN(len_in); @@ -1207,14 +1182,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh return madvise_inject_error(behavior, start, start + len_in); #endif - write = madvise_need_mmap_write(behavior); - if (write) { - if (mmap_write_lock_killable(mm)) - return -EINTR; - } else { - mmap_read_lock(mm); - } - /* * If the interval [start,end) covers some unmapped address * ranges, just ignore them, but return -ENOMEM at the end. @@ -1224,7 +1191,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh if (vma && start > vma->vm_start) prev = vma; - blk_start_plug(&plug); for (;;) { /* Still start < end. */ error = -ENOMEM; @@ -1260,15 +1226,72 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh vma = find_vma(mm, start); } out: - blk_finish_plug(&plug); - if (write) - mmap_write_unlock(mm); - else - mmap_read_unlock(mm); return error; } +static int +madvise_start(struct mm_struct *mm, struct madvise_info behavior_info, + struct blk_plug *plug) +{ + if (!behavior_info.no_mmap_lock) { + if (behavior_info.need_mmap_read_only) + mmap_read_lock(mm); + else if (mmap_write_lock_killable(mm)) + return -EINTR; + } + + blk_start_plug(plug); + return 0; +} + +static void +madvise_finish(struct mm_struct *mm, struct madvise_info behavior_info, + struct blk_plug *plug) +{ + blk_finish_plug(plug); + + if (!behavior_info.no_mmap_lock) { + if (behavior_info.need_mmap_read_only) + mmap_read_unlock(mm); + else + mmap_write_unlock(mm); + } +} + +static struct madvise_info madvise_behavior_info(int behavior) +{ + if (behavior >= ARRAY_SIZE(madvise_info) || behavior < 0) { + const struct madvise_info invalid = {0}; + return invalid; + } + + behavior = array_index_nospec(behavior, ARRAY_SIZE(madvise_info)); + return madvise_info[behavior]; +} + +int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, + int behavior) +{ + struct madvise_info behavior_info; + struct blk_plug plug; + int ret = -EINVAL; + + behavior_info = madvise_behavior_info(behavior); + + if (!behavior_info.behavior_valid) + return ret; + + ret = madvise_start(mm, behavior_info, &plug); + if (ret != 0) + return ret; + + ret = madvise_one_range(mm, start, len_in, behavior); + + madvise_finish(mm, behavior_info, &plug); + return ret; +} + SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior) { return do_madvise(current->mm, start, len_in, behavior); @@ -1286,6 +1309,8 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, struct mm_struct *mm; size_t total_len; unsigned int f_flags; + struct madvise_info behavior_info; + struct blk_plug plug; if (flags != 0) { ret = -EINVAL; @@ -1308,7 +1333,9 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, goto put_pid; } - if (!process_madvise_behavior_valid(&behavior)) { + behavior_info = madvise_behavior_info(behavior); + + if (!behavior_info.process_behavior_valid) { ret = -EINVAL; goto release_task; } @@ -1331,15 +1358,21 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, total_len = iov_iter_count(&iter); + ret = madvise_start(mm, behavior_info, &plug); + if (ret != 0) + goto release_mm; + while (iov_iter_count(&iter)) { iovec = iov_iter_iovec(&iter); - ret = do_madvise(mm, (unsigned long)iovec.iov_base, - iovec.iov_len, behavior); + ret = madvise_one_range(mm, (unsigned long)iovec.iov_base, + iovec.iov_len, behavior); if (ret < 0) break; iov_iter_advance(&iter, iovec.iov_len); } + madvise_finish(mm, behavior_info, &plug); + if (ret == 0) ret = total_len - iov_iter_count(&iter); From patchwork Sun Sep 26 16:12:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 058F7C433FE for ; Sun, 26 Sep 2021 23:44:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9727561154 for ; Sun, 26 Sep 2021 23:44:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9727561154 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 1C0696B007E; Sun, 26 Sep 2021 19:44:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1676C6B0080; Sun, 26 Sep 2021 19:44:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2714900002; Sun, 26 Sep 2021 19:44:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id DBA8B6B007E for ; Sun, 26 Sep 2021 19:44:03 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 97EC68249980 for ; Sun, 26 Sep 2021 23:44:03 +0000 (UTC) X-FDA: 78631355166.22.752A073 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf18.hostedemail.com (Postfix) with ESMTP id 49EAD4002087 for ; Sun, 26 Sep 2021 23:44:03 +0000 (UTC) Received: by mail-pg1-f180.google.com with SMTP id h3so16045778pgb.7 for ; Sun, 26 Sep 2021 16:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OkEF0/Q6rbx9FExzFJEmcylCwn+F9BK/PAVlqDzqCuk=; b=Tl1nreO8ZaENYCUeyUvf3Ic0WiU+vCsHdK5O+hJJHl8u6WsUXxobXZpORdPtNvK6IA OE4Z2YMCMbdDachKu+3TxvHTbmJ4TDa9/EJuI3xLg3OLfVr6zVbUnYGjeqamKB5RIbLg FY/Z9APV3f3FlThudedelEZZuncC1xDPPXY+OEQXv3mbJr+znGld+b3GBdN3/i2Q1Sr2 evxIstLnH3dhVtpZ9ZbgJtPWjKvRY8PuwgPjfJeY0/u9bNRVV4CmEAeN5JJDyJ8oA6ih Az6cRRcfm9DW2Y+zpxhJe+opXPtCE0zrp1S+JE+tc+nc9T7otIW5cdjkbtom1v3EoRSk TI6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OkEF0/Q6rbx9FExzFJEmcylCwn+F9BK/PAVlqDzqCuk=; b=iz/5hlPsX7D+rTwMCuPda25PzAhAMr8dCmbQzdfRPzqsAWJ9m1QfoapqYwlvIBPqLS wf5J4uLF2d9w9sLxm6lVitt7ORGM0RfS9npda/2ktWTYyCxfGbLTZEbCCIdX59gWH1uB SBxpIrY9PmcTlzktOT60gFvpHF3NGLwJ1kfO5v5fnCMaI8YJupfoDR05LVygZYsZfkI8 1BrQ16LN2SdvLChuL/d9AXLARNBzbGdW+1YNmczRH+r9gIX054LGebbRRAEYKSh2N4l0 eza3kUaxFE8uOUtF4m6asigndXp4qzQnM2ErryG3dPOWAjVClRcDxvxQVtqesctFL14w 5ZRQ== X-Gm-Message-State: AOAM532KAS7k0ZpialRuvGGsIwmUd8+5SEfXs+mSG9CxCvnAhyLWSm0N 3BGT/DHx99VQw035dtco6VA= X-Google-Smtp-Source: ABdhPJyhOgaxFKSJHYAE0/nHMseXbjf23Cg7H2wwtSb134c4XmaFXwSEjSXcFhZgWJ81w8SLx9TtUA== X-Received: by 2002:a63:4606:: with SMTP id t6mr13935096pga.388.1632699842239; Sun, 26 Sep 2021 16:44:02 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.44.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:44:01 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 6/8] mm/madvise: more aggressive TLB batching Date: Sun, 26 Sep 2021 09:12:57 -0700 Message-Id: <20210926161259.238054-7-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 49EAD4002087 X-Stat-Signature: tk8hbcca1ixkmwq63mgq4hzr3zd4cc6a Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Tl1nreO8; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf18.hostedemail.com: domain of mail-pg1-f180.google.com has no SPF policy when checking 209.85.215.180) smtp.helo=mail-pg1-f180.google.com X-HE-Tag: 1632699843-771419 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit process_madvise(MADV_COLD) and future variants lose the opportunity to perform TLB batching across IO vectors. As a result, a TLB flush (and potentially a shootdown) is performed for each IO vector, instead of one for the whole operation. Batch TLB operations across IO-vectors. Open code zap_page_range() to do so. A future patch will eliminate redundancies between madvise_free_single_vma() and madvise_dontneed_single_vma(). Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/internal.h | 5 +++ mm/madvise.c | 104 +++++++++++++++++++++++++++++--------------------- mm/memory.c | 2 +- 3 files changed, 67 insertions(+), 44 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 18256e32a14c..2313a6739ce7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -59,6 +59,11 @@ void unmap_page_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, struct zap_details *details); +void unmap_single_vma(struct mmu_gather *tlb, + struct vm_area_struct *vma, unsigned long start_addr, + unsigned long end_addr, + struct zap_details *details); + void do_page_cache_ra(struct readahead_control *, unsigned long nr_to_read, unsigned long lookahead_size); void force_page_cache_ra(struct readahead_control *, unsigned long nr); diff --git a/mm/madvise.c b/mm/madvise.c index 84b86ae85671..e679cfa94655 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -51,6 +51,7 @@ struct madvise_info { * to only take it for reading. */ u8 need_mmap_read_only: 1; + u8 tlb_batching: 1; }; static const struct madvise_info madvise_info[MADV_SOFT_OFFLINE+1] = { @@ -81,20 +82,24 @@ static const struct madvise_info madvise_info[MADV_SOFT_OFFLINE+1] = { [MADV_DONTNEED] = { .behavior_valid = 1, .need_mmap_read_only = 1, + .tlb_batching = 1, }, [MADV_FREE] = { .behavior_valid = 1, .need_mmap_read_only = 1, + .tlb_batching = 1, }, [MADV_COLD] = { .behavior_valid = 1, .process_behavior_valid = 1, .need_mmap_read_only = 1, + .tlb_batching = 1, }, [MADV_PAGEOUT] = { .behavior_valid = 1, .process_behavior_valid = 1, .need_mmap_read_only = 1, + .tlb_batching = 1, }, #ifdef CONFIG_KSM [MADV_MERGEABLE] = { @@ -578,21 +583,16 @@ static void madvise_cold_page_range(struct mmu_gather *tlb, tlb_end_vma(tlb, vma); } -static long madvise_cold(struct vm_area_struct *vma, - struct vm_area_struct **prev, - unsigned long start_addr, unsigned long end_addr) +static long madvise_cold(struct mmu_gather *tlb, struct vm_area_struct *vma, + struct vm_area_struct **prev, unsigned long start_addr, + unsigned long end_addr) { - struct mm_struct *mm = vma->vm_mm; - struct mmu_gather tlb; - *prev = vma; if (!can_madv_lru_vma(vma)) return -EINVAL; lru_add_drain(); - tlb_gather_mmu(&tlb, mm); - madvise_cold_page_range(&tlb, vma, start_addr, end_addr); - tlb_finish_mmu(&tlb); + madvise_cold_page_range(tlb, vma, start_addr, end_addr); return 0; } @@ -628,13 +628,10 @@ static inline bool can_do_pageout(struct vm_area_struct *vma) file_permission(vma->vm_file, MAY_WRITE) == 0; } -static long madvise_pageout(struct vm_area_struct *vma, +static long madvise_pageout(struct mmu_gather *tlb, struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start_addr, unsigned long end_addr) { - struct mm_struct *mm = vma->vm_mm; - struct mmu_gather tlb; - *prev = vma; if (!can_madv_lru_vma(vma)) return -EINVAL; @@ -643,9 +640,7 @@ static long madvise_pageout(struct vm_area_struct *vma, return 0; lru_add_drain(); - tlb_gather_mmu(&tlb, mm); - madvise_pageout_page_range(&tlb, vma, start_addr, end_addr); - tlb_finish_mmu(&tlb); + madvise_pageout_page_range(tlb, vma, start_addr, end_addr); return 0; } @@ -787,12 +782,12 @@ static const struct mm_walk_ops madvise_free_walk_ops = { .pmd_entry = madvise_free_pte_range, }; -static int madvise_free_single_vma(struct vm_area_struct *vma, +static int madvise_free_single_vma(struct mmu_gather *tlb, + struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr) { struct mm_struct *mm = vma->vm_mm; struct mmu_notifier_range range; - struct mmu_gather tlb; /* MADV_FREE works for only anon vma at the moment */ if (!vma_is_anonymous(vma)) @@ -802,16 +797,14 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, start_addr, end_addr); lru_add_drain(); - tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); mmu_notifier_invalidate_range_start(&range); tlb_start_vma(&tlb, vma); walk_page_range(vma->vm_mm, range.start, range.end, - &madvise_free_walk_ops, &tlb); - tlb_end_vma(&tlb, vma); + &madvise_free_walk_ops, tlb); + tlb_end_vma(tlb, vma); mmu_notifier_invalidate_range_end(&range); - tlb_finish_mmu(&tlb); return 0; } @@ -820,7 +813,7 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, * Application no longer needs these pages. If the pages are dirty, * it's OK to just throw them away. The app will be more careful about * data it wants to keep. Be sure to free swap resources too. The - * zap_page_range call sets things up for shrink_active_list to actually free + * unmap_single_vma call sets things up for shrink_active_list to actually free * these pages later if no one else has touched them in the meantime, * although we could add these pages to a global reuse list for * shrink_active_list to pick up before reclaiming other pages. @@ -835,14 +828,28 @@ static int madvise_free_single_vma(struct vm_area_struct *vma, * An interface that causes the system to free clean pages and flush * dirty pages is already available as msync(MS_INVALIDATE). */ -static long madvise_dontneed_single_vma(struct vm_area_struct *vma, +static long madvise_dontneed_single_vma(struct mmu_gather *tlb, + struct vm_area_struct *vma, unsigned long start, unsigned long end) { - zap_page_range(vma, start, end - start); + struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, start, + end); + + lru_add_drain(); + update_hiwater_rss(mm); + + mmu_notifier_invalidate_range_start(&range); + unmap_single_vma(tlb, vma, start, end, NULL); + mmu_notifier_invalidate_range_end(&range); + return 0; } -static long madvise_dontneed_free(struct vm_area_struct *vma, +static long madvise_dontneed_free(struct mmu_gather *tlb, + struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long *pend, int behavior) @@ -895,9 +902,9 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, } if (behavior == MADV_DONTNEED) - return madvise_dontneed_single_vma(vma, start, end); + return madvise_dontneed_single_vma(tlb, vma, start, end); else /* behavior == MADV_FREE */ - return madvise_free_single_vma(vma, start, end); + return madvise_free_single_vma(tlb, vma, start, end); } static long madvise_populate(struct vm_area_struct *vma, @@ -1055,8 +1062,9 @@ static int madvise_inject_error(int behavior, #endif static long -madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, - unsigned long start, unsigned long *pend, int behavior) +madvise_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, + struct vm_area_struct **prev, unsigned long start, + unsigned long *pend, int behavior) { unsigned long end = *pend; @@ -1066,12 +1074,13 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, case MADV_WILLNEED: return madvise_willneed(vma, prev, start, end); case MADV_COLD: - return madvise_cold(vma, prev, start, end); + return madvise_cold(tlb, vma, prev, start, end); case MADV_PAGEOUT: - return madvise_pageout(vma, prev, start, end); + return madvise_pageout(tlb, vma, prev, start, end); case MADV_FREE: case MADV_DONTNEED: - return madvise_dontneed_free(vma, prev, start, pend, behavior); + return madvise_dontneed_free(tlb, vma, prev, start, pend, + behavior); case MADV_POPULATE_READ: case MADV_POPULATE_WRITE: return madvise_populate(vma, prev, start, end, behavior); @@ -1151,7 +1160,7 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev, * -EAGAIN - a kernel resource was temporarily unavailable. */ int madvise_one_range(struct mm_struct *mm, unsigned long start, size_t len_in, - int behavior) + int behavior, struct mmu_gather *tlb) { unsigned long end, tmp; struct vm_area_struct *vma, *prev; @@ -1211,7 +1220,7 @@ int madvise_one_range(struct mm_struct *mm, unsigned long start, size_t len_in, tmp = end; /* Here vma->vm_start <= start < tmp <= (end|vma->vm_end). */ - error = madvise_vma(vma, &prev, start, &tmp, behavior); + error = madvise_vma(tlb, vma, &prev, start, &tmp, behavior); if (error) goto out; start = tmp; @@ -1225,6 +1234,7 @@ int madvise_one_range(struct mm_struct *mm, unsigned long start, size_t len_in, else /* madvise_remove dropped mmap_lock */ vma = find_vma(mm, start); } + out: return error; @@ -1232,7 +1242,7 @@ int madvise_one_range(struct mm_struct *mm, unsigned long start, size_t len_in, static int madvise_start(struct mm_struct *mm, struct madvise_info behavior_info, - struct blk_plug *plug) + struct mmu_gather *tlb, struct blk_plug *plug) { if (!behavior_info.no_mmap_lock) { if (behavior_info.need_mmap_read_only) @@ -1241,16 +1251,22 @@ madvise_start(struct mm_struct *mm, struct madvise_info behavior_info, return -EINTR; } + if (behavior_info.tlb_batching) + tlb_gather_mmu(tlb, mm); + blk_start_plug(plug); return 0; } static void madvise_finish(struct mm_struct *mm, struct madvise_info behavior_info, - struct blk_plug *plug) + struct mmu_gather *tlb, struct blk_plug *plug) { blk_finish_plug(plug); + if (behavior_info.tlb_batching) + tlb_finish_mmu(tlb); + if (!behavior_info.no_mmap_lock) { if (behavior_info.need_mmap_read_only) mmap_read_unlock(mm); @@ -1274,6 +1290,7 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior) { struct madvise_info behavior_info; + struct mmu_gather tlb; struct blk_plug plug; int ret = -EINVAL; @@ -1282,13 +1299,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, if (!behavior_info.behavior_valid) return ret; - ret = madvise_start(mm, behavior_info, &plug); + ret = madvise_start(mm, behavior_info, &tlb, &plug); if (ret != 0) return ret; - ret = madvise_one_range(mm, start, len_in, behavior); + ret = madvise_one_range(mm, start, len_in, behavior, &tlb); - madvise_finish(mm, behavior_info, &plug); + madvise_finish(mm, behavior_info, &tlb, &plug); return ret; } @@ -1310,6 +1327,7 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, size_t total_len; unsigned int f_flags; struct madvise_info behavior_info; + struct mmu_gather tlb; struct blk_plug plug; if (flags != 0) { @@ -1358,20 +1376,20 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec, total_len = iov_iter_count(&iter); - ret = madvise_start(mm, behavior_info, &plug); + ret = madvise_start(mm, behavior_info, &tlb, &plug); if (ret != 0) goto release_mm; while (iov_iter_count(&iter)) { iovec = iov_iter_iovec(&iter); ret = madvise_one_range(mm, (unsigned long)iovec.iov_base, - iovec.iov_len, behavior); + iovec.iov_len, behavior, &tlb); if (ret < 0) break; iov_iter_advance(&iter, iovec.iov_len); } - madvise_finish(mm, behavior_info, &plug); + madvise_finish(mm, behavior_info, &tlb, &plug); if (ret == 0) ret = total_len - iov_iter_count(&iter); diff --git a/mm/memory.c b/mm/memory.c index 12a7b2094434..a00fb705a77f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1541,7 +1541,7 @@ void unmap_page_range(struct mmu_gather *tlb, } -static void unmap_single_vma(struct mmu_gather *tlb, +void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct zap_details *details) From patchwork Sun Sep 26 16:12:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F14CAC433EF for ; Sun, 26 Sep 2021 23:44:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A750B61154 for ; Sun, 26 Sep 2021 23:44:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A750B61154 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5895D6B0080; Sun, 26 Sep 2021 19:44:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 538816B0081; Sun, 26 Sep 2021 19:44:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38A59900002; Sun, 26 Sep 2021 19:44:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0188.hostedemail.com [216.40.44.188]) by kanga.kvack.org (Postfix) with ESMTP id 2136C6B0080 for ; Sun, 26 Sep 2021 19:44:05 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B9FFC181B04B5 for ; Sun, 26 Sep 2021 23:44:04 +0000 (UTC) X-FDA: 78631355208.37.63D13EB Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf13.hostedemail.com (Postfix) with ESMTP id 82245103200C for ; Sun, 26 Sep 2021 23:44:04 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id on12-20020a17090b1d0c00b001997c60aa29so10371217pjb.1 for ; Sun, 26 Sep 2021 16:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nT513pHzGiMpZgaNWaHuaQRUoarxEdG+vAAtn4ZSI3I=; b=VGC5hFjSViPa8UHidrJ+zGmm4I8RPo6B7NhutVuOVsVcLYUwKbN/PoKdD3xiLZf+aJ 2IOUexZRK1VfbIoDyIRMQgknoPmXlBigfdQqKZlIDe4zqGJhVLh4ZnGIge5Fcqt3qWfE G+qfmVSSKT9ury1J76cbONnaDrXRyELY35rDHMDgMpo1/5ImYxUyf6WtF2Egyp824oiG 5HgNQH3Mi7HDBi4WcV5y8UwFDosNb7948/+VIxJAg9Vt9xZore2YBCJe/tyC2qxM/znX VAKeIIgY2ri6qy2XDBtI8D9wIPbEtQMGtia55vazq3FxpZ6D5erBdureiFmy6KdsgN5y sOog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nT513pHzGiMpZgaNWaHuaQRUoarxEdG+vAAtn4ZSI3I=; b=CeKI/A3uGvita7K6jIV0+e5fYW68U09g2+GsKsPR0z9jhDCzek6uvvSJuZOkiaQF87 I5qFxFu4Kq/vSl6biqdglwzZH2i8rZgThtdSXdLwOVJQRdkGJgLNCyP/aWc9qvKItsAD TJqmbOa2jR2oY8oiEsmfOBiDrsxDdqwVFyHq3N2Qan0LZYXNICYghWjN4z1yv3qCjDqx roQRPE99tzGb8CLJ6kt+V/qfJLnChD1nsyrr2H0Wh+qw6vYONuoh22Xk5uRDb7BiQpNH pu8EIgI3Gqs53m+1TWe+aMOTstalAxsVRwm0v6bJua7bvQPemQ/DT0Dex6hSfsAFHqWd RyDw== X-Gm-Message-State: AOAM531WjI9cyqhm8MI/kk0xoUiYOZcKWSW+88/lmPG4VPTb5jTAP1Cm vlQKZ6Ny0y4niBVSXZ1RnfY= X-Google-Smtp-Source: ABdhPJzAfS8fiN3mGvz5Edu6ZGWMLuREZwgR5MIc7fZhEaI1QNeFZ6PR8DwlKpq2jWtYM1A1vIEMBA== X-Received: by 2002:a17:90b:33c8:: with SMTP id lk8mr16244064pjb.241.1632699843558; Sun, 26 Sep 2021 16:44:03 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:44:03 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 7/8] mm/madvise: deduplicate code in madvise_dontneed_free() Date: Sun, 26 Sep 2021 09:12:58 -0700 Message-Id: <20210926161259.238054-8-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 82245103200C X-Stat-Signature: 5bqkfrrfwfe9hkizeazcyrqceticuf6z Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=VGC5hFjS; spf=none (imf13.hostedemail.com: domain of mail-pj1-f52.google.com has no SPF policy when checking 209.85.216.52) smtp.helo=mail-pj1-f52.google.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1632699844-331355 X-Bogosity: Ham, tests=bogofilter, spamicity=0.002827, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Following the previous patches, madvise_dontneed_single_vma() and madvise_free_single_vma() have redundant code. Consolidate it together into madvise_dontneed_free(). Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/madvise.c | 40 +++++++++++++++------------------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index e679cfa94655..9528c38fb6a4 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -786,25 +786,14 @@ static int madvise_free_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr) { - struct mm_struct *mm = vma->vm_mm; - struct mmu_notifier_range range; - /* MADV_FREE works for only anon vma at the moment */ if (!vma_is_anonymous(vma)) return -EINVAL; - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, - start_addr, end_addr); - - lru_add_drain(); - update_hiwater_rss(mm); - - mmu_notifier_invalidate_range_start(&range); tlb_start_vma(&tlb, vma); - walk_page_range(vma->vm_mm, range.start, range.end, + walk_page_range(vma->vm_mm, start_addr, end_addr, &madvise_free_walk_ops, tlb); tlb_end_vma(tlb, vma); - mmu_notifier_invalidate_range_end(&range); return 0; } @@ -832,18 +821,7 @@ static long madvise_dontneed_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start, unsigned long end) { - struct mm_struct *mm = vma->vm_mm; - struct mmu_notifier_range range; - - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, start, - end); - - lru_add_drain(); - update_hiwater_rss(mm); - - mmu_notifier_invalidate_range_start(&range); unmap_single_vma(tlb, vma, start, end, NULL); - mmu_notifier_invalidate_range_end(&range); return 0; } @@ -855,7 +833,9 @@ static long madvise_dontneed_free(struct mmu_gather *tlb, int behavior) { struct mm_struct *mm = vma->vm_mm; + struct mmu_notifier_range range; unsigned long end = *pend; + long ret; *prev = vma; if (!can_madv_lru_vma(vma)) @@ -901,10 +881,20 @@ static long madvise_dontneed_free(struct mmu_gather *tlb, VM_WARN_ON(start >= end); } + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, start, + end); + + lru_add_drain(); + update_hiwater_rss(mm); + mmu_notifier_invalidate_range_start(&range); + if (behavior == MADV_DONTNEED) - return madvise_dontneed_single_vma(tlb, vma, start, end); + ret = madvise_dontneed_single_vma(tlb, vma, start, end); else /* behavior == MADV_FREE */ - return madvise_free_single_vma(tlb, vma, start, end); + ret = madvise_free_single_vma(tlb, vma, start, end); + + mmu_notifier_invalidate_range_end(&range); + return ret; } static long madvise_populate(struct vm_area_struct *vma, From patchwork Sun Sep 26 16:12:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12518951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2487CC433F5 for ; Sun, 26 Sep 2021 23:44:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C964D61154 for ; Sun, 26 Sep 2021 23:44:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C964D61154 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id CB3696B0081; Sun, 26 Sep 2021 19:44:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA04E900002; Sun, 26 Sep 2021 19:44:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB88A6B0083; Sun, 26 Sep 2021 19:44:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 91E846B0081 for ; Sun, 26 Sep 2021 19:44:06 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 404611802F645 for ; Sun, 26 Sep 2021 23:44:06 +0000 (UTC) X-FDA: 78631355292.24.F84B9BB Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf13.hostedemail.com (Postfix) with ESMTP id 024E1103200C for ; Sun, 26 Sep 2021 23:44:05 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id w19so14099872pfn.12 for ; Sun, 26 Sep 2021 16:44:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8K0wNqs/UoQMVWcbLzg7zpAq9uT/7eob5sdcWrFFCjE=; b=kFD/Z5J2xj2ugTXgq4UIoOmypwygBjvLHOo6K1Tc2eigh5LmmXmw4Zso7FtsJioTot NCveaxdSuE0wWDPtgXUB105caBHoki1Ykx3c6yGx1jzhRfbO6iVmun2NDokYZuBown1b jrDg1MmxL3N3x8JfVYUi5c5Bn9vT7lxFTM0DMbhZtblAQGV73y2w/ClWXvz/ykQVfJ/3 rp8weZtfpGoRSnQOOH2wRKK3bliUnB8OUuU/rYZR5mwQyzNYblNCGOLbyOlq1gl9Q6hB ApCFinCHC7VktDousGIAl5ap9EPULJ2GijlEFQ6okPzKb+v3evU1HXqnoQiYwu9pZqzA wLlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8K0wNqs/UoQMVWcbLzg7zpAq9uT/7eob5sdcWrFFCjE=; b=HyJpaKEBPInpc28LojppxxMCvMV5NoDeku/dvBbN/xXYs+xXnvuUPuuLC45dHfPYOQ sgRpBsuorpxkgNT3bgMvSuYKgO0T5bDRaP2y/dge/tHA3XEkS6BLMnN1rRjMURCDxpd5 5mJnFC1RFYR3xSgyJPz1oXEMdYOLIdV/AKGiT2TIZuVQ3SmRl7qoE3WxeWgmdnqMNGat uBJS3m3QggvJ1cOMPBP57raiupF14cWTUwyRL76L3Z7fFJ9/H0NcZcD7t8Lxzpbu5StV uB6wo0dMpCShrKohqUniOU1KZ9MG6ttyUgzAXIa16x1n9KB8o53ciX9GM1nHXB6X5ylm EmhA== X-Gm-Message-State: AOAM5336I8Vv+WMwVaMEaG49XRIz7y5jj72WDXjCXcwU2/qGOZ0AtJiC OiweXZpMQ0ddSJWlmmFnJEBg/fdcE+Y= X-Google-Smtp-Source: ABdhPJzac/DPGQPxntJXg+GNkdPOEat1QboVk7R+cPMmUqJpSRSm0/fyONTMqEx0qTMlguk/EPtXmQ== X-Received: by 2002:a65:6a4b:: with SMTP id o11mr13448662pgu.278.1632699844886; Sun, 26 Sep 2021 16:44:04 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id n22sm16783686pgc.55.2021.09.26.16.44.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Sep 2021 16:44:04 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Peter Xu , Nadav Amit , Andrea Arcangeli , Minchan Kim , Colin Cross , Suren Baghdasarya , Mike Rapoport Subject: [RFC PATCH 8/8] mm/madvise: process_madvise(MADV_DONTNEED) Date: Sun, 26 Sep 2021 09:12:59 -0700 Message-Id: <20210926161259.238054-9-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210926161259.238054-1-namit@vmware.com> References: <20210926161259.238054-1-namit@vmware.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 024E1103200C X-Stat-Signature: g6k857jnxoxahg4rbgjzcrzn8p9441sa Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="kFD/Z5J2"; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf13.hostedemail.com: domain of mail-pf1-f178.google.com has no SPF policy when checking 209.85.210.178) smtp.helo=mail-pf1-f178.google.com X-HE-Tag: 1632699845-364985 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Userfaultfd users, for the sake of memory management, debugging or other types of monitoring may wish to use process_madvise(MADV_DONTNEED). Moreover, since process_madvise() supports vectored operations, and now supports efficient TLB flushes, existing users of madvise(MADV_DONTNEED) that wish to perform advices on non-contiguous memory may prefer the vectored process_madvise() flavor for performance reasons. Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Minchan Kim Cc: Colin Cross Cc: Suren Baghdasarya Cc: Mike Rapoport Signed-off-by: Nadav Amit --- mm/madvise.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/madvise.c b/mm/madvise.c index 9528c38fb6a4..d8f70960680e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -81,6 +81,7 @@ static const struct madvise_info madvise_info[MADV_SOFT_OFFLINE+1] = { }, [MADV_DONTNEED] = { .behavior_valid = 1, + .process_behavior_valid = 1, .need_mmap_read_only = 1, .tlb_batching = 1, },