From patchwork Sat Jul 3 04:04:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Zhou X-Patchwork-Id: 12356897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0289EC07E97 for ; Sat, 3 Jul 2021 04:04:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6C271613E6 for ; Sat, 3 Jul 2021 04:04:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C271613E6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C1E276B0011; Sat, 3 Jul 2021 00:04:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCE986B0036; Sat, 3 Jul 2021 00:04:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6F016B005D; Sat, 3 Jul 2021 00:04:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 857806B0011 for ; Sat, 3 Jul 2021 00:04:53 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 23702250D0 for ; Sat, 3 Jul 2021 04:04:53 +0000 (UTC) X-FDA: 78319935666.03.97CDF20 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) by imf27.hostedemail.com (Postfix) with ESMTP id D2D29700009D for ; Sat, 3 Jul 2021 04:04:52 +0000 (UTC) Received: by mail-io1-f47.google.com with SMTP id k11so14080561ioa.5 for ; Fri, 02 Jul 2021 21:04:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=CVoiGwg8bDCGqfxgKgsIzljf6zxgJKc8KUJPsMXiyQA=; b=YvjTgJ+SmVTTSYQ/layZZkVPiwBP/gDo7qp3xbCmeRm0Y6po7Gy5s4poiIme3dIglC VfnU9KEvzSve9SRJgcBCoIjPT/5lWtYC2zI6ChN6cDrLUTG1xrrpOiWUEjFtVH9qv4qC ErIgstpveWk3fbqNJypP4ZQi9YTQ/D23VzU9V5WZ1jD59m/b8OQk8nFEhsaPJLQvTnHp 0P17/+PEKI9nfHSy3IG9KDrxPL1P/5nnnVvfVjq7LFSiJFGiNMKNOFKH3jJgg33pOyq2 UpPD/lhcKbGowlkmDCJ5QnXvtMGVsLlcIZwDCUss0DKTsoQwzuFoqrsE9394NCFw9x8R 8V3A== X-Gm-Message-State: AOAM530BgI4x3pz6OxJhbwBmq3+h+gCpwmmNnoFq4rAIqr/iAWV0gfd5 c2+WmmqIiuMX0LzXwYlilU4= X-Google-Smtp-Source: ABdhPJzd5aKJFzyezOczroeaBohpXe0o+RjufI1q6AU0IOwT1bkykFGnPgvnbDLF21W7EETdx+xL7Q== X-Received: by 2002:a6b:e403:: with SMTP id u3mr2563815iog.22.1625285092276; Fri, 02 Jul 2021 21:04:52 -0700 (PDT) Received: from abasin.c.googlers.com.com (243.199.238.35.bc.googleusercontent.com. [35.238.199.243]) by smtp.gmail.com with ESMTPSA id a11sm2996675ioq.12.2021.07.02.21.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jul 2021 21:04:51 -0700 (PDT) From: Dennis Zhou To: Tejun Heo , Christoph Lameter , Guenter Roeck Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dennis Zhou Subject: [PATCH] percpu: flush tlb after pcpu_depopulate_chunk() Date: Sat, 3 Jul 2021 04:04:49 +0000 Message-Id: <20210703040449.3213210-1-dennis@kernel.org> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D2D29700009D X-Stat-Signature: donb4aotw3ze5iditr8gsqcwsnydsd9o Authentication-Results: imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of dennisszhou@gmail.com designates 209.85.166.47 as permitted sender) smtp.mailfrom=dennisszhou@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=kernel.org (policy=none) X-HE-Tag: 1625285092-71253 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Prior to "percpu: implement partial chunk depopulation", pcpu_depopulate_chunk() was called only on the destruction path. This meant the virtual address range was on its way back to vmalloc which will handle flushing the tlbs for us. However, now that we call pcpu_depopulate_chunk() during the active lifecycle of a chunk, we need to flush the tlb as well otherwise we can end up accessing the wrong page through an invalid tlb mapping. This was reported in [1]. [1] https://lore.kernel.org/lkml/20210702191140.GA3166599@roeck-us.net/ Fixes: f183324133ea ("percpu: implement partial chunk depopulation") Reported-by: Guenter Roeck Signed-off-by: Dennis Zhou --- mm/percpu-km.c | 3 ++- mm/percpu-vm.c | 11 +++++++++-- mm/percpu.c | 7 ++++--- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/mm/percpu-km.c b/mm/percpu-km.c index c9d529dc7651..6875fc3b2ed7 100644 --- a/mm/percpu-km.c +++ b/mm/percpu-km.c @@ -39,7 +39,8 @@ static int pcpu_populate_chunk(struct pcpu_chunk *chunk, } static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, - int page_start, int page_end) + int page_start, int page_end, + bool flush_tlb) { /* nada */ } diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index ee5d89fcd66f..6353cda1718e 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -299,6 +299,7 @@ static int pcpu_populate_chunk(struct pcpu_chunk *chunk, * @chunk: chunk to depopulate * @page_start: the start page * @page_end: the end page + * @flush_tlb: if should we flush the tlb * * For each cpu, depopulate and unmap pages [@page_start,@page_end) * from @chunk. @@ -307,7 +308,8 @@ static int pcpu_populate_chunk(struct pcpu_chunk *chunk, * pcpu_alloc_mutex. */ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, - int page_start, int page_end) + int page_start, int page_end, + bool flush_tlb) { struct page **pages; @@ -324,7 +326,12 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, pcpu_unmap_pages(chunk, pages, page_start, page_end); - /* no need to flush tlb, vmalloc will handle it lazily */ + /* + * We need to flush the tlb unless the caller will pass it to vmalloc, + * which will handle flushing for us. + */ + if (flush_tlb) + pcpu_post_unmap_tlb_flush(chunk, page_start, page_end); pcpu_free_pages(chunk, pages, page_start, page_end); } diff --git a/mm/percpu.c b/mm/percpu.c index b4cebeca4c0c..e23ba0d22220 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1580,7 +1580,8 @@ static void pcpu_chunk_depopulated(struct pcpu_chunk *chunk, static int pcpu_populate_chunk(struct pcpu_chunk *chunk, int page_start, int page_end, gfp_t gfp); static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, - int page_start, int page_end); + int page_start, int page_end, + bool flush_tlb); static struct pcpu_chunk *pcpu_create_chunk(gfp_t gfp); static void pcpu_destroy_chunk(struct pcpu_chunk *chunk); static struct page *pcpu_addr_to_page(void *addr); @@ -2016,7 +2017,7 @@ static void pcpu_balance_free(bool empty_only) bitmap_for_each_set_region(chunk->populated, rs, re, 0, chunk->nr_pages) { - pcpu_depopulate_chunk(chunk, rs, re); + pcpu_depopulate_chunk(chunk, rs, re, false); spin_lock_irq(&pcpu_lock); pcpu_chunk_depopulated(chunk, rs, re); spin_unlock_irq(&pcpu_lock); @@ -2189,7 +2190,7 @@ static void pcpu_reclaim_populated(void) continue; spin_unlock_irq(&pcpu_lock); - pcpu_depopulate_chunk(chunk, i + 1, end + 1); + pcpu_depopulate_chunk(chunk, i + 1, end + 1, true); cond_resched(); spin_lock_irq(&pcpu_lock);