From patchwork Wed Jul 25 14:06:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544063 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B20D2180E for ; Wed, 25 Jul 2018 14:06:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A216420408 for ; Wed, 25 Jul 2018 14:06:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 961B6206E2; Wed, 25 Jul 2018 14:06:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 46BA820408 for ; Wed, 25 Jul 2018 14:06:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED7E46B02B6; Wed, 25 Jul 2018 10:06:56 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E62A36B02B8; Wed, 25 Jul 2018 10:06:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D037D6B02B9; Wed, 25 Jul 2018 10:06:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 8B7536B02B8 for ; Wed, 25 Jul 2018 10:06:56 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id r20-v6so4246174pgv.20 for ; Wed, 25 Jul 2018 07:06:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=qIuwOscxl1EW4BeFkvSG6ETyVL4wwncYhTvemKPOb6Q=; b=kqdGVwqLlBj2ZI10MEq5AaenU0FZf+aks9lRWSDYDWVGG1GbZs7vxgydv/3tQZiKgA jyjGdvK5IkukZKTl7F90OJYOIt6RUFZKJoQ+yiE27izWLV4AL32aXUOy/HesbpxWydnv H/QDUB/X2Ulhevh5pALsT8P0WYOu15OZl5aD8L+qN0PgRL+4Ryr5Qy3Uj5WhJYKMW6sz 50QwtKY/4seXD6XHsNqHVwXjfXtrs501yPjsHqFN0ZJ0kbvBa8j/xLGPquQr0jYe8VGd l5fkxKLx9kfS26FZvQ50osmdEmrXK2MqVDXDTMCmDdhFq9BAdWX86BvnnO3yLqDwDvl3 kAgw== X-Gm-Message-State: AOUpUlF6GdmmSEaWLdkOze15NFiLoh3Kaka1xh36U+M1pW/Q2HQ1alNA gKdh2jG8Ja9G0DjZeeUUwKliAl2HkxOxabNLKoylKXfbiwjLMuR8TnnTC4/Xhxuj81EQjARuosZ TJsFJkr2gelxaUw9084CbnL+kLYhRT3g1GK1iHLT2K1Lkz2Rm4ypi/GfOiSVOGlVtalpCii1Tpy f3yTNxAOHoDKuB26XRQ0SwMkBEurRuXr4LzU9kpOje8wozReQRQcXlFMNBWCRn+2QtaC6hTypwg o3v/Jg62TI86r0nb0aVO+ol38aw7ykjDnaMKQ2OSckEb9jMzcP9/5it2UYLWFvopiVos8O/+I7a /zpsRqxmLx8ub89GVhqNDoFeWMaU5VpFY3/xFZpF1QUaCV7k4LG4uftFVM7G+VlQwc8ukD1UvQk D X-Received: by 2002:a17:902:201:: with SMTP id 1-v6mr21513145plc.310.1532527616251; Wed, 25 Jul 2018 07:06:56 -0700 (PDT) X-Received: by 2002:a17:902:201:: with SMTP id 1-v6mr21513110plc.310.1532527615560; Wed, 25 Jul 2018 07:06:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532527615; cv=none; d=google.com; s=arc-20160816; b=eIudEgp8ryMUWnVZmom4mMzAc/v2mnH5eW6NwDp4zSngZ4i1zONBPG+pA1YkrHEZWW vPUQmDpSzfbp9nekx3k/fVqZQbwezEQLFGPamJpHB6dqLfiYO0WvuLWut4ePv/FT3Li2 NUogAp5f6wxZAG+pW5e+SP4xKyzg2gMsHQ+L4wvY9k1NbvKWECqy3OJvrKdztJ8xfhTq uUCMp2ld80TXcGX7Vwb98usOgNnURRNOetZ1K2+i+1x55+CPhIr8p0OuXnZGN9HgAs0z DTM3dFw1cqGiazRi3wv2adOkviI7CS/UjZ+nFZccpwWkoqtb1gooTdJXwd/F/H9aYCq3 ArGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=qIuwOscxl1EW4BeFkvSG6ETyVL4wwncYhTvemKPOb6Q=; b=rNmWi6ywmsII66xd7PqoiphMwfecvrL/Z7eZmjYKb9mGyndlTJm8Z7BtY20oDmqPMn zssMBm+aUK3m9lUL4K2pfJ4OAceBnEkfI0aCKNOA+g2rOZltez6KY6TKE+ayr0u9dhpF Xm2Ca/utY6kkUjKF1bbDf7PPpUtkHGSBGDLlC9GzrKlURRWMwjZCEfGDFT/I2sARSJag YAIVv/7GUtKOC+sYbM2niQk/hgrnroatExWJN3X8EjQos4GQzBYTkKkiATgmjA/DMy4Z ACMpd/5FOJ7zMq8ujIyIB6GCt8NfjQOaPY4G3JeKo+W0C0Sc8WzxCL/zrCWzAoInGYXU TKZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dzwz6C7k; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id h6-v6sor3627918pgb.415.2018.07.25.07.06.55 for (Google Transport Security); Wed, 25 Jul 2018 07:06:55 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dzwz6C7k; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qIuwOscxl1EW4BeFkvSG6ETyVL4wwncYhTvemKPOb6Q=; b=dzwz6C7ksXXQUMV7dQjFsq+iLH4YhL4uk/iB+Bxk0woJ/mzxIeP+44zCyoQ36g30+P 6ug7/AVQxzn1Qb/Wq9Myr07U8uH4d1n5mGVHBURmbp3DM/dSZkPTXA4KAvirXXDtsrb6 a59RxONojKXjsmW98j8qJOYWW2/S63AL6zWxeZUoOuw8Vv1ITBDIHqqys1KDDg5CulUw U0vWA1AIq9Rhsiecum0dpBKgLF3nXsW/pWWdUGdJ0L/96+xh/Pkd2fJWwFOQXBvYzVnB qluufQQSbzmh+J7kD+Wl2psjbZmnNoxeALjDYjs1gk3Lxba/EBJa4WwVMfGEBa92Vfma a2ig== X-Google-Smtp-Source: AAOMgpfwf14R92RD5z2T+Q6XLzanWkVmVxsh3MPbqEVttaK81oTV2itdmfykO/Td2y3hCxZfnhMX+Q== X-Received: by 2002:a63:4450:: with SMTP id t16-v6mr20744377pgk.102.1532527615190; Wed, 25 Jul 2018 07:06:55 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.06.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:06:54 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 1/4] mm: move tlb_table_flush to tlb_flush_mmu_free Date: Thu, 26 Jul 2018 00:06:38 +1000 Message-Id: <20180725140641.30372-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There is no need to call this from tlb_flush_mmu_tlbonly, it logically belongs with tlb_flush_mmu_free. This allows some code consolidation with a subsequent fix. Signed-off-by: Nicholas Piggin --- mm/memory.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7206a634270b..bc053d5e9d41 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -245,9 +245,6 @@ static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) tlb_flush(tlb); mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); -#ifdef CONFIG_HAVE_RCU_TABLE_FREE - tlb_table_flush(tlb); -#endif __tlb_reset_range(tlb); } @@ -255,6 +252,9 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb) { struct mmu_gather_batch *batch; +#ifdef CONFIG_HAVE_RCU_TABLE_FREE + tlb_table_flush(tlb); +#endif for (batch = &tlb->local; batch && batch->nr; batch = batch->next) { free_pages_and_swap_cache(batch->pages, batch->nr); batch->nr = 0; From patchwork Wed Jul 25 14:06:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544065 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 923CB1805 for ; Wed, 25 Jul 2018 14:07:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 819E620408 for ; Wed, 25 Jul 2018 14:07:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 75A1B206E2; Wed, 25 Jul 2018 14:07:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22FD820408 for ; Wed, 25 Jul 2018 14:07:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 898E26B02B8; Wed, 25 Jul 2018 10:07:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 849AD6B02BA; Wed, 25 Jul 2018 10:07:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C5E86B02BB; Wed, 25 Jul 2018 10:07:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id 2467A6B02B8 for ; Wed, 25 Jul 2018 10:07:00 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id 70-v6so5506978plc.1 for ; Wed, 25 Jul 2018 07:07:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=7UxphlsljR+NlBYsTH8Esis+vxdkqmrjRFhGi68hhJU=; b=VvEzi+pEzbXEYp5MvZR9hOwMRe08AfGLyo+XLEC+sUUiJ7G0chXYJ1WGMZ/MStomKQ kfINbGvdzGvCvoO0/5Zl+qdkuEep2FH9lcK8KcuYx8t5WsfRCy8lWe7XHJiAAeo4e0ZD F1y+/BpNSQrL9uLrgWYI5OK7qwft3sCvaIcLZxIthcEbmL5neLGoHUVXq61FU9NgMVUK fjP26nwpaIZYXpJmmrbw4OY++gE4v9ZCtHD5aqWfTIGY+HNCmLcHesXSkq89mK4GJZaV o8UyF3tSstCsHRjpIdOPFxjQ8djTQIYRHTBToBeYBTYruy/PRRoeiDE445CrT9V7CtZz qokg== X-Gm-Message-State: AOUpUlE25sm1QZDksTxFfKgXZ7Q3hHsGIO5GpTbVwVQuJmVEsOU6b2Yb r6URswyzC+37uqJ9cXF1qezBQTq+kolXAB6e2XP2Bon3+NU1K+UZoFAj3MOWR4Dr/FV4qzf0etw zufAZFCKn9NMPR3aZieAniCr38cFq/qSkl9dj9Iu8BOedhqor361Il0wAtp620LqpjcQeph/e5L u/w2FJeXgEmOeUV+pfvN1voXWq3yK407wZ0NS1mqolavOqBHl4UZrlofW8oiF+czVuPl4jZIHf7 MkiR0VXCaKxmSN5/MHwiRrw3cRA3qNczj+tPYvemo0JhN4YE3JM/ZgVSVbuqITRCA18PQy+Sb7y y2tzZGy2PlnbZ0o9pmUQHOXL2zuDvW2zQcfX1/eS1fd0hPMtPExLdH+3ng4EKiVnadl3+AhMGao 0 X-Received: by 2002:a62:da07:: with SMTP id c7-v6mr22246100pfh.106.1532527619844; Wed, 25 Jul 2018 07:06:59 -0700 (PDT) X-Received: by 2002:a62:da07:: with SMTP id c7-v6mr22246066pfh.106.1532527619236; Wed, 25 Jul 2018 07:06:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532527619; cv=none; d=google.com; s=arc-20160816; b=ap7gfHnGsVnC5T+xr7lxFMETqlSwMZnff22RoNyu8JxPs8ExDfAAGOIPPjhU9Ox+0l oP3W07yl0avdHCsT0A7lYhJMDpgNjuDaLzEt9V+ewegZs6IWpaSznw/1c9SfvpSQf8XR 6amoSf4EbmLUMsc2WJcgURNqLS0NVo+eaTrJtv9nYBysIWc0TUSp4LTAngRRNHwhcXt/ x0eAivP9H9NUUGGFNQF6XX+il1aDCj5wLk6mSI2DsQeahtkqISXjW6yXPGdkS58JcGTn C+BO1m0i8ph4O1wS4JSfdFk/6W4RtvIAr7/XB+Iza/Zfb05znWEKCYfib+E+2KHwuBzj 1qQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=7UxphlsljR+NlBYsTH8Esis+vxdkqmrjRFhGi68hhJU=; b=tsLqynORoUm+5cewHhroQFKzbCjxwsevL3Fnsw2lTlxbVLBDi/LTREH1N67VbkPK8G v1BBnlBbeUt1AKn47uI+rTgN+0DnEUKRzzeG8TLbh8rUcp8CKu7IvFPuXFQAkG+LqMkT xbtGZeubXis7i3Soh/p1fIAJEW86XNrO663D/vT+ttZTs0QzLmjoktqvEt78gwskeQGi sVHeaM32heUDAswqOQDXIg/jw8syI+4YxBXc0/1qikiuRymy19bC2RbfJXhwiAHu6z5K K5es/n9sZ+GiVqsLMtZDrNywdDa8qNkiEjgOAucBF1gYt+Z2cAgMb8M2Fr1iV5fH8sxe VTiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=XYwUKztU; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id r9-v6sor4829619pls.43.2018.07.25.07.06.59 for (Google Transport Security); Wed, 25 Jul 2018 07:06:59 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=XYwUKztU; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7UxphlsljR+NlBYsTH8Esis+vxdkqmrjRFhGi68hhJU=; b=XYwUKztUXh78GoFkh5YY9kJxt1SDxMMuG1PsEcnsnNXBRKhePtsbFe3AnAfT9CnG3F 76/aXAEa+jKfRKzD3EGLQyJIfLNGmBxyLEU0g26pqCCN/WXJp6Lujfr1lcShyyN7SPG0 qjVS9IbZDC422uQn8tJgc3XkOhmB2XNE/iJPkLuBI8+/Rt/gEfGPYhVpzTYms+vjMJ9s QYzif4mG6CltSruM3MlDSCP5mE/r+BLYQGDtpyb/sflG1t/JHyat7zeoRQ/rC8gaq3cg onN8GAHAJnDbEckhAAw0JxwVqJKWGpbISgM0TewXOwIdXnNuIJkzCEMnnPSzUa7SNFd0 G6LQ== X-Google-Smtp-Source: AAOMgpdCqh7xbrmOwc7Mb7NkJWlmmf0rHmvB7dgvBNPoBO+OeJs5vUqKg07Ng8VHsiopNqqLXn0+RA== X-Received: by 2002:a17:902:2f43:: with SMTP id s61-v6mr20949070plb.274.1532527618840; Wed, 25 Jul 2018 07:06:58 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.06.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:06:57 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 2/4] mm: mmu_notifier fix for tlb_end_vma Date: Thu, 26 Jul 2018 00:06:39 +1000 Message-Id: <20180725140641.30372-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The generic tlb_end_vma does not call invalidate_range mmu notifier, and it resets resets the mmu_gather range, which means the notifier won't be called on part of the range in case of an unmap that spans multiple vmas. ARM64 seems to be the only arch I could see that has notifiers and uses the generic tlb_end_vma. I have not actually tested it. --- include/asm-generic/tlb.h | 17 +++++++++++++---- mm/memory.c | 10 ---------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 3063125197ad..b3353e21f3b3 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -15,6 +15,7 @@ #ifndef _ASM_GENERIC__TLB_H #define _ASM_GENERIC__TLB_H +#include #include #include #include @@ -138,6 +139,16 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) } } +static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) +{ + if (!tlb->end) + return; + + tlb_flush(tlb); + mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); + __tlb_reset_range(tlb); +} + static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { @@ -186,10 +197,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define __tlb_end_vma(tlb, vma) \ do { \ - if (!tlb->fullmm && tlb->end) { \ - tlb_flush(tlb); \ - __tlb_reset_range(tlb); \ - } \ + if (!tlb->fullmm) \ + tlb_flush_mmu_tlbonly(tlb); \ } while (0) #ifndef tlb_end_vma diff --git a/mm/memory.c b/mm/memory.c index bc053d5e9d41..135d18b31e44 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -238,16 +238,6 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, __tlb_reset_range(tlb); } -static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) -{ - if (!tlb->end) - return; - - tlb_flush(tlb); - mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); - __tlb_reset_range(tlb); -} - static void tlb_flush_mmu_free(struct mmu_gather *tlb) { struct mmu_gather_batch *batch; From patchwork Wed Jul 25 14:06:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544067 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 965021805 for ; Wed, 25 Jul 2018 14:07:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 859352022C for ; Wed, 25 Jul 2018 14:07:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 79692206E2; Wed, 25 Jul 2018 14:07:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1BE182022C for ; Wed, 25 Jul 2018 14:07:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3B6616B02BA; Wed, 25 Jul 2018 10:07:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 365B76B02BC; Wed, 25 Jul 2018 10:07:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20B2B6B02BD; Wed, 25 Jul 2018 10:07:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id CCC0C6B02BA for ; Wed, 25 Jul 2018 10:07:03 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id q12-v6so4891866pgp.6 for ; Wed, 25 Jul 2018 07:07:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=/jOJ/+W2JSaRGc382cEHsBJ4YMQ1luV4W/5/qP+a/jo=; b=iqLeHHCkwlZPC+0aMc154dwdQ2SJAcr1R492FcMYcdxBd2woS1EdoJI+iefg42FwGa P7Zqgw0GGyWlR8NBV9ReysgjbFUnjrsUA/ry7GID729l5kqS6WWlISX2m+HKbHv9ElAY S+NblY4L3CL7lu+pMfBd48XADk74ttjImtOJP9djWBlOlsBQl6p+jibHWq3fnfO8yeCa 3nqxprrY9yxUmkZHdlFDVOgdhmaZQQvopTpWmjJEwcXNdogEGczEmS3AVBxBuVuBvBOZ oAY32awMd1PMYlUt67nSCSTXU/fFBxOL2LKnxHcLWYVJQpdn272sh6AoZlFXgSyI+Sa7 C71Q== X-Gm-Message-State: AOUpUlGAatAbPFkekKjxKbtbuW7WjDqfCO6faXg5U/HK9B3MDzwrqJVB R1oeHgxCvvea8iVCfqDA952M+2HuvIWmcGElfwcLyCMEAsGtIW45cFAxbwyuBzxyHLaUG34Uw49 uH6ityEP9MrYKMl5BDw2Ex6Fu22Gfw6f6NaZVG3NUbOEH289halDjs92HhdyYdntdSJ0A4qCsdT YB1UmMfdAXtLwA4YmFIts5Sja80/Dy0MBYLAj8XZJROp3SfnL5WI96AJqPI86z7VOaZjPUT44XZ bF06z/fG0bW99FBzj8g3w7JOUpXx8d1RE6SafQ02QLK7ix4hI97oYWJlpqXpwL3llKPGK7l+do+ fesGiEfEmMQlYQGnQbk8gMizaEuKMhru6zCWFvwiSvHRfBXutlf+TYkjk6AICB/ejGJpwWIKV3e f X-Received: by 2002:a17:902:5281:: with SMTP id a1-v6mr21047496pli.73.1532527623519; Wed, 25 Jul 2018 07:07:03 -0700 (PDT) X-Received: by 2002:a17:902:5281:: with SMTP id a1-v6mr21047454pli.73.1532527622780; Wed, 25 Jul 2018 07:07:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532527622; cv=none; d=google.com; s=arc-20160816; b=YryJ/uic3qya3MmicwfCzoJA33gqj/vn2RTBvqWxj7a6f/DnfBLFdgqwqnJzHsZCiX 0Hnz1YqwPjNCiFgCH6A4ZMJhqBGvBLxkzVMSaBnDhJnvgDEEkRnEuZ4BeTHBCxDID9O6 0saaSv7R5Zkc/NeZQ58zI5Up5LBgjgANauLsk4tmRjxsHMs3V0EBd/Zx+Y6BkGvxJfKC n+VZef4rgG0ENzVtzm9+6Wf+fQFcPSzFL4fBWD27MR3yOFikAKdndMlXJgx17KAtoCfv 52PNrgef3wi4K8QBCwZOoYwpckBHIqBekNPrqaNebIYSKfLNsVjgmk+AOeKPmR3pVOJa 4Bpg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=/jOJ/+W2JSaRGc382cEHsBJ4YMQ1luV4W/5/qP+a/jo=; b=lsZ2wVxEw2BoJ3Ohd0zABS4FPzpcxZw2W9lfmjHtbMpVjBxceI8lnmVBYRcz33JV71 euDnRXV9IbeaCv00swD0syvE+CwhXvvrHTNeCfUPBK4NU9o9QIcN+aa+WdFL47sNNKBe jxMZC6ZiZsLe8CjvDCnkXWwdSyJq4c4eGeSXEweFXjhlgD68tutazy0nzzhq+XkkaTRY NI3px8ibVz/1+ahACjo01/fEcArrMTSveoHx3FuJO7SyuYLNXDcLdvCJO7ZXFyPYZbih FfKty8kdN0E8zUfYyxgQ02BBrKJORlh4lEvAa9VrrUtUaOuOIC34TARnPfvoKo/7414F 7lDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AVI4jmNY; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x13-v6sor4435317pll.106.2018.07.25.07.07.02 for (Google Transport Security); Wed, 25 Jul 2018 07:07:02 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AVI4jmNY; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/jOJ/+W2JSaRGc382cEHsBJ4YMQ1luV4W/5/qP+a/jo=; b=AVI4jmNYjgqDSMI3WJjS/MJXnTw/tSu8EiFLIssrPgdZhTHZLaKvPOCoEpnbETjqS4 WHjd/x8OQXdeOlIJZUbPRVxOpMGMwrcDzsTehdv1ptJRS1/qcrW9aF1kTRIh0XLb0sib XV1qaHllNNEDcAI65jnDZZ7P3MreQAr0+cZP1hvfEyu+KAC9ZUfhPnMW50xPszAp0y3H C9/m8i+9xcJCwcTacvN8SqAkzeONsqUQeD9sYBf05f2IZuZh+RGeze6vtqjx0oYHku1A l/GAg75BAYlWzv78239VYONTyVhDgOtMQjcHuhGA10RhHYWoC4q1YscpehOQ/d52IXnC LYfw== X-Google-Smtp-Source: AAOMgpdxnMOTaEEmTL31SXnAKEmdYiRjkCxG7VwMPoXvmtNuKHYh+0w2cwP9DLIdwrLmpng7F58Sfw== X-Received: by 2002:a17:902:8697:: with SMTP id g23-v6mr21137522plo.292.1532527622385; Wed, 25 Jul 2018 07:07:02 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.06.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:07:01 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 3/4] mm: allow arch to have tlb_flush caled on an empty TLB range Date: Thu, 26 Jul 2018 00:06:40 +1000 Message-Id: <20180725140641.30372-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP powerpc wants to de-couple page table caching structure flushes from TLB flushes, which will make it possible to have mmu_gather with freed page table pages but no TLB range. These must be sent to tlb_flush, so allow the arch to specify when mmu_gather with empty ranges should have tlb_flush called. Signed-off-by: Nicholas Piggin --- include/asm-generic/tlb.h | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b3353e21f3b3..b320c0cc8996 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -139,14 +139,27 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) } } +/* + * arch_tlb_mustflush specifies if tlb_flush is to be called even if the + * TLB range is empty (this can be the case for freeing page table pages + * if the arch does not adjust TLB range to cover them). + */ +#ifndef arch_tlb_mustflush +#define arch_tlb_mustflush(tlb) false +#endif + static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { - if (!tlb->end) + unsigned long start = tlb->start; + unsigned long end = tlb->end; + + if (!(end || arch_tlb_mustflush(tlb))) return; tlb_flush(tlb); - mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); __tlb_reset_range(tlb); + if (end) + mmu_notifier_invalidate_range(tlb->mm, start, end); } static inline void tlb_remove_page_size(struct mmu_gather *tlb, From patchwork Wed Jul 25 14:06:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544069 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C23E21805 for ; Wed, 25 Jul 2018 14:07:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0D7A2022C for ; Wed, 25 Jul 2018 14:07:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A4A5C206E2; Wed, 25 Jul 2018 14:07:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F57F2022C for ; Wed, 25 Jul 2018 14:07:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 543CD6B02BC; Wed, 25 Jul 2018 10:07:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4C6B66B02BE; Wed, 25 Jul 2018 10:07:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 366F26B02BF; Wed, 25 Jul 2018 10:07:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id E11A06B02BC for ; Wed, 25 Jul 2018 10:07:07 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id az8-v6so5486186plb.15 for ; Wed, 25 Jul 2018 07:07:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=bEa2VC1PhTklj7TrxhDQMGrxkMdS1Gq+3HFfcqOkvaE=; b=De/frg+PH1w/Sh/AkhH+g1cJvblm83byMpliKjw1zdw/a9qxNR6QrJClFCidlZtxmf Mpc/VdQkrJyHMCOyg403+lit+gG5dxqeAz9BQPRoMZ0JOdm1ACt/ZF922XvTlykswoRY piV+I8f3asdjflF/vifetuPXuD9dkSW9Hum3yMm4MJ4xPPQAWi2fzsgKu25D8y+B1+Lw MbEpnc4ci6lV+hQjc32WaFBxkpYA544GEOZrj2lIZvogmWvNQrENxkBhf88K4qfOPttj 34IIItMx2eI0X5yh3Kyj+BWNdvS88Yy/VEc9Wl/VxDIgGMD3AmcnVBpuSw4PXMyosOE/ LM7g== X-Gm-Message-State: AOUpUlH3Q4Bbz0xdh3iLpWHx6xkDjlNkNMWwDlYE622gKtqkOz0QhbDR ngpkXVGxKOOnUNAS212eIgUbOOuf2E8pyv8schc6KAT0W9qGctTV//5T5kEVYFCra64Zgsaz2NO TcjGLUDhDY/ugf2PAGIyLQWuvq0VxShpsAkRukQ2PUZjSOk/qwq5oPHqqnTQPuAjE6J99B5dS/m REkhY6B+U6PvoHqGCT6xwNuoQYTf2g1cbTe5cDG2aA0vlqXVKTnDNXJTfmOa3Nd//CqP0npYF32 hRb16JECpe5f9di1VQQIGPebbI04i+PZBKX9ITkFeHLN9JBVHZGhekl3vn+QQ3KP5k78XEbcw4B Q4W+mFahP3YqMBskEbPl+y0kCwybST5KZYwSP/KvGy9x5NyLCqG9Y20wjvZGaWGk+fjg7XJKT+Z i X-Received: by 2002:a62:5486:: with SMTP id i128-v6mr22066428pfb.166.1532527627601; Wed, 25 Jul 2018 07:07:07 -0700 (PDT) X-Received: by 2002:a62:5486:: with SMTP id i128-v6mr22066378pfb.166.1532527626725; Wed, 25 Jul 2018 07:07:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532527626; cv=none; d=google.com; s=arc-20160816; b=Laa/zHiZtgC5g/XqVghVzfeYbUexA26ab7M/T+yyUA9tD0niYKUx2X0dFujpjij0st C9J7UALhPOlUm3YDpOBhAG1nMnn3DGFm+up5Q4cH53JU0dvD53FVQTxcif6l4jpHPR2B vbQ8UGt+iZYmkpIfn//NfT/ZOLzM4fC3tEiUraDBCA7Af90GamezQtaryIQlgaNNYLz9 svVSozypNFW6OHPDbigbHbsySQkdy7ok6iKqSKjaId1qnKTSId99lyTCuqhiwG+8Xzbt dPcMHQ1Txx8HHSC7EBNbqdqwJYk9j5ZeWa+Inzkd84yQriV/vuAVCY5mOuNzTSi474m6 IfDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=bEa2VC1PhTklj7TrxhDQMGrxkMdS1Gq+3HFfcqOkvaE=; b=syZoU9F1FFrO0LJ35Cuf3nQOnuAHIgqRC0R+FVitvyj9szjzJwhv+rzx6FrsVIj2OH YDS/dDbVF73Mu+rlC2SnQrOiKwyiK++EDjRkS/5HC4Trshzj6sbl9lmySsoQuxYT/5JJ d9fE+JuoO1e8pkf+LXKrdLFjT/mnA4yh2fhjILf2TReuKNJKEentdE9kLwbiOIJA0Vh9 pE2W1m9e0nVWNuxLFPjGAzQfZZfnUrF30Ouoepw2uQIoBP/WH1OrfYOWO3gANgqCBvc9 YlTSgeb/eeSWPm3Yp7jkSH3XJfRKVhA4ztXhi4XemKz5VJnCWF95IpHNVAPIG4K8T0CF 8QGA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="s/oaPYvU"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g14-v6sor4030662pgh.236.2018.07.25.07.07.06 for (Google Transport Security); Wed, 25 Jul 2018 07:07:06 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="s/oaPYvU"; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bEa2VC1PhTklj7TrxhDQMGrxkMdS1Gq+3HFfcqOkvaE=; b=s/oaPYvUXPdAEv1dgAf1aUK8hLBsA4DKGN5fR0/nvxAdQrOg3/KjjiNtDOA7QaQXWz NEzTiowWzU3LAgvm7OI0XcufZkXEoHCS6LVDtWDQvm18bpOnMwPHS52W4Qso3tHkoyi5 B8iSqrksRtRCUClCWEVLQGu7XgPhGCEREcWB8BIKl0cm4A2dJig9mDzc78FAgmxeSQ5c ic7WTfiRxTcuCnUU9DfhJxF/CiyT/PYxg80ATjMSUXRcnQdbkDn3u+352F83n3lZq342 /IGaDM9kJ27wVD5LcdiBh5wbGEtAdF1CVwRIiD1Hk89zzAJK9ARB+44ObGUaSgoxkWDZ aKgQ== X-Google-Smtp-Source: AAOMgpcAXooHLfepiFNZvQ1EZXOE1oPa36WafIdHB3mJLvOw0mOJ8azm5i49n1gtoOVCew/36zGZlA== X-Received: by 2002:a65:660a:: with SMTP id w10-v6mr20323698pgv.366.1532527626243; Wed, 25 Jul 2018 07:07:06 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.07.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:07:05 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 4/4] powerpc/64s/radix: optimise TLB flush with precise TLB ranges in mmu_gather Date: Thu, 26 Jul 2018 00:06:41 +1000 Message-Id: <20180725140641.30372-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The mmu_gather APIs keep track of the invalidated address range, and the generic page table freeing accessors expand the invalidated range to cover the addresses corresponding to the page tables even if there are no ptes and therefore no TLB entries to invalidate. This is done for architectures that have paging structure caches that are invalidated with their TLB invalidate instructions (e.g., x86). powerpc/64s/radix does have a "page walk cache" (PWC), but it is invalidated with a specific instruction and tracked independently in the mmu_gather (using the need_flush_all flag to indicate PWC must be flushed). Therefore TLB invalidation does not have to be expanded to cover freed page tables. This patch defines p??_free_tlb functions for 64s, which do not expand the TLB flush range over page table pages. This brings the number of tlbiel instructions required by a kernel compile from 33M to 25M, most avoided from exec => shift_arg_pages(). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/tlb.h | 34 ++++++++++++++++++++++++++++++++++ arch/powerpc/mm/tlb-radix.c | 10 ++++++++++ include/asm-generic/tlb.h | 5 +++++ 3 files changed, 49 insertions(+) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 9138baccebb0..5d3107f2b014 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -30,6 +30,40 @@ #define __tlb_remove_tlb_entry __tlb_remove_tlb_entry #define tlb_remove_check_page_size_change tlb_remove_check_page_size_change +#ifdef CONFIG_PPC_BOOK3S_64 +/* + * powerpc book3s hash does not have page table structure caches, and + * radix requires explicit management with PWC invalidate tlb type, so + * there is no need to expand the mmu_gather range over invalidated page + * table pages like the generic code does. + */ + +#define pte_free_tlb(tlb, ptep, address) \ + do { \ + __pte_free_tlb(tlb, ptep, address); \ + } while (0) + +#define pmd_free_tlb(tlb, pmdp, address) \ + do { \ + __pmd_free_tlb(tlb, pmdp, address); \ + } while (0) + +#define pud_free_tlb(tlb, pudp, address) \ + do { \ + __pud_free_tlb(tlb, pudp, address); \ + } while (0) + +/* + * Radix sets need_flush_all when page table pages have been unmapped + * and the PWC needs flushing. Generic code must call our tlb_flush + * even on empty ranges in this case. + * + * This will always be false for hash. + */ +#define arch_tlb_mustflush(tlb) (tlb->need_flush_all) + +#endif + extern void tlb_flush(struct mmu_gather *tlb); /* Get the generic bits... */ diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 1135b43a597c..238b20a513e7 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -862,6 +862,16 @@ void radix__tlb_flush(struct mmu_gather *tlb) unsigned long start = tlb->start; unsigned long end = tlb->end; + /* + * This can happen if need_flush_all is set due to a page table + * invalidate, but no ptes under it freed (see arch_tlb_mustflush). + * Set end = start to prevent any TLB flushing here (only PWC). + */ + if (!end) { + WARN_ON_ONCE(!tlb->need_flush_all); + end = start; + } + /* * if page size is not something we understand, do a full mm flush * diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b320c0cc8996..a55ef1425f0d 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -285,6 +285,11 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, * http://lkml.kernel.org/r/CA+55aFzBggoXtNXQeng5d_mRoDnaMBE5Y+URs+PHR67nUpMtaw@mail.gmail.com * * For now w.r.t page table cache, mark the range_size as PAGE_SIZE + * + * Update: powerpc (Book3S 64-bit, radix MMU) has an architected page table + * cache (called PWC), and invalidates it specifically. It sets the + * need_flush_all flag to indicate the PWC requires flushing, so it defines + * its own p??_free_tlb functions which do not expand the TLB range. */ #ifndef pte_free_tlb