From patchwork Fri May 4 18:33:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 10381363 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 99CF86037D for ; Fri, 4 May 2018 18:59:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 885BE29572 for ; Fri, 4 May 2018 18:59:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7CC8E29589; Fri, 4 May 2018 18:59:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E33B029572 for ; Fri, 4 May 2018 18:59:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 036406B026D; Fri, 4 May 2018 14:33:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F01156B026B; Fri, 4 May 2018 14:33:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C18196B026D; Fri, 4 May 2018 14:33:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf0-f197.google.com (mail-pf0-f197.google.com [209.85.192.197]) by kanga.kvack.org (Postfix) with ESMTP id 7AA796B0269 for ; Fri, 4 May 2018 14:33:24 -0400 (EDT) Received: by mail-pf0-f197.google.com with SMTP id w7so14881924pfd.9 for ; Fri, 04 May 2018 11:33:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Jc5MwzrDjk8ce/4bpqUP2/d4GKMd/RUIjoXKolUQisM=; b=Md7UI7C/0PYk3gAWvEp+vRdEwPLGFYzmKxrhr7LscGSZYOuZy32Ls3/+SmCRhY8WJW 87D3mfgS5wofb71wudur+k27Yi6uPPObUDBnuXZo9MhuNbd/IjpR8chIL+Zfvq0dcaOT RWZPnci88bQc6Dw6G37k9ptTyUCL0yvEA8BXb3EuGtY5fK1LvH5Dy2GvuD7GQWxD50y4 OuvGV7HtkVW5MGOpUvJ/+Ab1WOp9zl1iY8euiSPcVAdoRx6qlsNRpEWiqVQJvpgKr6QW jno8pRfavZNdvZVp9joV2iQB1YzbdwB8Dbtfd9/DYkUFpFueWLTXn2f1FWrLz19unsLW KS0Q== X-Gm-Message-State: ALQs6tAKHoHpaOSPDAK43y4pWEuRUgy+nCUml96Xx4PJasx6yKRCsmJf prjwP77SPi3W6ZTgbGwGVLqViZC+toiFnkcvyK/RlWGD3Dsa0Q2DIN4EfZ00hUWPa6eC7nhBPVw 6eVd7UbxecS+LAFFlAvaFtRSjEuvdMUwJOMBaCee39K/sHabhBbrnfXsOGHKMuSBpMg== X-Received: by 2002:a17:902:7488:: with SMTP id h8-v6mr29180921pll.124.1525458804189; Fri, 04 May 2018 11:33:24 -0700 (PDT) X-Google-Smtp-Source: AB8JxZoYiPjUKkqyAis598bfhhLaxemqFyIXEHeqTwl+er0dy+8PsVfX86gYYef4yruINYPyvQmu X-Received: by 2002:a17:902:7488:: with SMTP id h8-v6mr29180855pll.124.1525458802516; Fri, 04 May 2018 11:33:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525458802; cv=none; d=google.com; s=arc-20160816; b=fBkIf70FkZKKyMXqxRN4B80bI8NOHqwPzNpq33wpVbBDt4U1kGPDGviPmH5nU78MCc mXIPeR5gQ3raJJoV+IrSczeIHvgNAE6x2pytuOI7J/dm0H82M6QSRbWejMJmpejqkjKj Kn6xxtyOlS5U12wBavAibkOx+p4RSgI47FDu4f9EoOuImTUytYn3iRywljCteAVmGFN2 mJHZOqOqGqXby5N7c3C6ixUm/T6J/sQwOYjFYFJcs7qcbXr/7Fn5pdl44dk4C6JibFDq gAjQb/gErZX7CBiQk3CwxxxnOoNodPPZrgx8PCVbRwUwuWz/1nv1AmJ7fkfL98jqIGnu 1tZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=Jc5MwzrDjk8ce/4bpqUP2/d4GKMd/RUIjoXKolUQisM=; b=zriiFJXETPoubUnM7OdAKFjZbKKi1czvXIHvd8NGOIvVkz0DqrtrZp7l6iBMNeoItY zl7t9f/OSOlbfeKrzGWukQbwjcFEiVILe07wqjNco9QGQPxGIKyzIBjwoF8YkqCL46o5 HLdAO5Pd9KLL0dvYjff2ZKSJAgNahkwP5KwFauYWfu/qbCQMCydQEI0kKYH58bjB4KmX Ck4nPb7UKVrbn9Zbb++z2VE9j39/tkEn/tsw93fmzuJ5x9EUsePK/wE2oc3Dl0frhzpo 3AW/htDeDqxl2GJEzTNZTGoTq87cmXH2A0hz7+pRMZYay4uqw713H/wi+qU18oZVG10L mu9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=BZFIlelI; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org Received: from bombadil.infradead.org (bombadil.infradead.org. [2607:7c80:54:e::133]) by mx.google.com with ESMTPS id z7-v6si6903345pgv.614.2018.05.04.11.33.22 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 04 May 2018 11:33:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) client-ip=2607:7c80:54:e::133; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=BZFIlelI; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=Jc5MwzrDjk8ce/4bpqUP2/d4GKMd/RUIjoXKolUQisM=; b=BZFIlelIE8sNHEUplSxigHUD8 WoTIgrSSMC7luAfDHEyemv36n0sD8pGowJ9nF+aTJaC4iTCFVaLAYvYHQj3cq11W3GmwWBuRbtC6q yp1BiWdXWvPUWI8qNpr1hoTUoSpBg8f/Nd0tNjZt4uIFMnx5BVgxPJyW3WnIUONgYeCyhhTyRNPSt Y4CnGsXfNzH+ef6/t0AcVhVWlUgHdNP0606m8RPOzCnYbFsUXfZjquvP50itIPdtk51EXa+t7aUxu 03WmKx4QC+KVwSJZUBTl2JmlT9tul1XuEEUE+bvMOf5NGQ6eiJU+XBq6d4kZSCeIiWqIDFtO788Ec i6xWz1NDg==; Received: from willy by bombadil.infradead.org with local (Exim 4.90_1 #2 (Red Hat Linux)) id 1fEfWC-0003lY-Cw; Fri, 04 May 2018 18:33:20 +0000 From: Matthew Wilcox To: linux-mm@kvack.org Cc: Matthew Wilcox , Andrew Morton , "Kirill A . Shutemov" , Christoph Lameter , Lai Jiangshan , Pekka Enberg , Vlastimil Babka , Dave Hansen , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= Subject: [PATCH v5 01/17] s390: Use _refcount for pgtables Date: Fri, 4 May 2018 11:33:02 -0700 Message-Id: <20180504183318.14415-2-willy@infradead.org> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180504183318.14415-1-willy@infradead.org> References: <20180504183318.14415-1-willy@infradead.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Matthew Wilcox s390 borrows the storage used for _mapcount in struct page in order to account whether the bottom or top half is being used for 2kB page tables. I want to use that for something else, so use the top byte of _refcount instead of the bottom byte of _mapcount. _refcount may temporarily be incremented by other CPUs that see a stale pointer to this page in the page cache, but each CPU can only increment it by one, and there are no systems with 2^24 CPUs today, so they will not change the upper byte of _refcount. We do have to be a little careful not to lose any of their writes (as they will subsequently decrement the counter). Signed-off-by: Matthew Wilcox Acked-by: Martin Schwidefsky --- arch/s390/mm/pgalloc.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c index 562f72955956..84bd6329a88d 100644 --- a/arch/s390/mm/pgalloc.c +++ b/arch/s390/mm/pgalloc.c @@ -190,14 +190,15 @@ unsigned long *page_table_alloc(struct mm_struct *mm) if (!list_empty(&mm->context.pgtable_list)) { page = list_first_entry(&mm->context.pgtable_list, struct page, lru); - mask = atomic_read(&page->_mapcount); + mask = atomic_read(&page->_refcount) >> 24; mask = (mask | (mask >> 4)) & 3; if (mask != 3) { table = (unsigned long *) page_to_phys(page); bit = mask & 1; /* =1 -> second 2K */ if (bit) table += PTRS_PER_PTE; - atomic_xor_bits(&page->_mapcount, 1U << bit); + atomic_xor_bits(&page->_refcount, + 1U << (bit + 24)); list_del(&page->lru); } } @@ -218,12 +219,12 @@ unsigned long *page_table_alloc(struct mm_struct *mm) table = (unsigned long *) page_to_phys(page); if (mm_alloc_pgste(mm)) { /* Return 4K page table with PGSTEs */ - atomic_set(&page->_mapcount, 3); + atomic_xor_bits(&page->_refcount, 3 << 24); memset64((u64 *)table, _PAGE_INVALID, PTRS_PER_PTE); memset64((u64 *)table + PTRS_PER_PTE, 0, PTRS_PER_PTE); } else { /* Return the first 2K fragment of the page */ - atomic_set(&page->_mapcount, 1); + atomic_xor_bits(&page->_refcount, 1 << 24); memset64((u64 *)table, _PAGE_INVALID, 2 * PTRS_PER_PTE); spin_lock_bh(&mm->context.lock); list_add(&page->lru, &mm->context.pgtable_list); @@ -242,7 +243,8 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) /* Free 2K page table fragment of a 4K page */ bit = (__pa(table) & ~PAGE_MASK)/(PTRS_PER_PTE*sizeof(pte_t)); spin_lock_bh(&mm->context.lock); - mask = atomic_xor_bits(&page->_mapcount, 1U << bit); + mask = atomic_xor_bits(&page->_refcount, 1U << (bit + 24)); + mask >>= 24; if (mask & 3) list_add(&page->lru, &mm->context.pgtable_list); else @@ -253,7 +255,6 @@ void page_table_free(struct mm_struct *mm, unsigned long *table) } pgtable_page_dtor(page); - atomic_set(&page->_mapcount, -1); __free_page(page); } @@ -274,7 +275,8 @@ void page_table_free_rcu(struct mmu_gather *tlb, unsigned long *table, } bit = (__pa(table) & ~PAGE_MASK) / (PTRS_PER_PTE*sizeof(pte_t)); spin_lock_bh(&mm->context.lock); - mask = atomic_xor_bits(&page->_mapcount, 0x11U << bit); + mask = atomic_xor_bits(&page->_refcount, 0x11U << (bit + 24)); + mask >>= 24; if (mask & 3) list_add_tail(&page->lru, &mm->context.pgtable_list); else @@ -296,12 +298,13 @@ static void __tlb_remove_table(void *_table) break; case 1: /* lower 2K of a 4K page table */ case 2: /* higher 2K of a 4K page table */ - if (atomic_xor_bits(&page->_mapcount, mask << 4) != 0) + mask = atomic_xor_bits(&page->_refcount, mask << (4 + 24)); + mask >>= 24; + if (mask != 0) break; /* fallthrough */ case 3: /* 4K page table with pgstes */ pgtable_page_dtor(page); - atomic_set(&page->_mapcount, -1); __free_page(page); break; }