From patchwork Mon Jul 2 00:56:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: john.hubbard@gmail.com X-Patchwork-Id: 10500281 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 51D1D602D6 for ; Mon, 2 Jul 2018 00:57:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E69A28736 for ; Mon, 2 Jul 2018 00:57:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 313B72876C; Mon, 2 Jul 2018 00:57:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C067728736 for ; Mon, 2 Jul 2018 00:57:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D2DC6B000C; Sun, 1 Jul 2018 20:57:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 032636B000D; Sun, 1 Jul 2018 20:57:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCB8D6B000E; Sun, 1 Jul 2018 20:57:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot0-f197.google.com (mail-ot0-f197.google.com [74.125.82.197]) by kanga.kvack.org (Postfix) with ESMTP id A14956B000C for ; Sun, 1 Jul 2018 20:57:41 -0400 (EDT) Received: by mail-ot0-f197.google.com with SMTP id n10-v6so11172266otl.2 for ; Sun, 01 Jul 2018 17:57:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=uQq7Rhh2IoBOG5dCp0+cMTRTysWrcwIj4JspKeAqpgw=; b=IWNv4HShtJ3TfRXgtJyN8fXoXQARe6eZlWsdlAbqtfILlpoq50LbNUd8HSRnQnYjus EzlHLog2ZyDuW6+xyiHoku2moru8T4SzaenffSF2kfdOIrb9tfoEQJM8Nf87BqpfE5gQ SRKt7MMXFgqthIHez6YlfgGQq7xp1HfJllRkIcN+va0jAP8btaMqM8TrS3FSye6cOVkA t0Tl8jBU+KWli2pD0fAe+WvzC/p7WhxUAUrN9mJ+xjV9FFj/lZ2cpVffhe+39Xp9/gpI 7z1JtwqE+9bCS0MIZ2z4Gw8wo0xqGN8paoBYF8Xn6MXA9gi50t7kIDoR94584PXX7rta sCnw== X-Gm-Message-State: APt69E2fuBMSxmyt8uXJyeVXYbnTht3x5v/gP2X4Up82FVSGDQUA8cKA 8HtoHxjs5v4zaWl644wR4oVnCzj4y2Si6Ic0jo8+FIpPisv/0Fo4FyLHrjnptgNesCYyJeWsBxz I0WO8q3eXDjEK17/B4vbDkMM/xb8Tg+i9p9M2Ql56b2Wbyypp1lmGRObuLUmiAM9RF3KlJaZ8sd yXCELaMSpGLv90gWyT2juzj4HF97ujAl5QxI6kpzSvdfppu2gdSJ6Dq3TH7089d1CEsJ1hNezNR +naNyYbLRfS/rVuVhcqY0sxP3o0wf/1RqXEPw7sA/VO2PqR4N5mFi3IshSuhhYGs+mVnapjvhK9 drKR42hgswlDgmROKIAG/ANpqyyDNNMU3l8JgU+K5kGfEZX8y8ABOxvbfZANWlORHGtlShQLAep w X-Received: by 2002:aca:745:: with SMTP id 66-v6mr12737879oih.295.1530493061432; Sun, 01 Jul 2018 17:57:41 -0700 (PDT) X-Received: by 2002:aca:745:: with SMTP id 66-v6mr12737849oih.295.1530493060520; Sun, 01 Jul 2018 17:57:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530493060; cv=none; d=google.com; s=arc-20160816; b=JffV19w6bxnTxNallDWi5uateUNMNu4MQ1v21lcske0fPnddvtzS6G6dTKM747GpZp x5JT5Wlrt4klU3ey53cNzmlGvgv/dDkXybG2zQnKqEsyiidjQ3hpsADqnjoS+KU/Rw38 lLxVqh4wIa4brrGDvuLgL0Q9/tzXEBYa6mxjEAdyyq161Fke2noCm/HKAySk5/v1Q53T OrQHhOreCtITJDK+zqUKG8e/yRfTQ5akTP4gIuhxbbyXwbAWOiAwMcs1xMeZ9Rd6KGwX 8x+ywiSrMNaT2lj/g2FiNd78w3OQbhDQoZfPZ9FnXSMuzbTqDtFrxBydKIWqkMy4yEVR YrCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=uQq7Rhh2IoBOG5dCp0+cMTRTysWrcwIj4JspKeAqpgw=; b=G+BCwzeKN68kf7rdEFkBhCMeqbLU5RJ/97OLHh5KvqZNmbBDF8DD6hqZJBiR/m+RWh RByjYE5g3c/7RSwyt1fqYDVjiZxT6YYJ/HHP/8Rf/yuu1581J90NwrT6XUQ/wRcNEf5G OvwrH8iJBZENlkw5pTs3kac8qH2l6d0mmjMuawfXXRQ8+s/uQ2bWJQmjYDEW/onDLi1C RUJWIydbjSg6K5koRz4QtwrJm20hjNQijL2KldEMW7c70t+2CVP9ijz9tQedLCIBw/92 ttdsd8vd2hvN+lJhfI4+yYjbG08Yx8QcDEuUTRixLnHay9VtYLDPn6uBW7v4RE7994gB uITg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R7NKw7Mj; spf=pass (google.com: domain of john.hubbard@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.hubbard@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x12-v6sor545591oie.29.2018.07.01.17.57.40 for (Google Transport Security); Sun, 01 Jul 2018 17:57:40 -0700 (PDT) Received-SPF: pass (google.com: domain of john.hubbard@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R7NKw7Mj; spf=pass (google.com: domain of john.hubbard@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=john.hubbard@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uQq7Rhh2IoBOG5dCp0+cMTRTysWrcwIj4JspKeAqpgw=; b=R7NKw7MjzwO2OAqPIqAKIb+0PZqdQgxMlT/+wbZ+5gPFLwCcqC9eAgg2KW5YgQT5nr 0rPcxMA9dA5mU+0Xds4payqHHKyaxEyjuOpNqGaNwyIbj+l+OFsHqL/aTnPzwLJaho1a /ZJvVa2SFD7o4pOnzI2EbaSEIuMfQrDSBkpBssZqRM3cUG8xQKjDChb3iHsVkrIpEB0j chJW3e+biapq1UMMntmhfjmG0zAvkiQ3BrrLx497Cy6Q4+9fdZRXVHHghj50apibuluT 7MmhPuFGnZo+uFsDFm/RmLU159Duko+IY1JXwPburS2vFn2TRyqr+wJDiFxfohXbX6Re jYWA== X-Google-Smtp-Source: AAOMgpdg+glJKExTO1EfoZ4DYd7+hfIDaB2gFDtabtdRxzXfyF0VL0cWDVe3gZSE9hDK0mPjFiEaRw== X-Received: by 2002:aca:5c46:: with SMTP id q67-v6mr12867959oib.17.1530493060311; Sun, 01 Jul 2018 17:57:40 -0700 (PDT) Received: from sandstorm.nvidia.com ([2600:1700:43b0:3120:feaa:14ff:fe9e:34cb]) by smtp.gmail.com with ESMTPSA id v6-v6sm4111672oix.30.2018.07.01.17.57.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 01 Jul 2018 17:57:39 -0700 (PDT) From: john.hubbard@gmail.com X-Google-Original-From: jhubbard@nvidia.com To: Matthew Wilcox , Michal Hocko , Christopher Lameter , Jason Gunthorpe , Dan Williams , Jan Kara Cc: linux-mm@kvack.org, LKML , linux-rdma , linux-fsdevel@vger.kernel.org, John Hubbard Subject: [PATCH v2 3/6] mm: introduce zone_gup_lock, for dma-pinned pages Date: Sun, 1 Jul 2018 17:56:51 -0700 Message-Id: <20180702005654.20369-4-jhubbard@nvidia.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180702005654.20369-1-jhubbard@nvidia.com> References: <20180702005654.20369-1-jhubbard@nvidia.com> X-NVConfidentiality: public X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: John Hubbard The page->dma_pinned_flags and _count fields require lock protection. A lock at approximately the granularity of the zone_lru_lock is called for, but adding to the locking contention of zone_lru_lock is undesirable, because that is a pre-existing hot spot. Fortunately, these new dma_pinned_* fields can use an independent lock, so this patch creates an entirely new lock, right next to the zone_lru_lock. Why "zone_gup_lock"? Most of the naming refers to "DMA-pinned pages", but "zone DMA lock" has other meanings already, so this is called zone_gup_lock instead. The "dma pinning" is a result of get_user_pages (gup) being called, so the name still helps explain its use. Signed-off-by: John Hubbard --- include/linux/mmzone.h | 7 +++++++ mm/page_alloc.c | 1 + 2 files changed, 8 insertions(+) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 32699b2dc52a..5b4ceef82657 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -662,6 +662,8 @@ typedef struct pglist_data { int kswapd_failures; /* Number of 'reclaimed == 0' runs */ + spinlock_t pinned_dma_lock; + #ifdef CONFIG_COMPACTION int kcompactd_max_order; enum zone_type kcompactd_classzone_idx; @@ -740,6 +742,11 @@ static inline spinlock_t *zone_lru_lock(struct zone *zone) return &zone->zone_pgdat->lru_lock; } +static inline spinlock_t *zone_gup_lock(struct zone *zone) +{ + return &zone->zone_pgdat->pinned_dma_lock; +} + static inline struct lruvec *node_lruvec(struct pglist_data *pgdat) { return &pgdat->lruvec; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1521100f1e63..9c493442b57c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6211,6 +6211,7 @@ static void __paginginit free_area_init_core(struct pglist_data *pgdat) int nid = pgdat->node_id; pgdat_resize_init(pgdat); + spin_lock_init(&pgdat->pinned_dma_lock); #ifdef CONFIG_NUMA_BALANCING spin_lock_init(&pgdat->numabalancing_migrate_lock); pgdat->numabalancing_migrate_nr_pages = 0;