From patchwork Wed Nov 10 22:11:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 12613219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31503C433F5 for ; Wed, 10 Nov 2021 22:11:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D59776112F for ; Wed, 10 Nov 2021 22:11:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D59776112F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 67E366B0074; Wed, 10 Nov 2021 17:11:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 607B16B00CD; Wed, 10 Nov 2021 17:11:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AA4D6B00CF; Wed, 10 Nov 2021 17:11:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0151.hostedemail.com [216.40.44.151]) by kanga.kvack.org (Postfix) with ESMTP id 3484A6B0074 for ; Wed, 10 Nov 2021 17:11:25 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EAC1E82D35F2 for ; Wed, 10 Nov 2021 22:11:24 +0000 (UTC) X-FDA: 78794417562.19.9A2E686 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 8681F801AB07 for ; Wed, 10 Nov 2021 22:11:24 +0000 (UTC) Received: by mail-pg1-f201.google.com with SMTP id x14-20020a63cc0e000000b002a5bc462947so2166718pgf.20 for ; Wed, 10 Nov 2021 14:11:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:cc; bh=v8H09r/rEbSUS7qLFj/cfoJge3XtWP5j00NAM9JQP/A=; b=nusR9Rb3OkVDT1Bifyl87B+zxqQGPPKcs8suHSsmgd3VeoicUED4dcmA1mN0QN4/+N OCKKSWVcqWYlAH3CbXlZFpLfOyW58ky23J3jpJMJoP3YruUDtZmdXacxQ6AXOCJatpDn 4eXiqzjSLdR5x4jfzGyPc/ZlVHE4uReCGr8qNQTGLltJKHeX95WTUhLU6YKhVZ95XqUW qBy4zuP7sLkIW4f9D+FPz/UjlWMcIgWajqldCY16ZcwPnzsZlc9eIwMht9hcigQMhMh3 b/KsRhsDaBBbIEcmS0aGsriHzPyDUhxmdAODNmSopOryDCcUl3wZrqK2Okih1EIi1eB4 88xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:cc; bh=v8H09r/rEbSUS7qLFj/cfoJge3XtWP5j00NAM9JQP/A=; b=Y8yabFKp1rHQCjzbGkqYsDACTMKbpRHlKMJ/4RNNqb5gg5HYHGYekfbbhPUCI2ZCpA ZPZ3I72n2pKuYglsCcQHtifQ0hIynFaReMFlzKhr+N6J7luE8C+MfWRUhQI5fJW86BhK jZU9SwUIk3eAHw1XLCHkFNq3I6m0J+SmFyzJsQDZWB2839k8cRgGCXJPBEvNR02MDc7c bv2OnvBKLQt7HIqS7MbJuFmCaB5/mQNIMdHczoKnem4Uf+7z5caXIb4GHyMMJIMQKJ1K r0pHlBpELY+nMWpxKzo2P2C6Ye18/8JbcV6v0nLEHNpW6q8aKFx2ftNg+JogWW4VHVNn kzHg== X-Gm-Message-State: AOAM533wO8/g5Tn9+xmdUGG5rmCpamtJ/W8KmbwF4IOVDknYK8raDjDI 85fHqBNejBO5qbkM+mQSa4nvXRavz3kydXKrOQ== X-Google-Smtp-Source: ABdhPJybvdFJpiMChKdShyThUJHkZm5Fi3ii66PueltUfyE9ZGbCvH7JaR2k7AFvI21B4kp62P3Pt4fRqIjU/ejNNA== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2cd:202:59c8:7b4e:e859:9db0]) (user=almasrymina job=sendgmr) by 2002:a17:902:ea05:b0:141:c6c8:824c with SMTP id s5-20020a170902ea0500b00141c6c8824cmr2277755plg.86.1636582283375; Wed, 10 Nov 2021 14:11:23 -0800 (PST) Date: Wed, 10 Nov 2021 14:11:20 -0800 Message-Id: <20211110221120.3833294-1-almasrymina@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.34.0.rc0.344.g81b53c2807-goog Subject: [PATCH v5] mm: Add PM_HUGE_THP_MAPPING to /proc/pid/pagemap From: Mina Almasry Cc: Mina Almasry , David Hildenbrand , Matthew Wilcox , "Paul E . McKenney" , Yu Zhao , Jonathan Corbet , Andrew Morton , Peter Xu , Ivan Teterevkov , Florian Schmidt , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8681F801AB07 X-Stat-Signature: hi479kc5gmw4pw5o9647e4bk79p4je4m Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nusR9Rb3; spf=pass (imf06.hostedemail.com: domain of 3i0OMYQsKCFYy9AyGFMA6By4CC492.0CA96BIL-AA8Jy08.CF4@flex--almasrymina.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3i0OMYQsKCFYy9AyGFMA6By4CC492.0CA96BIL-AA8Jy08.CF4@flex--almasrymina.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1636582284-469859 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add PM_HUGE_THP MAPPING to allow userspace to detect whether a given virt address is currently mapped by a transparent huge page or not. Example use case is a process requesting THPs from the kernel (via a huge tmpfs mount for example), for a performance critical region of memory. The userspace may want to query whether the kernel is actually backing this memory by hugepages or not. PM_HUGE_THP_MAPPING bit is set if the virt address is mapped at the PMD level and the underlying page is a transparent huge page. A few options were considered: 1. Add /proc/pid/pageflags that exports the same info as /proc/kpageflags. This is not appropriate because many kpageflags are inappropriate to expose to userspace processes. 2. Simply get this info from the existing /proc/pid/smaps interface. There are a couple of issues with that: 1. /proc/pid/smaps output is human readable and unfriendly to programatically parse. 2. /proc/pid/smaps is slow. The cost of reading /proc/pid/smaps into userspace buffers is about ~800us per call, and this doesn't include parsing the output to get the information you need. The cost of querying 1 virt address in /proc/pid/pagemaps however is around 5-7us. Tested manually by adding logging into transhuge-stress, and by allocating THP and querying the PM_HUGE_THP_MAPPING flag at those virtual addresses. Signed-off-by: Mina Almasry Cc: David Hildenbrand Cc: Matthew Wilcox Cc: David Rientjes rientjes@google.com Cc: Paul E. McKenney Cc: Yu Zhao Cc: Jonathan Corbet Cc: Andrew Morton Cc: Peter Xu Cc: Ivan Teterevkov Cc: Florian Schmidt Cc: linux-kernel@vger.kernel.org Cc: linux-fsdevel@vger.kernel.org Cc: linux-mm@kvack.org --- Changes in v5: - Added justification for this interface in the commit message! Changes in v4: - Removed unnecessary moving of flags variable declaration Changes in v3: - Renamed PM_THP to PM_HUGE_THP_MAPPING - Fixed checks to set PM_HUGE_THP_MAPPING - Added PM_HUGE_THP_MAPPING docs --- Documentation/admin-guide/mm/pagemap.rst | 3 ++- fs/proc/task_mmu.c | 3 +++ tools/testing/selftests/vm/transhuge-stress.c | 21 +++++++++++++++---- 3 files changed, 22 insertions(+), 5 deletions(-) -- 2.34.0.rc0.344.g81b53c2807-goog diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index fdc19fbc10839..8a0f0064ff336 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -23,7 +23,8 @@ There are four components to pagemap: * Bit 56 page exclusively mapped (since 4.2) * Bit 57 pte is uffd-wp write-protected (since 5.13) (see :ref:`Documentation/admin-guide/mm/userfaultfd.rst `) - * Bits 57-60 zero + * Bit 58 page is a huge (PMD size) THP mapping + * Bits 59-60 zero * Bit 61 page is file-page or shared-anon (since 3.5) * Bit 62 page swapped * Bit 63 page present diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index ad667dbc96f5c..6f1403f83b310 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1302,6 +1302,7 @@ struct pagemapread { #define PM_SOFT_DIRTY BIT_ULL(55) #define PM_MMAP_EXCLUSIVE BIT_ULL(56) #define PM_UFFD_WP BIT_ULL(57) +#define PM_HUGE_THP_MAPPING BIT_ULL(58) #define PM_FILE BIT_ULL(61) #define PM_SWAP BIT_ULL(62) #define PM_PRESENT BIT_ULL(63) @@ -1456,6 +1457,8 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, if (page && page_mapcount(page) == 1) flags |= PM_MMAP_EXCLUSIVE; + if (page && is_transparent_hugepage(page)) + flags |= PM_HUGE_THP_MAPPING; for (; addr != end; addr += PAGE_SIZE) { pagemap_entry_t pme = make_pme(frame, flags); diff --git a/tools/testing/selftests/vm/transhuge-stress.c b/tools/testing/selftests/vm/transhuge-stress.c index fd7f1b4a96f94..7dce18981fff5 100644 --- a/tools/testing/selftests/vm/transhuge-stress.c +++ b/tools/testing/selftests/vm/transhuge-stress.c @@ -16,6 +16,12 @@ #include #include +/* + * We can use /proc/pid/pagemap to detect whether the kernel was able to find + * hugepages or no. This can be very noisy, so is disabled by default. + */ +#define NO_DETECT_HUGEPAGES + #define PAGE_SHIFT 12 #define HPAGE_SHIFT 21 @@ -23,6 +29,7 @@ #define HPAGE_SIZE (1 << HPAGE_SHIFT) #define PAGEMAP_PRESENT(ent) (((ent) & (1ull << 63)) != 0) +#define PAGEMAP_THP(ent) (((ent) & (1ull << 58)) != 0) #define PAGEMAP_PFN(ent) ((ent) & ((1ull << 55) - 1)) int pagemap_fd; @@ -47,10 +54,16 @@ int64_t allocate_transhuge(void *ptr) (uintptr_t)ptr >> (PAGE_SHIFT - 3)) != sizeof(ent)) err(2, "read pagemap"); - if (PAGEMAP_PRESENT(ent[0]) && PAGEMAP_PRESENT(ent[1]) && - PAGEMAP_PFN(ent[0]) + 1 == PAGEMAP_PFN(ent[1]) && - !(PAGEMAP_PFN(ent[0]) & ((1 << (HPAGE_SHIFT - PAGE_SHIFT)) - 1))) - return PAGEMAP_PFN(ent[0]); + if (PAGEMAP_PRESENT(ent[0]) && PAGEMAP_PRESENT(ent[1])) { +#ifndef NO_DETECT_HUGEPAGES + if (!PAGEMAP_THP(ent[0])) + fprintf(stderr, "WARNING: detected non THP page\n"); +#endif + if (PAGEMAP_PFN(ent[0]) + 1 == PAGEMAP_PFN(ent[1]) && + !(PAGEMAP_PFN(ent[0]) & + ((1 << (HPAGE_SHIFT - PAGE_SHIFT)) - 1))) + return PAGEMAP_PFN(ent[0]); + } return -1; }