From patchwork Thu Oct 22 22:53:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11852089 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 41CA161C for ; Thu, 22 Oct 2020 22:53:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 73EE5241A6 for ; Thu, 22 Oct 2020 22:53:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="j0ejsR2Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73EE5241A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 78C446B005D; Thu, 22 Oct 2020 18:53:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 73D7A6B0062; Thu, 22 Oct 2020 18:53:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 652C06B0068; Thu, 22 Oct 2020 18:53:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0075.hostedemail.com [216.40.44.75]) by kanga.kvack.org (Postfix) with ESMTP id 3A42D6B005D for ; Thu, 22 Oct 2020 18:53:22 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D1BC6180AD801 for ; Thu, 22 Oct 2020 22:53:21 +0000 (UTC) X-FDA: 77401064202.04.clam20_62126bc27254 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id B2D8B800ABD1 for ; Thu, 22 Oct 2020 22:53:21 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=7564a8c327=guro@fb.com,,RULES_HIT:30012:30029:30034:30036:30054:30064:30091,0,RBL:67.231.153.30:@fb.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04ygz6ito5cq3c9pcqh99uj7c5zhjypxzwyt1bktw7dpofwgpkgpknipe11riw5.ymwr3qxmuqrxuqr65qfnim4f66ykaw8fse9x4qcbqywd9yafa7339jsy1udmdic.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: clam20_62126bc27254 X-Filterd-Recvd-Size: 9245 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Thu, 22 Oct 2020 22:53:21 +0000 (UTC) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MMoZXK018530 for ; Thu, 22 Oct 2020 15:53:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=Yr1xrojJjtI4pSCSS8C4LN1/v3pKcraNnqvQv1M5jcE=; b=j0ejsR2Zy2sL003ci/clcfhthc9j1bpRT+AArNLykKxYFgEC3vV8SfJ+JGqf9zVkvEGI yxra5QFllhqGGCF9URPz6iHD6LKQC+pSU6rOSEldCM+tKV+0mDCgOMHmxrlqn6kFxYPG BzDnNYiW3sZV8SeCbg151aBM0XvnTuA+/iI= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 34b07se0a9-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 22 Oct 2020 15:53:20 -0700 Received: from intmgw002.41.prn1.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 22 Oct 2020 15:53:16 -0700 Received: by devvm1755.vll0.facebook.com (Postfix, from userid 111017) id E94601A3AE4A; Thu, 22 Oct 2020 15:53:12 -0700 (PDT) From: Roman Gushchin To: Andrew Morton CC: Zi Yan , Joonsoo Kim , Mike Kravetz , , , , , Roman Gushchin Subject: [PATCH v1 1/2] mm: cma: introduce cma_release_nowait() Date: Thu, 22 Oct 2020 15:53:07 -0700 Message-ID: <20201022225308.2927890-2-guro@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20201022225308.2927890-1-guro@fb.com> References: <20201022225308.2927890-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_17:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 phishscore=0 mlxlogscore=999 spamscore=0 bulkscore=0 impostorscore=0 malwarescore=0 mlxscore=0 adultscore=0 priorityscore=1501 suspectscore=2 lowpriorityscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220146 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: cma_release() has to lock the cma_lock mutex to clear the cma bitmap. It makes it a blocking function, which complicates its usage from non-blocking contexts. For instance, hugetlbfs code is temporarily dropping the hugetlb_lock spinlock to call cma_release(). This patch introduces a non-blocking cma_release_nowait(), which postpones the cma bitmap clearance. It's done later from a work context. The first page in the cma allocation is used to store the work struct. Because CMA allocations and de-allocations are usually not that frequent, a single global workqueue is used. To make sure that subsequent cma_alloc() call will pass, cma_alloc() flushes the cma_release_wq workqueue. To avoid a performance regression in the case when only cma_release() is used, gate it by a per-cma area flag, which is set by the first call of cma_release_nowait(). Signed-off-by: Roman Gushchin --- include/linux/cma.h | 2 + mm/cma.c | 93 +++++++++++++++++++++++++++++++++++++++++++++ mm/cma.h | 5 +++ 3 files changed, 100 insertions(+) diff --git a/include/linux/cma.h b/include/linux/cma.h index 217999c8a762..497eca478c2f 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -47,6 +47,8 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, extern struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align, bool no_warn); extern bool cma_release(struct cma *cma, const struct page *pages, unsigned int count); +extern bool cma_release_nowait(struct cma *cma, const struct page *pages, + unsigned int count); extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data); #endif diff --git a/mm/cma.c b/mm/cma.c index 7f415d7cda9f..9fcfddcf1a6c 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -36,10 +36,19 @@ #include "cma.h" +struct cma_clear_bitmap_work { + struct work_struct work; + struct cma *cma; + unsigned long pfn; + unsigned int count; +}; + struct cma cma_areas[MAX_CMA_AREAS]; unsigned cma_area_count; static DEFINE_MUTEX(cma_mutex); +struct workqueue_struct *cma_release_wq; + phys_addr_t cma_get_base(const struct cma *cma) { return PFN_PHYS(cma->base_pfn); @@ -148,6 +157,10 @@ static int __init cma_init_reserved_areas(void) for (i = 0; i < cma_area_count; i++) cma_activate_area(&cma_areas[i]); + cma_release_wq = create_workqueue("cma_release"); + if (!cma_release_wq) + return -ENOMEM; + return 0; } core_initcall(cma_init_reserved_areas); @@ -205,6 +218,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, cma->base_pfn = PFN_DOWN(base); cma->count = size >> PAGE_SHIFT; + cma->flags = 0; cma->order_per_bit = order_per_bit; *res_cma = cma; cma_area_count++; @@ -437,6 +451,14 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align, return NULL; for (;;) { + /* + * If the CMA bitmap is cleared asynchronously after + * cma_release_nowait(), cma release workqueue has to be + * flushed here in order to make the allocation succeed. + */ + if (test_bit(CMA_DELAYED_RELEASE, &cma->flags)) + flush_workqueue(cma_release_wq); + mutex_lock(&cma->lock); bitmap_no = bitmap_find_next_zero_area_off(cma->bitmap, bitmap_maxno, start, bitmap_count, mask, @@ -528,6 +550,77 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count) return true; } +static void cma_clear_bitmap_fn(struct work_struct *work) +{ + struct cma_clear_bitmap_work *w; + + w = container_of(work, struct cma_clear_bitmap_work, work); + + cma_clear_bitmap(w->cma, w->pfn, w->count); + + __free_page(pfn_to_page(w->pfn)); +} + +/** + * cma_release_nowait() - release allocated pages without blocking + * @cma: Contiguous memory region for which the allocation is performed. + * @pages: Allocated pages. + * @count: Number of allocated pages. + * + * Similar to cma_release(), this function releases memory allocated + * by cma_alloc(), but unlike cma_release() is non-blocking and can be + * called from an atomic context. + * It returns false when provided pages do not belong to contiguous area + * and true otherwise. + */ +bool cma_release_nowait(struct cma *cma, const struct page *pages, + unsigned int count) +{ + struct cma_clear_bitmap_work *work; + unsigned long pfn; + + if (!cma || !pages) + return false; + + pr_debug("%s(page %p)\n", __func__, (void *)pages); + + pfn = page_to_pfn(pages); + + if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) + return false; + + VM_BUG_ON(pfn + count > cma->base_pfn + cma->count); + + /* + * Set CMA_DELAYED_RELEASE flag: subsequent cma_alloc()'s + * will wait for the async part of cma_release_nowait() to + * finish. + */ + if (unlikely(!test_bit(CMA_DELAYED_RELEASE, &cma->flags))) + set_bit(CMA_DELAYED_RELEASE, &cma->flags); + + /* + * To make cma_release_nowait() non-blocking, cma bitmap is cleared + * from a work context (see cma_clear_bitmap_fn()). The first page + * in the cma allocation is used to store the work structure, + * so it's released after the cma bitmap clearance. Other pages + * are released immediately as previously. + */ + if (count > 1) + free_contig_range(pfn + 1, count - 1); + + work = (struct cma_clear_bitmap_work *)page_to_virt(pages); + INIT_WORK(&work->work, cma_clear_bitmap_fn); + work->cma = cma; + work->pfn = pfn; + work->count = count; + queue_work(cma_release_wq, &work->work); + + trace_cma_release(pfn, pages, count); + + return true; +} + int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data) { int i; diff --git a/mm/cma.h b/mm/cma.h index 42ae082cb067..e9293871d122 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -7,6 +7,7 @@ struct cma { unsigned long base_pfn; unsigned long count; + unsigned long flags; unsigned long *bitmap; unsigned int order_per_bit; /* Order of pages represented by one bit */ struct mutex lock; @@ -18,6 +19,10 @@ struct cma { char name[CMA_MAX_NAME]; }; +enum cma_flags { + CMA_DELAYED_RELEASE, /* cma bitmap is cleared asynchronously */ +}; + extern struct cma cma_areas[MAX_CMA_AREAS]; extern unsigned cma_area_count; From patchwork Thu Oct 22 22:53:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11852093 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B450B6A2 for ; Thu, 22 Oct 2020 22:53:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5F0F024658 for ; Thu, 22 Oct 2020 22:53:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="O6OT7p9i" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F0F024658 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D0CCD6B0068; Thu, 22 Oct 2020 18:53:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C6CA06B006E; Thu, 22 Oct 2020 18:53:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4E916B0070; Thu, 22 Oct 2020 18:53:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id 74CB36B0068 for ; Thu, 22 Oct 2020 18:53:25 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1F7DC1EE6 for ; Thu, 22 Oct 2020 22:53:25 +0000 (UTC) X-FDA: 77401064370.04.mask95_140a40327254 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id 04077800B28C for ; Thu, 22 Oct 2020 22:53:24 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=7564a8c327=guro@fb.com,,RULES_HIT:4423:30054:30064,0,RBL:67.231.145.42:@fb.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04y8zy35mkp6uqffr8ag46mab6cq4opr4i6cgqbspf3anzayf5i373k5osbbxti.6gyfqu5zd7tmqcquiuru4tbd3t3nc14qwqoyb1m67rugym17xby4b9php8gspfe.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: mask95_140a40327254 X-Filterd-Recvd-Size: 4241 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Thu, 22 Oct 2020 22:53:24 +0000 (UTC) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 09MMiSIQ008606 for ; Thu, 22 Oct 2020 15:53:23 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=jCwwODX7YwrK220gHIqPM0Cu0J/BfkukpANadFRweU0=; b=O6OT7p9iIl6c9a4dpB5r0W1paZTT9NnFX/MN2WnY0i9b9xwSSFtBeOGVdupFRJaiVCQA 0iWfRntwU9X2HyAeo6YfFUAKBJdTtVCb9m5O6G1OKc+bFohoO0nLgEB5gBSyCzgPi4vV 64/VQWj5r+gBn76OiYB8L2UKDgNhkQnPoF0= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 34a57yx3fb-14 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Thu, 22 Oct 2020 15:53:23 -0700 Received: from intmgw003.06.prn3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 22 Oct 2020 15:53:18 -0700 Received: by devvm1755.vll0.facebook.com (Postfix, from userid 111017) id ED8AD1A3AE4C; Thu, 22 Oct 2020 15:53:12 -0700 (PDT) From: Roman Gushchin To: Andrew Morton CC: Zi Yan , Joonsoo Kim , Mike Kravetz , , , , , Roman Gushchin Subject: [PATCH v1 2/2] mm: hugetlb: don't drop hugetlb_lock around cma_release() call Date: Thu, 22 Oct 2020 15:53:08 -0700 Message-ID: <20201022225308.2927890-3-guro@fb.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20201022225308.2927890-1-guro@fb.com> References: <20201022225308.2927890-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.737 definitions=2020-10-22_17:2020-10-20,2020-10-22 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 mlxlogscore=612 lowpriorityscore=0 bulkscore=0 spamscore=0 mlxscore=0 phishscore=0 clxscore=1015 adultscore=0 priorityscore=1501 suspectscore=2 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2010220145 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace blocking cma_release() with a non-blocking cma_release_nowait() call, so there is no more need to temporarily drop hugetlb_lock. Signed-off-by: Roman Gushchin --- mm/hugetlb.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fe76f8fd5a73..230e9b6c9a2b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1224,10 +1224,11 @@ static void free_gigantic_page(struct page *page, unsigned int order) { /* * If the page isn't allocated using the cma allocator, - * cma_release() returns false. + * cma_release_nowait() returns false. */ #ifdef CONFIG_CMA - if (cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order)) + if (cma_release_nowait(hugetlb_cma[page_to_nid(page)], page, + 1 << order)) return; #endif @@ -1312,14 +1313,8 @@ static void update_and_free_page(struct hstate *h, struct page *page) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); set_page_refcounted(page); if (hstate_is_gigantic(h)) { - /* - * Temporarily drop the hugetlb_lock, because - * we might block in free_gigantic_page(). - */ - spin_unlock(&hugetlb_lock); destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); - spin_lock(&hugetlb_lock); } else { __free_pages(page, huge_page_order(h)); }