From patchwork Tue Aug 7 16:45:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EED8C13B4 for ; Tue, 7 Aug 2018 16:45:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB8872A621 for ; Tue, 7 Aug 2018 16:45:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CC04B2A6AF; Tue, 7 Aug 2018 16:45:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6AE042A621 for ; Tue, 7 Aug 2018 16:45:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388581AbeHGTAT (ORCPT ); Tue, 7 Aug 2018 15:00:19 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56304 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727286AbeHGTAT (ORCPT ); Tue, 7 Aug 2018 15:00:19 -0400 X-ASG-Debug-ID: 1533660308-0fb3b01fb342b7e0001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id 8ZvagaLz2o389S9s (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:45:08 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324850; Tue, 07 Aug 2018 12:45:08 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 01/10] dmapool: fix boundary comparison Message-ID: X-ASG-Orig-Subj: [PATCH v3 01/10] dmapool: fix boundary comparison Date: Tue, 7 Aug 2018 12:45:08 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660308 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 1475 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Fix the boundary comparison when constructing the list of free blocks for the case that 'size' is a power of two. Since 'boundary' is also a power of two, that would make 'boundary' a multiple of 'size', in which case a single block would never cross the boundary. This bug would cause some of the allocated memory to be wasted (but not leaked). Example: size = 512 boundary = 2048 allocation = 4096 Address range 0 - 511 512 - 1023 1024 - 1535 1536 - 2047 * 2048 - 2559 2560 - 3071 3072 - 3583 3584 - 4095 * Prior to this fix, the address ranges marked with "*" would not have been used even though they didn't cross the given boundary. Fixes: e34f44b3517f ("pool: Improve memory usage for devices which can't cross boundaries") Signed-off-by: Tony Battersby --- No changes since v2. Even though I described this as a "fix", it does not seem important enough to Cc: stable from a strict reading of the stable kernel rules. IOW, it is not "bothering" anyone. --- linux/mm/dmapool.c.orig 2018-08-01 17:57:04.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-01 17:57:16.000000000 -0400 @@ -210,7 +210,7 @@ static void pool_initialise_page(struct do { unsigned int next = offset + pool->size; - if (unlikely((next + pool->size) >= next_boundary)) { + if (unlikely((next + pool->size) > next_boundary)) { next = next_boundary; next_boundary += pool->boundary; } From patchwork Tue Aug 7 16:45:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D75E13B4 for ; Tue, 7 Aug 2018 16:45:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58E1E2A688 for ; Tue, 7 Aug 2018 16:45:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 49B082A6B5; Tue, 7 Aug 2018 16:45:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE0C62A688 for ; Tue, 7 Aug 2018 16:45:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388543AbeHGTAy (ORCPT ); Tue, 7 Aug 2018 15:00:54 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56396 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727286AbeHGTAx (ORCPT ); Tue, 7 Aug 2018 15:00:53 -0400 X-ASG-Debug-ID: 1533660342-0fb3b01fb342b830001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id ckqYIo1C7Ii7OcuH (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:45:42 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324853; Tue, 07 Aug 2018 12:45:42 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 02/10] dmapool: remove checks for dev == NULL Message-ID: <99c5f149-90a7-dcc5-73a5-179ca009f914@cybernetics.com> X-ASG-Orig-Subj: [PATCH v3 02/10] dmapool: remove checks for dev == NULL Date: Tue, 7 Aug 2018 12:45:42 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660342 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 3650 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP dmapool originally tried to support pools without a device because dma_alloc_coherent() supports allocations without a device. But nobody ended up using dma pools without a device, so the current checks in dmapool.c for pool->dev == NULL are both insufficient and causing bloat. Remove them. Signed-off-by: Tony Battersby --- Changes since v2: *) This was "dmapool: cleanup error messages" in v2. *) Remove one more check for dev == NULL in dma_pool_destroy() that is unrelated to error messages. --- linux/mm/dmapool.c.orig 2018-08-03 16:12:23.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-03 16:13:44.000000000 -0400 @@ -277,7 +277,7 @@ void dma_pool_destroy(struct dma_pool *p mutex_lock(&pools_reg_lock); mutex_lock(&pools_lock); list_del(&pool->pools); - if (pool->dev && list_empty(&pool->dev->dma_pools)) + if (list_empty(&pool->dev->dma_pools)) empty = true; mutex_unlock(&pools_lock); if (empty) @@ -289,13 +289,9 @@ void dma_pool_destroy(struct dma_pool *p page = list_entry(pool->page_list.next, struct dma_page, page_list); if (is_page_busy(page)) { - if (pool->dev) - dev_err(pool->dev, - "dma_pool_destroy %s, %p busy\n", - pool->name, page->vaddr); - else - pr_err("dma_pool_destroy %s, %p busy\n", - pool->name, page->vaddr); + dev_err(pool->dev, + "dma_pool_destroy %s, %p busy\n", + pool->name, page->vaddr); /* leak the still-in-use consistent memory */ list_del(&page->page_list); kfree(page); @@ -357,13 +353,9 @@ void *dma_pool_alloc(struct dma_pool *po for (i = sizeof(page->offset); i < pool->size; i++) { if (data[i] == POOL_POISON_FREED) continue; - if (pool->dev) - dev_err(pool->dev, - "dma_pool_alloc %s, %p (corrupted)\n", - pool->name, retval); - else - pr_err("dma_pool_alloc %s, %p (corrupted)\n", - pool->name, retval); + dev_err(pool->dev, + "dma_pool_alloc %s, %p (corrupted)\n", + pool->name, retval); /* * Dump the first 4 bytes even if they are not @@ -418,13 +410,9 @@ void dma_pool_free(struct dma_pool *pool page = pool_find_page(pool, dma); if (!page) { spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, - "dma_pool_free %s, %p/%lx (bad dma)\n", - pool->name, vaddr, (unsigned long)dma); - else - pr_err("dma_pool_free %s, %p/%lx (bad dma)\n", - pool->name, vaddr, (unsigned long)dma); + dev_err(pool->dev, + "dma_pool_free %s, %p/%lx (bad dma)\n", + pool->name, vaddr, (unsigned long)dma); return; } @@ -432,13 +420,9 @@ void dma_pool_free(struct dma_pool *pool #ifdef DMAPOOL_DEBUG if ((dma - page->dma) != offset) { spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, - "dma_pool_free %s, %p (bad vaddr)/%pad\n", - pool->name, vaddr, &dma); - else - pr_err("dma_pool_free %s, %p (bad vaddr)/%pad\n", - pool->name, vaddr, &dma); + dev_err(pool->dev, + "dma_pool_free %s, %p (bad vaddr)/%pad\n", + pool->name, vaddr, &dma); return; } { @@ -449,12 +433,9 @@ void dma_pool_free(struct dma_pool *pool continue; } spin_unlock_irqrestore(&pool->lock, flags); - if (pool->dev) - dev_err(pool->dev, "dma_pool_free %s, dma %pad already free\n", - pool->name, &dma); - else - pr_err("dma_pool_free %s, dma %pad already free\n", - pool->name, &dma); + dev_err(pool->dev, + "dma_pool_free %s, dma %pad already free\n", + pool->name, &dma); return; } } From patchwork Tue Aug 7 16:46:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E272114E5 for ; Tue, 7 Aug 2018 16:46:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D00902A621 for ; Tue, 7 Aug 2018 16:46:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C2A012A6AF; Tue, 7 Aug 2018 16:46:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B3C82A621 for ; Tue, 7 Aug 2018 16:46:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727286AbeHGTB3 (ORCPT ); Tue, 7 Aug 2018 15:01:29 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56474 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725881AbeHGTB3 (ORCPT ); Tue, 7 Aug 2018 15:01:29 -0400 X-ASG-Debug-ID: 1533660378-0fb3b01fb342b850001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id 1q7UScR8LZmYJ9oQ (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:46:18 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324855; Tue, 07 Aug 2018 12:46:18 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 03/10] dmapool: cleanup dma_pool_destroy Message-ID: <9fb39095-4dd8-877a-b857-649e76fedd59@cybernetics.com> X-ASG-Orig-Subj: [PATCH v3 03/10] dmapool: cleanup dma_pool_destroy Date: Tue, 7 Aug 2018 12:46:18 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660378 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 2099 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Remove a small amount of code duplication between dma_pool_destroy() and pool_free_page() in preparation for adding more code without having to duplicate it. No functional changes. Signed-off-by: Tony Battersby --- No changes since v2. --- linux/mm/dmapool.c.orig 2018-08-02 09:59:15.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-02 10:01:26.000000000 -0400 @@ -249,13 +249,22 @@ static inline bool is_page_busy(struct d static void pool_free_page(struct dma_pool *pool, struct dma_page *page) { + void *vaddr = page->vaddr; dma_addr_t dma = page->dma; + list_del(&page->page_list); + + if (is_page_busy(page)) { + dev_err(pool->dev, + "dma_pool_destroy %s, %p busy\n", + pool->name, vaddr); + /* leak the still-in-use consistent memory */ + } else { #ifdef DMAPOOL_DEBUG - memset(page->vaddr, POOL_POISON_FREED, pool->allocation); + memset(vaddr, POOL_POISON_FREED, pool->allocation); #endif - dma_free_coherent(pool->dev, pool->allocation, page->vaddr, dma); - list_del(&page->page_list); + dma_free_coherent(pool->dev, pool->allocation, vaddr, dma); + } kfree(page); } @@ -269,6 +278,7 @@ static void pool_free_page(struct dma_po */ void dma_pool_destroy(struct dma_pool *pool) { + struct dma_page *page; bool empty = false; if (unlikely(!pool)) @@ -284,19 +294,10 @@ void dma_pool_destroy(struct dma_pool *p device_remove_file(pool->dev, &dev_attr_pools); mutex_unlock(&pools_reg_lock); - while (!list_empty(&pool->page_list)) { - struct dma_page *page; - page = list_entry(pool->page_list.next, - struct dma_page, page_list); - if (is_page_busy(page)) { - dev_err(pool->dev, - "dma_pool_destroy %s, %p busy\n", - pool->name, page->vaddr); - /* leak the still-in-use consistent memory */ - list_del(&page->page_list); - kfree(page); - } else - pool_free_page(pool, page); + while ((page = list_first_entry_or_null(&pool->page_list, + struct dma_page, + page_list))) { + pool_free_page(pool, page); } kfree(pool); From patchwork Tue Aug 7 16:46:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 61DA413B4 for ; Tue, 7 Aug 2018 16:46:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F6952A621 for ; Tue, 7 Aug 2018 16:46:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 43A4A2A692; Tue, 7 Aug 2018 16:46:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC54C2A621 for ; Tue, 7 Aug 2018 16:46:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729922AbeHGTCF (ORCPT ); Tue, 7 Aug 2018 15:02:05 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56558 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725881AbeHGTCF (ORCPT ); Tue, 7 Aug 2018 15:02:05 -0400 X-ASG-Debug-ID: 1533660413-0fb3b01fb342b8a0001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id tpA8NN95pS060sn0 (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:46:53 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324863; Tue, 07 Aug 2018 12:46:53 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 04/10] dmapool: improve scalability of dma_pool_alloc Message-ID: <1186a929-8fd8-5ead-606d-d3699527c795@cybernetics.com> X-ASG-Orig-Subj: [PATCH v3 04/10] dmapool: improve scalability of dma_pool_alloc Date: Tue, 7 Aug 2018 12:46:53 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660413 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 6785 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP dma_pool_alloc() scales poorly when allocating a large number of pages because it does a linear scan of all previously-allocated pages before allocating a new one. Improve its scalability by maintaining a separate list of pages that have free blocks ready to (re)allocate. In big O notation, this improves the algorithm from O(n^2) to O(n). Signed-off-by: Tony Battersby --- Changes since v2: *) Use list_move()/list_move_tail() instead of list_del+list_add(). *) Renamed POOL_N_LISTS to POOL_MAX_IDX. *) Use defined names instead of 0/1 indexes for INIT_LIST_HEAD(). --- linux/mm/dmapool.c.orig 2018-08-03 16:16:49.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-03 16:45:33.000000000 -0400 @@ -15,11 +15,16 @@ * Many older drivers still have their own code to do this. * * The current design of this allocator is fairly simple. The pool is - * represented by the 'struct dma_pool' which keeps a doubly-linked list of - * allocated pages. Each page in the page_list is split into blocks of at - * least 'size' bytes. Free blocks are tracked in an unsorted singly-linked - * list of free blocks within the page. Used blocks aren't tracked, but we - * keep a count of how many are currently allocated from each page. + * represented by the 'struct dma_pool'. Each allocated page is split into + * blocks of at least 'size' bytes. Free blocks are tracked in an unsorted + * singly-linked list of free blocks within the page. Used blocks aren't + * tracked, but we keep a count of how many are currently allocated from each + * page. + * + * The pool keeps two doubly-linked list of allocated pages. The 'available' + * list tracks pages that have one or more free blocks, and the 'full' list + * tracks pages that have no free blocks. Pages are moved from one list to + * the other as their blocks are allocated and freed. */ #include @@ -43,7 +48,10 @@ #endif struct dma_pool { /* the pool */ - struct list_head page_list; +#define POOL_FULL_IDX 0 +#define POOL_AVAIL_IDX 1 +#define POOL_MAX_IDX 2 + struct list_head page_list[POOL_MAX_IDX]; spinlock_t lock; size_t size; struct device *dev; @@ -54,7 +62,7 @@ struct dma_pool { /* the pool */ }; struct dma_page { /* cacheable header for 'allocation' bytes */ - struct list_head page_list; + struct list_head dma_list; void *vaddr; dma_addr_t dma; unsigned int in_use; @@ -70,7 +78,6 @@ show_pools(struct device *dev, struct de unsigned temp; unsigned size; char *next; - struct dma_page *page; struct dma_pool *pool; next = buf; @@ -84,11 +91,18 @@ show_pools(struct device *dev, struct de list_for_each_entry(pool, &dev->dma_pools, pools) { unsigned pages = 0; unsigned blocks = 0; + int list_idx; spin_lock_irq(&pool->lock); - list_for_each_entry(page, &pool->page_list, page_list) { - pages++; - blocks += page->in_use; + for (list_idx = 0; list_idx < POOL_MAX_IDX; list_idx++) { + struct dma_page *page; + + list_for_each_entry(page, + &pool->page_list[list_idx], + dma_list) { + pages++; + blocks += page->in_use; + } } spin_unlock_irq(&pool->lock); @@ -163,7 +177,8 @@ struct dma_pool *dma_pool_create(const c retval->dev = dev; - INIT_LIST_HEAD(&retval->page_list); + INIT_LIST_HEAD(&retval->page_list[POOL_FULL_IDX]); + INIT_LIST_HEAD(&retval->page_list[POOL_AVAIL_IDX]); spin_lock_init(&retval->lock); retval->size = size; retval->boundary = boundary; @@ -252,7 +267,7 @@ static void pool_free_page(struct dma_po void *vaddr = page->vaddr; dma_addr_t dma = page->dma; - list_del(&page->page_list); + list_del(&page->dma_list); if (is_page_busy(page)) { dev_err(pool->dev, @@ -278,8 +293,8 @@ static void pool_free_page(struct dma_po */ void dma_pool_destroy(struct dma_pool *pool) { - struct dma_page *page; bool empty = false; + int list_idx; if (unlikely(!pool)) return; @@ -294,10 +309,15 @@ void dma_pool_destroy(struct dma_pool *p device_remove_file(pool->dev, &dev_attr_pools); mutex_unlock(&pools_reg_lock); - while ((page = list_first_entry_or_null(&pool->page_list, - struct dma_page, - page_list))) { - pool_free_page(pool, page); + for (list_idx = 0; list_idx < POOL_MAX_IDX; list_idx++) { + struct dma_page *page; + + while ((page = list_first_entry_or_null( + &pool->page_list[list_idx], + struct dma_page, + dma_list))) { + pool_free_page(pool, page); + } } kfree(pool); @@ -325,10 +345,11 @@ void *dma_pool_alloc(struct dma_pool *po might_sleep_if(gfpflags_allow_blocking(mem_flags)); spin_lock_irqsave(&pool->lock, flags); - list_for_each_entry(page, &pool->page_list, page_list) { - if (page->offset < pool->allocation) - goto ready; - } + page = list_first_entry_or_null(&pool->page_list[POOL_AVAIL_IDX], + struct dma_page, + dma_list); + if (page) + goto ready; /* pool_alloc_page() might sleep, so temporarily drop &pool->lock */ spin_unlock_irqrestore(&pool->lock, flags); @@ -339,11 +360,15 @@ void *dma_pool_alloc(struct dma_pool *po spin_lock_irqsave(&pool->lock, flags); - list_add(&page->page_list, &pool->page_list); + list_add(&page->dma_list, &pool->page_list[POOL_AVAIL_IDX]); ready: page->in_use++; offset = page->offset; page->offset = *(int *)(page->vaddr + offset); + if (page->offset >= pool->allocation) + /* Move page from the "available" list to the "full" list. */ + list_move_tail(&page->dma_list, + &pool->page_list[POOL_FULL_IDX]); retval = offset + page->vaddr; *handle = offset + page->dma; #ifdef DMAPOOL_DEBUG @@ -381,13 +406,19 @@ EXPORT_SYMBOL(dma_pool_alloc); static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t dma) { - struct dma_page *page; + int list_idx; - list_for_each_entry(page, &pool->page_list, page_list) { - if (dma < page->dma) - continue; - if ((dma - page->dma) < pool->allocation) - return page; + for (list_idx = 0; list_idx < POOL_MAX_IDX; list_idx++) { + struct dma_page *page; + + list_for_each_entry(page, + &pool->page_list[list_idx], + dma_list) { + if (dma < page->dma) + continue; + if ((dma - page->dma) < pool->allocation) + return page; + } } return NULL; } @@ -444,6 +475,9 @@ void dma_pool_free(struct dma_pool *pool #endif page->in_use--; + if (page->offset >= pool->allocation) + /* Move page from the "full" list to the "available" list. */ + list_move(&page->dma_list, &pool->page_list[POOL_AVAIL_IDX]); *(int *)vaddr = page->offset; page->offset = offset; /* From patchwork Tue Aug 7 16:47:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558915 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B935114E5 for ; Tue, 7 Aug 2018 16:47:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6C9E2A621 for ; Tue, 7 Aug 2018 16:47:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9A61D2A692; Tue, 7 Aug 2018 16:47:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37F082A621 for ; Tue, 7 Aug 2018 16:47:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733095AbeHGTCj (ORCPT ); Tue, 7 Aug 2018 15:02:39 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56636 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732347AbeHGTCj (ORCPT ); Tue, 7 Aug 2018 15:02:39 -0400 X-ASG-Debug-ID: 1533660448-0fb3b01fb342b8c0001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id aoLRbHEitRbZoBdk (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:47:28 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324865; Tue, 07 Aug 2018 12:47:28 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 05/10] dmapool: rename fields in dma_page Message-ID: X-ASG-Orig-Subj: [PATCH v3 05/10] dmapool: rename fields in dma_page Date: Tue, 7 Aug 2018 12:47:28 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660448 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 3410 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Rename fields in 'struct dma_page' in preparation for moving them into 'struct page'. No functional changes. in_use -> dma_in_use offset -> dma_free_off Signed-off-by: Tony Battersby --- Changes since v2: Use dma_free_off instead of dma_free_o. --- linux/mm/dmapool.c.orig 2018-08-03 17:46:13.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-03 17:46:24.000000000 -0400 @@ -65,8 +65,8 @@ struct dma_page { /* cacheable header f struct list_head dma_list; void *vaddr; dma_addr_t dma; - unsigned int in_use; - unsigned int offset; + unsigned int dma_in_use; + unsigned int dma_free_off; }; static DEFINE_MUTEX(pools_lock); @@ -101,7 +101,7 @@ show_pools(struct device *dev, struct de &pool->page_list[list_idx], dma_list) { pages++; - blocks += page->in_use; + blocks += page->dma_in_use; } } spin_unlock_irq(&pool->lock); @@ -248,8 +248,8 @@ static struct dma_page *pool_alloc_page( memset(page->vaddr, POOL_POISON_FREED, pool->allocation); #endif pool_initialise_page(pool, page); - page->in_use = 0; - page->offset = 0; + page->dma_in_use = 0; + page->dma_free_off = 0; } else { kfree(page); page = NULL; @@ -259,7 +259,7 @@ static struct dma_page *pool_alloc_page( static inline bool is_page_busy(struct dma_page *page) { - return page->in_use != 0; + return page->dma_in_use != 0; } static void pool_free_page(struct dma_pool *pool, struct dma_page *page) @@ -362,10 +362,10 @@ void *dma_pool_alloc(struct dma_pool *po list_add(&page->dma_list, &pool->page_list[POOL_AVAIL_IDX]); ready: - page->in_use++; - offset = page->offset; - page->offset = *(int *)(page->vaddr + offset); - if (page->offset >= pool->allocation) + page->dma_in_use++; + offset = page->dma_free_off; + page->dma_free_off = *(int *)(page->vaddr + offset); + if (page->dma_free_off >= pool->allocation) /* Move page from the "available" list to the "full" list. */ list_move_tail(&page->dma_list, &pool->page_list[POOL_FULL_IDX]); @@ -375,8 +375,8 @@ void *dma_pool_alloc(struct dma_pool *po { int i; u8 *data = retval; - /* page->offset is stored in first 4 bytes */ - for (i = sizeof(page->offset); i < pool->size; i++) { + /* page->dma_free_off is stored in first 4 bytes */ + for (i = sizeof(page->dma_free_off); i < pool->size; i++) { if (data[i] == POOL_POISON_FREED) continue; dev_err(pool->dev, @@ -458,7 +458,7 @@ void dma_pool_free(struct dma_pool *pool return; } { - unsigned int chain = page->offset; + unsigned int chain = page->dma_free_off; while (chain < pool->allocation) { if (chain != offset) { chain = *(int *)(page->vaddr + chain); @@ -474,12 +474,12 @@ void dma_pool_free(struct dma_pool *pool memset(vaddr, POOL_POISON_FREED, pool->size); #endif - page->in_use--; - if (page->offset >= pool->allocation) + page->dma_in_use--; + if (page->dma_free_off >= pool->allocation) /* Move page from the "full" list to the "available" list. */ list_move(&page->dma_list, &pool->page_list[POOL_AVAIL_IDX]); - *(int *)vaddr = page->offset; - page->offset = offset; + *(int *)vaddr = page->dma_free_off; + page->dma_free_off = offset; /* * Resist a temptation to do * if (!is_page_busy(page)) pool_free_page(pool, page); From patchwork Tue Aug 7 16:48:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558919 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C5E2013B4 for ; Tue, 7 Aug 2018 16:48:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B01BD2A440 for ; Tue, 7 Aug 2018 16:48:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9F6AA2A446; Tue, 7 Aug 2018 16:48:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E5BC72A440 for ; Tue, 7 Aug 2018 16:48:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388569AbeHGTDP (ORCPT ); Tue, 7 Aug 2018 15:03:15 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56714 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388360AbeHGTDP (ORCPT ); Tue, 7 Aug 2018 15:03:15 -0400 X-ASG-Debug-ID: 1533660483-0fb3b01fb342b910001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id folF0MDlsSuJcxid (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:48:03 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324868; Tue, 07 Aug 2018 12:48:03 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 06/10] dmapool: improve scalability of dma_pool_free Message-ID: X-ASG-Orig-Subj: [PATCH v3 06/10] dmapool: improve scalability of dma_pool_free Date: Tue, 7 Aug 2018 12:48:03 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660483 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 8854 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP dma_pool_free() scales poorly when the pool contains many pages because pool_find_page() does a linear scan of all allocated pages. Improve its scalability by replacing the linear scan with virt_to_page() and storing dmapool private data directly in 'struct page', thereby eliminating 'struct dma_page'. In big O notation, this improves the algorithm from O(n^2) to O(n) while also reducing memory usage. Thanks to Matthew Wilcox for the suggestion to use struct page. Signed-off-by: Tony Battersby --- Changes since v2: Just a re-diff after the changes in prior patches. --- linux/include/linux/mm_types.h.orig 2018-08-01 17:59:46.000000000 -0400 +++ linux/include/linux/mm_types.h 2018-08-01 17:59:56.000000000 -0400 @@ -153,6 +153,12 @@ struct page { unsigned long _zd_pad_1; /* uses mapping */ }; + struct { /* dma_pool pages */ + struct list_head dma_list; + dma_addr_t dma; + unsigned int dma_free_off; + }; + /** @rcu_head: You can use this to free a page by RCU. */ struct rcu_head rcu_head; }; @@ -174,6 +180,8 @@ struct page { unsigned int active; /* SLAB */ int units; /* SLOB */ + + unsigned int dma_in_use; /* dma_pool pages */ }; /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ --- linux/mm/dmapool.c.orig 2018-08-03 17:47:03.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-03 17:47:22.000000000 -0400 @@ -25,6 +25,10 @@ * list tracks pages that have one or more free blocks, and the 'full' list * tracks pages that have no free blocks. Pages are moved from one list to * the other as their blocks are allocated and freed. + * + * When allocating DMA pages, we use some available space in 'struct page' to + * store data private to dmapool; search 'dma_pool' in the definition of + * 'struct page' for details. */ #include @@ -61,14 +65,6 @@ struct dma_pool { /* the pool */ struct list_head pools; }; -struct dma_page { /* cacheable header for 'allocation' bytes */ - struct list_head dma_list; - void *vaddr; - dma_addr_t dma; - unsigned int dma_in_use; - unsigned int dma_free_off; -}; - static DEFINE_MUTEX(pools_lock); static DEFINE_MUTEX(pools_reg_lock); @@ -95,7 +91,7 @@ show_pools(struct device *dev, struct de spin_lock_irq(&pool->lock); for (list_idx = 0; list_idx < POOL_MAX_IDX; list_idx++) { - struct dma_page *page; + struct page *page; list_for_each_entry(page, &pool->page_list[list_idx], @@ -218,7 +214,7 @@ struct dma_pool *dma_pool_create(const c } EXPORT_SYMBOL(dma_pool_create); -static void pool_initialise_page(struct dma_pool *pool, struct dma_page *page) +static void pool_initialize_free_block_list(struct dma_pool *pool, void *vaddr) { unsigned int offset = 0; unsigned int next_boundary = pool->boundary; @@ -229,47 +225,57 @@ static void pool_initialise_page(struct next = next_boundary; next_boundary += pool->boundary; } - *(int *)(page->vaddr + offset) = next; + *(int *)(vaddr + offset) = next; offset = next; } while (offset < pool->allocation); } -static struct dma_page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) +static struct page *pool_alloc_page(struct dma_pool *pool, gfp_t mem_flags) { - struct dma_page *page; + struct page *page; + dma_addr_t dma; + void *vaddr; - page = kmalloc(sizeof(*page), mem_flags); - if (!page) + vaddr = dma_alloc_coherent(pool->dev, pool->allocation, &dma, + mem_flags); + if (!vaddr) return NULL; - page->vaddr = dma_alloc_coherent(pool->dev, pool->allocation, - &page->dma, mem_flags); - if (page->vaddr) { + #ifdef DMAPOOL_DEBUG - memset(page->vaddr, POOL_POISON_FREED, pool->allocation); + memset(vaddr, POOL_POISON_FREED, pool->allocation); #endif - pool_initialise_page(pool, page); - page->dma_in_use = 0; - page->dma_free_off = 0; - } else { - kfree(page); - page = NULL; - } + pool_initialize_free_block_list(pool, vaddr); + + page = virt_to_page(vaddr); + page->dma = dma; + page->dma_free_off = 0; + page->dma_in_use = 0; + return page; } -static inline bool is_page_busy(struct dma_page *page) +static inline bool is_page_busy(struct page *page) { return page->dma_in_use != 0; } -static void pool_free_page(struct dma_pool *pool, struct dma_page *page) +static void pool_free_page(struct dma_pool *pool, struct page *page) { - void *vaddr = page->vaddr; + /* Save local copies of some page fields. */ + void *vaddr = page_to_virt(page); + bool busy = is_page_busy(page); dma_addr_t dma = page->dma; list_del(&page->dma_list); - if (is_page_busy(page)) { + /* Clear all the page fields we use. */ + page->dma_list.next = NULL; + page->dma_list.prev = NULL; + page->dma = 0; + page->dma_free_off = 0; + page_mapcount_reset(page); /* clear dma_in_use */ + + if (busy) { dev_err(pool->dev, "dma_pool_destroy %s, %p busy\n", pool->name, vaddr); @@ -280,7 +286,6 @@ static void pool_free_page(struct dma_po #endif dma_free_coherent(pool->dev, pool->allocation, vaddr, dma); } - kfree(page); } /** @@ -310,11 +315,11 @@ void dma_pool_destroy(struct dma_pool *p mutex_unlock(&pools_reg_lock); for (list_idx = 0; list_idx < POOL_MAX_IDX; list_idx++) { - struct dma_page *page; + struct page *page; while ((page = list_first_entry_or_null( &pool->page_list[list_idx], - struct dma_page, + struct page, dma_list))) { pool_free_page(pool, page); } @@ -338,15 +343,16 @@ void *dma_pool_alloc(struct dma_pool *po dma_addr_t *handle) { unsigned long flags; - struct dma_page *page; + struct page *page; size_t offset; void *retval; + void *vaddr; might_sleep_if(gfpflags_allow_blocking(mem_flags)); spin_lock_irqsave(&pool->lock, flags); page = list_first_entry_or_null(&pool->page_list[POOL_AVAIL_IDX], - struct dma_page, + struct page, dma_list); if (page) goto ready; @@ -362,14 +368,15 @@ void *dma_pool_alloc(struct dma_pool *po list_add(&page->dma_list, &pool->page_list[POOL_AVAIL_IDX]); ready: + vaddr = page_to_virt(page); page->dma_in_use++; offset = page->dma_free_off; - page->dma_free_off = *(int *)(page->vaddr + offset); + page->dma_free_off = *(int *)(vaddr + offset); if (page->dma_free_off >= pool->allocation) /* Move page from the "available" list to the "full" list. */ list_move_tail(&page->dma_list, &pool->page_list[POOL_FULL_IDX]); - retval = offset + page->vaddr; + retval = offset + vaddr; *handle = offset + page->dma; #ifdef DMAPOOL_DEBUG { @@ -404,25 +411,6 @@ void *dma_pool_alloc(struct dma_pool *po } EXPORT_SYMBOL(dma_pool_alloc); -static struct dma_page *pool_find_page(struct dma_pool *pool, dma_addr_t dma) -{ - int list_idx; - - for (list_idx = 0; list_idx < POOL_MAX_IDX; list_idx++) { - struct dma_page *page; - - list_for_each_entry(page, - &pool->page_list[list_idx], - dma_list) { - if (dma < page->dma) - continue; - if ((dma - page->dma) < pool->allocation) - return page; - } - } - return NULL; -} - /** * dma_pool_free - put block back into dma pool * @pool: the dma pool holding the block @@ -434,34 +422,35 @@ static struct dma_page *pool_find_page(s */ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) { - struct dma_page *page; + struct page *page; unsigned long flags; unsigned int offset; - spin_lock_irqsave(&pool->lock, flags); - page = pool_find_page(pool, dma); - if (!page) { - spin_unlock_irqrestore(&pool->lock, flags); + if (unlikely(!virt_addr_valid(vaddr))) { dev_err(pool->dev, - "dma_pool_free %s, %p/%lx (bad dma)\n", - pool->name, vaddr, (unsigned long)dma); + "dma_pool_free %s, %p (bad vaddr)/%pad\n", + pool->name, vaddr, &dma); return; } - offset = vaddr - page->vaddr; -#ifdef DMAPOOL_DEBUG - if ((dma - page->dma) != offset) { - spin_unlock_irqrestore(&pool->lock, flags); + page = virt_to_page(vaddr); + offset = offset_in_page(vaddr); + + if (unlikely((dma - page->dma) != offset)) { dev_err(pool->dev, - "dma_pool_free %s, %p (bad vaddr)/%pad\n", + "dma_pool_free %s, %p (bad vaddr)/%pad (or bad dma)\n", pool->name, vaddr, &dma); return; } + + spin_lock_irqsave(&pool->lock, flags); +#ifdef DMAPOOL_DEBUG { + void *page_vaddr = vaddr - offset; unsigned int chain = page->dma_free_off; while (chain < pool->allocation) { if (chain != offset) { - chain = *(int *)(page->vaddr + chain); + chain = *(int *)(page_vaddr + chain); continue; } spin_unlock_irqrestore(&pool->lock, flags); From patchwork Tue Aug 7 16:48:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0351F14E5 for ; Tue, 7 Aug 2018 16:48:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E40A12A70A for ; Tue, 7 Aug 2018 16:48:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D65EE2A70D; Tue, 7 Aug 2018 16:48:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D47F2A70A for ; Tue, 7 Aug 2018 16:48:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388671AbeHGTDu (ORCPT ); Tue, 7 Aug 2018 15:03:50 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56792 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388360AbeHGTDu (ORCPT ); Tue, 7 Aug 2018 15:03:50 -0400 X-ASG-Debug-ID: 1533660518-0fb3b01fb342b940001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id e9CaRxsAFf7Ms5D1 (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:48:38 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324871; Tue, 07 Aug 2018 12:48:38 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 07/10] dmapool: cleanup integer types Message-ID: <5d0aec14-73e0-280d-62fb-2b0fe6c01418@cybernetics.com> X-ASG-Orig-Subj: [PATCH v3 07/10] dmapool: cleanup integer types Date: Tue, 7 Aug 2018 12:48:38 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660518 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 3586 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To represent the size of a single allocation, dmapool currently uses 'unsigned int' in some places and 'size_t' in other places. Standardize on 'unsigned int' to reduce overhead, but use 'size_t' when counting all the blocks in the entire pool. Signed-off-by: Tony Battersby --- This was split off from "dmapool: reduce footprint in struct page" in v2. This puts an upper bound on 'size' of INT_MAX to avoid overflowing the following comparison in pool_initialize_free_block_list(): unsigned int offset = 0; unsigned int next = offset + pool->size; if (unlikely((next + pool->size) > ... The actual maximum allocation size is probably lower anyway, probably KMALLOC_MAX_SIZE, but that gets into the implementation details of other subsystems which don't export a predefined maximum, so I didn't want to hardcode it here. The purpose of the added bounds check is to avoid overflowing integers, not to check the actual (platform/device/config-specific?) maximum allocation size. 'boundary' is passed in as a size_t but gets stored as an unsigned int. 'boundary' values >= 'allocation' do not have any effect, so clipping 'boundary' to 'allocation' keeps it within the range of unsigned int without affecting anything else. A few lines above (not in the diff) you can see that if 'boundary' is passed in as 0 then it is set to 'allocation', so it is nothing new. For reference, here is the relevant code after being patched: if (!boundary) boundary = allocation; else if ((boundary < size) || (boundary & (boundary - 1))) return NULL; boundary = min(boundary, allocation); --- linux/mm/dmapool.c.orig 2018-08-06 17:48:19.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-06 17:48:54.000000000 -0400 @@ -57,10 +57,10 @@ struct dma_pool { /* the pool */ #define POOL_MAX_IDX 2 struct list_head page_list[POOL_MAX_IDX]; spinlock_t lock; - size_t size; + unsigned int size; struct device *dev; - size_t allocation; - size_t boundary; + unsigned int allocation; + unsigned int boundary; char name[32]; struct list_head pools; }; @@ -86,7 +86,7 @@ show_pools(struct device *dev, struct de mutex_lock(&pools_lock); list_for_each_entry(pool, &dev->dma_pools, pools) { unsigned pages = 0; - unsigned blocks = 0; + size_t blocks = 0; int list_idx; spin_lock_irq(&pool->lock); @@ -103,9 +103,10 @@ show_pools(struct device *dev, struct de spin_unlock_irq(&pool->lock); /* per-pool info, no real statistics yet */ - temp = scnprintf(next, size, "%-16s %4u %4zu %4zu %2u\n", + temp = scnprintf(next, size, "%-16s %4zu %4zu %4u %2u\n", pool->name, blocks, - pages * (pool->allocation / pool->size), + (size_t) pages * + (pool->allocation / pool->size), pool->size, pages); size -= temp; next += temp; @@ -150,7 +151,7 @@ struct dma_pool *dma_pool_create(const c else if (align & (align - 1)) return NULL; - if (size == 0) + if (size == 0 || size > INT_MAX) return NULL; else if (size < 4) size = 4; @@ -165,6 +166,8 @@ struct dma_pool *dma_pool_create(const c else if ((boundary < size) || (boundary & (boundary - 1))) return NULL; + boundary = min(boundary, allocation); + retval = kmalloc_node(sizeof(*retval), GFP_KERNEL, dev_to_node(dev)); if (!retval) return retval; @@ -344,7 +347,7 @@ void *dma_pool_alloc(struct dma_pool *po { unsigned long flags; struct page *page; - size_t offset; + unsigned int offset; void *retval; void *vaddr; From patchwork Tue Aug 7 16:49:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558927 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D1EB14E5 for ; Tue, 7 Aug 2018 16:49:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B3332A70A for ; Tue, 7 Aug 2018 16:49:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7F45E2A70D; Tue, 7 Aug 2018 16:49:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B1722A70A for ; Tue, 7 Aug 2018 16:49:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388935AbeHGTEZ (ORCPT ); Tue, 7 Aug 2018 15:04:25 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56864 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388360AbeHGTEZ (ORCPT ); Tue, 7 Aug 2018 15:04:25 -0400 X-ASG-Debug-ID: 1533660553-0fb3b01fb342b970001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id cMcEferQEOTLuIkw (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:49:13 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324867; Tue, 07 Aug 2018 12:49:13 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 08/10] dmapool: improve accuracy of debug statistics Message-ID: X-ASG-Orig-Subj: [PATCH v3 08/10] dmapool: improve accuracy of debug statistics Date: Tue, 7 Aug 2018 12:49:13 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660553 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 1709 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The "total number of blocks in pool" debug statistic currently does not take the boundary value into account, so it diverges from the "total number of blocks in use" statistic when a boundary is in effect. Add a calculation for the number of blocks per allocation that takes the boundary into account, and use it to replace the inaccurate calculation. Signed-off-by: Tony Battersby --- This was split off from "dmapool: reduce footprint in struct page" in v2. This depends on patch #1 "dmapool: fix boundary comparison" for the calculated blks_per_alloc value to be correct. The added blks_per_alloc value will also be used in the next patch. --- linux/mm/dmapool.c.orig 2018-08-06 17:48:54.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-06 17:52:53.000000000 -0400 @@ -61,6 +61,7 @@ struct dma_pool { /* the pool */ struct device *dev; unsigned int allocation; unsigned int boundary; + unsigned int blks_per_alloc; char name[32]; struct list_head pools; }; @@ -105,8 +106,7 @@ show_pools(struct device *dev, struct de /* per-pool info, no real statistics yet */ temp = scnprintf(next, size, "%-16s %4zu %4zu %4u %2u\n", pool->name, blocks, - (size_t) pages * - (pool->allocation / pool->size), + (size_t) pages * pool->blks_per_alloc, pool->size, pages); size -= temp; next += temp; @@ -182,6 +182,9 @@ struct dma_pool *dma_pool_create(const c retval->size = size; retval->boundary = boundary; retval->allocation = allocation; + retval->blks_per_alloc = + (allocation / boundary) * (boundary / size) + + (allocation % boundary) / size; INIT_LIST_HEAD(&retval->pools); From patchwork Tue Aug 7 16:50:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558931 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1BE2F14E5 for ; Tue, 7 Aug 2018 16:50:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0BDDF2A713 for ; Tue, 7 Aug 2018 16:50:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 09D272A74E; Tue, 7 Aug 2018 16:50:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 94E032A73B for ; Tue, 7 Aug 2018 16:50:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389118AbeHGTF2 (ORCPT ); Tue, 7 Aug 2018 15:05:28 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:56996 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389003AbeHGTF2 (ORCPT ); Tue, 7 Aug 2018 15:05:28 -0400 X-ASG-Debug-ID: 1533660615-0fb3b01fb342ba20001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id Kd0YbZwU8wOeYaST (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:50:15 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324877; Tue, 07 Aug 2018 12:50:15 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 09/10] dmapool: debug: prevent endless loop in case of corruption Message-ID: <63e9878d-4819-572d-7288-b1b2329f056b@cybernetics.com> X-ASG-Orig-Subj: [PATCH v3 09/10] dmapool: debug: prevent endless loop in case of corruption Date: Tue, 7 Aug 2018 12:50:15 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660615 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 1920 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Prevent a possible endless loop with DMAPOOL_DEBUG enabled if a buggy driver corrupts DMA pool memory. Signed-off-by: Tony Battersby --- Changes since v2: This is closer to the improved version from "dmapool: reduce footprint in struct page" in v2 thanks to a previous patch adding blks_per_alloc. --- linux/mm/dmapool.c.orig 2018-08-06 17:52:53.000000000 -0400 +++ linux/mm/dmapool.c 2018-08-06 17:53:31.000000000 -0400 @@ -454,17 +454,39 @@ void dma_pool_free(struct dma_pool *pool { void *page_vaddr = vaddr - offset; unsigned int chain = page->dma_free_off; + unsigned int free_blks = 0; + while (chain < pool->allocation) { - if (chain != offset) { - chain = *(int *)(page_vaddr + chain); - continue; + if (unlikely(chain == offset)) { + spin_unlock_irqrestore(&pool->lock, flags); + dev_err(pool->dev, + "dma_pool_free %s, dma %pad already free\n", + pool->name, &dma); + return; + } + + /* + * A buggy driver could corrupt the freelist by + * use-after-free, buffer overflow, etc. Besides + * checking for corruption, this also prevents an + * endless loop in case corruption causes a circular + * loop in the freelist. + */ + if (unlikely(++free_blks + page->dma_in_use > + pool->blks_per_alloc)) { + freelist_corrupt: + spin_unlock_irqrestore(&pool->lock, flags); + dev_err(pool->dev, + "dma_pool_free %s, freelist corrupted\n", + pool->name); + return; } - spin_unlock_irqrestore(&pool->lock, flags); - dev_err(pool->dev, - "dma_pool_free %s, dma %pad already free\n", - pool->name, &dma); - return; + + chain = *(int *)(page_vaddr + chain); } + if (unlikely(free_blks + page->dma_in_use != + pool->blks_per_alloc)) + goto freelist_corrupt; } memset(vaddr, POOL_POISON_FREED, pool->size); #endif From patchwork Tue Aug 7 16:51:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Battersby X-Patchwork-Id: 10558935 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E529B13B4 for ; Tue, 7 Aug 2018 16:52:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D100B2A70A for ; Tue, 7 Aug 2018 16:52:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C34962A713; Tue, 7 Aug 2018 16:52:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 258832A70A for ; Tue, 7 Aug 2018 16:52:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387835AbeHGTHi (ORCPT ); Tue, 7 Aug 2018 15:07:38 -0400 Received: from mail.cybernetics.com ([173.71.130.66]:57146 "EHLO mail.cybernetics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731069AbeHGTHi (ORCPT ); Tue, 7 Aug 2018 15:07:38 -0400 X-ASG-Debug-ID: 1533660683-0fb3b01fb342ba60001-ziuLRu Received: from cybernetics.com ([10.157.1.126]) by mail.cybernetics.com with ESMTP id DJIyl8MvvXFmy7ja (version=SSLv3 cipher=DES-CBC3-SHA bits=112 verify=NO); Tue, 07 Aug 2018 12:51:23 -0400 (EDT) X-Barracuda-Envelope-From: tonyb@cybernetics.com X-ASG-Whitelist: Client Received: from [10.157.2.224] (account tonyb HELO [192.168.200.1]) by cybernetics.com (CommuniGate Pro SMTP 5.1.14) with ESMTPSA id 8324890; Tue, 07 Aug 2018 12:51:23 -0400 To: Matthew Wilcox , Christoph Hellwig , Marek Szyprowski , Sathya Prakash , Chaitra P B , Suganath Prabu Subramani , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-scsi@vger.kernel.org" , "MPT-FusionLinux.pdl@broadcom.com" From: Tony Battersby Subject: [PATCH v3 10/10] [SCSI] mpt3sas: replace chain_dma_pool Message-ID: X-ASG-Orig-Subj: [PATCH v3 10/10] [SCSI] mpt3sas: replace chain_dma_pool Date: Tue, 7 Aug 2018 12:51:23 -0400 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Language: en-US X-Barracuda-Connect: UNKNOWN[10.157.1.126] X-Barracuda-Start-Time: 1533660683 X-Barracuda-Encrypted: DES-CBC3-SHA X-Barracuda-URL: https://10.157.1.122:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 7782 X-Virus-Scanned: by bsmtpd at cybernetics.com X-Barracuda-BRTS-Status: 1 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Replace chain_dma_pool with direct calls to dma_alloc_coherent() and dma_free_coherent(). Since the chain lookup can involve hundreds of thousands of allocations, it is worthwile to avoid the overhead of the dma_pool API. Signed-off-by: Tony Battersby --- No changes since v1. The original code called _base_release_memory_pools() before "goto out" if dma_pool_alloc() failed, but this was unnecessary because mpt3sas_base_attach() will call _base_release_memory_pools() after "goto out_free_resources". It may have been that way because the out-of-tree vendor driver (from https://www.broadcom.com/support/download-search) has a slightly-more-complicated error handler there that adjusts max_request_credit, calls _base_release_memory_pools() and then does "goto retry_allocation" under some circumstances, but that is missing from the in-tree driver. diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 569392d..2cb567a 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -4224,6 +4224,134 @@ void mpt3sas_base_clear_st(struct MPT3SAS_ADAPTER *ioc, } /** + * _base_release_chain_lookup - release chain_lookup memory pools + * @ioc: per adapter object + * + * Free memory allocated from _base_allocate_chain_lookup. + */ +static void +_base_release_chain_lookup(struct MPT3SAS_ADAPTER *ioc) +{ + unsigned int chains_avail = 0; + struct chain_tracker *ct; + int i, j; + + if (!ioc->chain_lookup) + return; + + /* + * NOTE + * + * To make this code easier to understand and maintain, the for loops + * and the management of the chains_avail value are designed to be + * similar to the _base_allocate_chain_lookup() function. That way, + * the code for freeing the memory is similar to the code for + * allocating the memory. + */ + for (i = 0; i < ioc->scsiio_depth; i++) { + if (!ioc->chain_lookup[i].chains_per_smid) + break; + + for (j = ioc->chains_per_prp_buffer; + j < ioc->chains_needed_per_io; j++) { + /* + * If chains_avail is 0, then the chain represents a + * real allocation, so free it. + * + * If chains_avail is nonzero, then the chain was + * initialized at an offset from a previous allocation, + * so don't free it. + */ + if (chains_avail == 0) { + ct = &ioc->chain_lookup[i].chains_per_smid[j]; + if (ct->chain_buffer) + dma_free_coherent( + &ioc->pdev->dev, + ioc->chain_allocation_sz, + ct->chain_buffer, + ct->chain_buffer_dma); + chains_avail = ioc->chains_per_allocation; + } + chains_avail--; + } + kfree(ioc->chain_lookup[i].chains_per_smid); + } + + kfree(ioc->chain_lookup); + ioc->chain_lookup = NULL; +} + +/** + * _base_allocate_chain_lookup - allocate chain_lookup memory pools + * @ioc: per adapter object + * @total_sz: external value that tracks total amount of memory allocated + * + * Return: 0 success, anything else error + */ +static int +_base_allocate_chain_lookup(struct MPT3SAS_ADAPTER *ioc, u32 *total_sz) +{ + unsigned int aligned_chain_segment_sz; + const unsigned int align = 16; + unsigned int chains_avail = 0; + struct chain_tracker *ct; + dma_addr_t dma_addr = 0; + void *vaddr = NULL; + int i, j; + + /* Round up the allocation size for alignment. */ + aligned_chain_segment_sz = ioc->chain_segment_sz; + if (aligned_chain_segment_sz % align != 0) + aligned_chain_segment_sz = + ALIGN(aligned_chain_segment_sz, align); + + /* Allocate a page of chain buffers at a time. */ + ioc->chain_allocation_sz = + max_t(unsigned int, aligned_chain_segment_sz, PAGE_SIZE); + + /* Calculate how many chain buffers we can get from one allocation. */ + ioc->chains_per_allocation = + ioc->chain_allocation_sz / aligned_chain_segment_sz; + + for (i = 0; i < ioc->scsiio_depth; i++) { + for (j = ioc->chains_per_prp_buffer; + j < ioc->chains_needed_per_io; j++) { + /* + * Check if there are any chain buffers left in the + * previously-allocated block. + */ + if (chains_avail == 0) { + /* Allocate a new block of chain buffers. */ + vaddr = dma_alloc_coherent( + &ioc->pdev->dev, + ioc->chain_allocation_sz, + &dma_addr, + GFP_KERNEL); + if (!vaddr) { + pr_err(MPT3SAS_FMT + "chain_lookup: dma_alloc_coherent failed\n", + ioc->name); + return -1; + } + chains_avail = ioc->chains_per_allocation; + } + + ct = &ioc->chain_lookup[i].chains_per_smid[j]; + ct->chain_buffer = vaddr; + ct->chain_buffer_dma = dma_addr; + + /* Go to the next chain buffer in the block. */ + vaddr += aligned_chain_segment_sz; + dma_addr += aligned_chain_segment_sz; + *total_sz += ioc->chain_segment_sz; + chains_avail--; + } + } + + return 0; +} + +/** * _base_release_memory_pools - release memory * @ioc: per adapter object * @@ -4235,8 +4363,6 @@ void mpt3sas_base_clear_st(struct MPT3SAS_ADAPTER *ioc, _base_release_memory_pools(struct MPT3SAS_ADAPTER *ioc) { int i = 0; - int j = 0; - struct chain_tracker *ct; struct reply_post_struct *rps; dexitprintk(ioc, pr_info(MPT3SAS_FMT "%s\n", ioc->name, @@ -4326,22 +4452,7 @@ void mpt3sas_base_clear_st(struct MPT3SAS_ADAPTER *ioc, kfree(ioc->hpr_lookup); kfree(ioc->internal_lookup); - if (ioc->chain_lookup) { - for (i = 0; i < ioc->scsiio_depth; i++) { - for (j = ioc->chains_per_prp_buffer; - j < ioc->chains_needed_per_io; j++) { - ct = &ioc->chain_lookup[i].chains_per_smid[j]; - if (ct && ct->chain_buffer) - dma_pool_free(ioc->chain_dma_pool, - ct->chain_buffer, - ct->chain_buffer_dma); - } - kfree(ioc->chain_lookup[i].chains_per_smid); - } - dma_pool_destroy(ioc->chain_dma_pool); - kfree(ioc->chain_lookup); - ioc->chain_lookup = NULL; - } + _base_release_chain_lookup(ioc); } /** @@ -4784,29 +4895,8 @@ void mpt3sas_base_clear_st(struct MPT3SAS_ADAPTER *ioc, total_sz += sz * ioc->scsiio_depth; } - ioc->chain_dma_pool = dma_pool_create("chain pool", &ioc->pdev->dev, - ioc->chain_segment_sz, 16, 0); - if (!ioc->chain_dma_pool) { - pr_err(MPT3SAS_FMT "chain_dma_pool: dma_pool_create failed\n", - ioc->name); + if (_base_allocate_chain_lookup(ioc, &total_sz)) goto out; - } - for (i = 0; i < ioc->scsiio_depth; i++) { - for (j = ioc->chains_per_prp_buffer; - j < ioc->chains_needed_per_io; j++) { - ct = &ioc->chain_lookup[i].chains_per_smid[j]; - ct->chain_buffer = dma_pool_alloc( - ioc->chain_dma_pool, GFP_KERNEL, - &ct->chain_buffer_dma); - if (!ct->chain_buffer) { - pr_err(MPT3SAS_FMT "chain_lookup: " - " pci_pool_alloc failed\n", ioc->name); - _base_release_memory_pools(ioc); - goto out; - } - } - total_sz += ioc->chain_segment_sz; - } dinitprintk(ioc, pr_info(MPT3SAS_FMT "chain pool depth(%d), frame_size(%d), pool_size(%d kB)\n", diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index f02974c..7ee81d5 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -1298,7 +1298,6 @@ struct MPT3SAS_ADAPTER { /* chain */ struct chain_lookup *chain_lookup; struct list_head free_chain_list; - struct dma_pool *chain_dma_pool; ulong chain_pages; u16 max_sges_in_main_message; u16 max_sges_in_chain_message; @@ -1306,6 +1305,8 @@ struct MPT3SAS_ADAPTER { u32 chain_depth; u16 chain_segment_sz; u16 chains_per_prp_buffer; + u32 chain_allocation_sz; + u32 chains_per_allocation; /* hi-priority queue */ u16 hi_priority_smid;