From patchwork Mon Aug 21 11:58:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sricharan Ramabadhran X-Patchwork-Id: 9912355 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F0E2F602D8 for ; Mon, 21 Aug 2017 11:58:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE5EA286E4 for ; Mon, 21 Aug 2017 11:58:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D330628765; Mon, 21 Aug 2017 11:58:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D00B4286E4 for ; Mon, 21 Aug 2017 11:58:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752234AbdHUL6p (ORCPT ); Mon, 21 Aug 2017 07:58:45 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:48894 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751814AbdHUL6o (ORCPT ); Mon, 21 Aug 2017 07:58:44 -0400 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 56B64603AF; Mon, 21 Aug 2017 11:58:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1503316723; bh=ZmdPhD45Wjf6OynXli6WdKayUx9ZZ/7nNKikxFpiVb4=; h=From:To:Cc:Subject:Date:From; b=mry216pj7shElYPJIpcQkQuhAqltZGiT2q/d/HQ1yFbckfHcnadypp/Ff5MUiTDpK 8W8ODOz5Hk6tbWQ0iZ2/vvFgZOE1RwIbuZMnCB8YiF1dtoNMIPqGIG4emeMgX6/osL +tV5K4fCnbIqgbOjLHdm5565CT1fqksqhn7r6fXA= Received: from srichara-linux.qualcomm.com (blr-c-bdr-fw-01_globalnat_allzones-outside.qualcomm.com [103.229.19.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: sricharan@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 46BC6603AF; Mon, 21 Aug 2017 11:58:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1503316722; bh=ZmdPhD45Wjf6OynXli6WdKayUx9ZZ/7nNKikxFpiVb4=; h=From:To:Cc:Subject:Date:From; b=KAtz40gF7KHhINKCWC+0yCcaf881EEXFZrRiTTMSLCrbHLKunCSrmORiNE/BOmlNI CP0Is0nLv0BRrLOwO6t79Otgea5MydUnWZOLmT51+arDeo9ezLsAdYyesNvxHMzUGY LbhdOrwEDsN8+VdF+xYJ+DCLwfqD7WbHSewHh/eE= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 46BC6603AF Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=sricharan@codeaurora.org From: Sricharan R To: absahu@codeaurora.org, vinod.koul@intel.com, andy.gross@linaro.org, david.brown@linaro.org, dan.j.williams@intel.com, linux-arm-msm@vger.kernel.org, linux-soc@vger.kernel.org, dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, dmaengine-owner@vger.kernel.org, stanimir.varbanov@linaro.org, architt@codeaurora.org Cc: sricharan@codeaurora.org Subject: [PATCH V3] dmaengine: qcom-bam: Process multiple pending descriptors Date: Mon, 21 Aug 2017 17:28:30 +0530 Message-Id: <1503316710-1204-1-git-send-email-sricharan@codeaurora.org> X-Mailer: git-send-email 1.9.1 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The bam dmaengine has a circular FIFO to which we add hw descriptors that describes the transaction. The FIFO has space for about 4096 hw descriptors. Currently we add one descriptor and wait for it to complete with interrupt and then add the next pending descriptor. In this way, the FIFO is underutilized since only one descriptor is processed at a time, although there is space in FIFO for the BAM to process more. Instead keep adding descriptors to FIFO till its full, that allows BAM to continue to work on the next descriptor immediately after signalling completion interrupt for the previous descriptor. Also when the client has not set the DMA_PREP_INTERRUPT for a descriptor, then do not configure BAM to trigger a interrupt upon completion of that descriptor. This way we get a interrupt only for the descriptor for which DMA_PREP_INTERRUPT was requested and there signal completion of all the previous completed descriptors. So we still do callbacks for all requested descriptors, but just that the number of interrupts are reduced. CURRENT: ------ ------- --------------- |DES 0| |DESC 1| |DESC 2 + INT | ------ ------- --------------- | | | | | | INTERRUPT: (INT) (INT) (INT) CALLBACK: (CB) (CB) (CB) MTD_SPEEDTEST READ PAGE: 3560 KiB/s MTD_SPEEDTEST WRITE PAGE: 2664 KiB/s IOZONE READ: 2456 KB/s IOZONE WRITE: 1230 KB/s bam dma interrupts (after tests): 96508 CHANGE: ------ ------- ------------- |DES 0| |DESC 1 |DESC 2 + INT | ------ ------- -------------- | | (INT) (CB for 0, 1, 2) MTD_SPEEDTEST READ PAGE: 3860 KiB/s MTD_SPEEDTEST WRITE PAGE: 2837 KiB/s IOZONE READ: 2677 KB/s IOZONE WRITE: 1308 KB/s bam dma interrupts (after tests): 58806 Signed-off-by: Sricharan R Tested-by: Abhishek Sahu Reviewed-by: Andy Gross --- [V3] Removed the use of is_busy variable and corrected terminate_all, added tags Please note that this will cause a merge conflict with https://patchwork.kernel.org/patch/9874691/ if that goes in first. drivers/dma/qcom/bam_dma.c | 169 +++++++++++++++++++++++++++++---------------- 1 file changed, 109 insertions(+), 60 deletions(-) diff --git a/drivers/dma/qcom/bam_dma.c b/drivers/dma/qcom/bam_dma.c index 03c4eb3..ad58634 100644 --- a/drivers/dma/qcom/bam_dma.c +++ b/drivers/dma/qcom/bam_dma.c @@ -46,6 +46,7 @@ #include #include #include +#include #include #include #include @@ -77,6 +78,8 @@ struct bam_async_desc { struct bam_desc_hw *curr_desc; + /* list node for the desc in the bam_chan list of descriptors */ + struct list_head desc_node; enum dma_transfer_direction dir; size_t length; struct bam_desc_hw desc[0]; @@ -346,6 +349,8 @@ struct reg_offset_data { #define BAM_DESC_FIFO_SIZE SZ_32K #define MAX_DESCRIPTORS (BAM_DESC_FIFO_SIZE / sizeof(struct bam_desc_hw) - 1) #define BAM_FIFO_SIZE (SZ_32K - 8) +#define IS_BUSY(chan) (CIRC_SPACE(bchan->tail, bchan->head,\ + MAX_DESCRIPTORS + 1) == 0) struct bam_chan { struct virt_dma_chan vc; @@ -355,8 +360,6 @@ struct bam_chan { /* configuration from device tree */ u32 id; - struct bam_async_desc *curr_txd; /* current running dma */ - /* runtime configuration */ struct dma_slave_config slave; @@ -371,6 +374,8 @@ struct bam_chan { unsigned int initialized; /* is the channel hw initialized? */ unsigned int paused; /* is the channel paused? */ unsigned int reconfigure; /* new slave config? */ + /* list of descriptors currently processed */ + struct list_head desc_list; struct list_head node; }; @@ -538,7 +543,7 @@ static void bam_free_chan(struct dma_chan *chan) vchan_free_chan_resources(to_virt_chan(chan)); - if (bchan->curr_txd) { + if (!list_empty(&bchan->desc_list)) { dev_err(bchan->bdev->dev, "Cannot free busy channel\n"); goto err; } @@ -631,8 +636,6 @@ static struct dma_async_tx_descriptor *bam_prep_slave_sg(struct dma_chan *chan, if (flags & DMA_PREP_INTERRUPT) async_desc->flags |= DESC_FLAG_EOT; - else - async_desc->flags |= DESC_FLAG_INT; async_desc->num_desc = num_alloc; async_desc->curr_desc = async_desc->desc; @@ -680,14 +683,16 @@ static struct dma_async_tx_descriptor *bam_prep_slave_sg(struct dma_chan *chan, static int bam_dma_terminate_all(struct dma_chan *chan) { struct bam_chan *bchan = to_bam_chan(chan); + struct bam_async_desc *async_desc, *tmp; unsigned long flag; LIST_HEAD(head); /* remove all transactions, including active transaction */ spin_lock_irqsave(&bchan->vc.lock, flag); - if (bchan->curr_txd) { - list_add(&bchan->curr_txd->vd.node, &bchan->vc.desc_issued); - bchan->curr_txd = NULL; + list_for_each_entry_safe(async_desc, tmp, + &bchan->desc_list, desc_node) { + list_add(&async_desc->vd.node, &bchan->vc.desc_issued); + list_del(&async_desc->desc_node); } vchan_get_all_descriptors(&bchan->vc, &head); @@ -759,9 +764,9 @@ static int bam_resume(struct dma_chan *chan) */ static u32 process_channel_irqs(struct bam_device *bdev) { - u32 i, srcs, pipe_stts; + u32 i, srcs, pipe_stts, offset, avail; unsigned long flags; - struct bam_async_desc *async_desc; + struct bam_async_desc *async_desc, *tmp; srcs = readl_relaxed(bam_addr(bdev, 0, BAM_IRQ_SRCS_EE)); @@ -781,27 +786,40 @@ static u32 process_channel_irqs(struct bam_device *bdev) writel_relaxed(pipe_stts, bam_addr(bdev, i, BAM_P_IRQ_CLR)); spin_lock_irqsave(&bchan->vc.lock, flags); - async_desc = bchan->curr_txd; - if (async_desc) { - async_desc->num_desc -= async_desc->xfer_len; - async_desc->curr_desc += async_desc->xfer_len; - bchan->curr_txd = NULL; + offset = readl_relaxed(bam_addr(bdev, i, BAM_P_SW_OFSTS)) & + P_SW_OFSTS_MASK; + offset /= sizeof(struct bam_desc_hw); + + /* Number of bytes available to read */ + avail = CIRC_CNT(offset, bchan->head, MAX_DESCRIPTORS + 1); + + list_for_each_entry_safe(async_desc, tmp, + &bchan->desc_list, desc_node) { + /* Not enough data to read */ + if (avail < async_desc->xfer_len) + break; /* manage FIFO */ bchan->head += async_desc->xfer_len; bchan->head %= MAX_DESCRIPTORS; + async_desc->num_desc -= async_desc->xfer_len; + async_desc->curr_desc += async_desc->xfer_len; + avail -= async_desc->xfer_len; + /* - * if complete, process cookie. Otherwise + * if complete, process cookie. Otherwise * push back to front of desc_issued so that * it gets restarted by the tasklet */ - if (!async_desc->num_desc) + if (!async_desc->num_desc) { vchan_cookie_complete(&async_desc->vd); - else + } else { list_add(&async_desc->vd.node, - &bchan->vc.desc_issued); + &bchan->vc.desc_issued); + } + list_del(&async_desc->desc_node); } spin_unlock_irqrestore(&bchan->vc.lock, flags); @@ -863,6 +881,7 @@ static enum dma_status bam_tx_status(struct dma_chan *chan, dma_cookie_t cookie, struct dma_tx_state *txstate) { struct bam_chan *bchan = to_bam_chan(chan); + struct bam_async_desc *async_desc; struct virt_dma_desc *vd; int ret; size_t residue = 0; @@ -878,11 +897,17 @@ static enum dma_status bam_tx_status(struct dma_chan *chan, dma_cookie_t cookie, spin_lock_irqsave(&bchan->vc.lock, flags); vd = vchan_find_desc(&bchan->vc, cookie); - if (vd) + if (vd) { residue = container_of(vd, struct bam_async_desc, vd)->length; - else if (bchan->curr_txd && bchan->curr_txd->vd.tx.cookie == cookie) - for (i = 0; i < bchan->curr_txd->num_desc; i++) - residue += bchan->curr_txd->curr_desc[i].size; + } else { + list_for_each_entry(async_desc, &bchan->desc_list, desc_node) { + if (async_desc->vd.tx.cookie != cookie) + continue; + + for (i = 0; i < async_desc->num_desc; i++) + residue += async_desc->curr_desc[i].size; + } + } spin_unlock_irqrestore(&bchan->vc.lock, flags); @@ -923,63 +948,86 @@ static void bam_start_dma(struct bam_chan *bchan) { struct virt_dma_desc *vd = vchan_next_desc(&bchan->vc); struct bam_device *bdev = bchan->bdev; - struct bam_async_desc *async_desc; + struct bam_async_desc *async_desc = NULL; struct bam_desc_hw *desc; struct bam_desc_hw *fifo = PTR_ALIGN(bchan->fifo_virt, sizeof(struct bam_desc_hw)); int ret; + unsigned int avail; + struct dmaengine_desc_callback cb; lockdep_assert_held(&bchan->vc.lock); if (!vd) return; - list_del(&vd->node); - - async_desc = container_of(vd, struct bam_async_desc, vd); - bchan->curr_txd = async_desc; - ret = pm_runtime_get_sync(bdev->dev); if (ret < 0) return; - /* on first use, initialize the channel hardware */ - if (!bchan->initialized) - bam_chan_init_hw(bchan, async_desc->dir); + while (vd && !IS_BUSY(bchan)) { + list_del(&vd->node); - /* apply new slave config changes, if necessary */ - if (bchan->reconfigure) - bam_apply_new_config(bchan, async_desc->dir); + async_desc = container_of(vd, struct bam_async_desc, vd); - desc = bchan->curr_txd->curr_desc; + /* on first use, initialize the channel hardware */ + if (!bchan->initialized) + bam_chan_init_hw(bchan, async_desc->dir); - if (async_desc->num_desc > MAX_DESCRIPTORS) - async_desc->xfer_len = MAX_DESCRIPTORS; - else - async_desc->xfer_len = async_desc->num_desc; + /* apply new slave config changes, if necessary */ + if (bchan->reconfigure) + bam_apply_new_config(bchan, async_desc->dir); - /* set any special flags on the last descriptor */ - if (async_desc->num_desc == async_desc->xfer_len) - desc[async_desc->xfer_len - 1].flags = - cpu_to_le16(async_desc->flags); - else - desc[async_desc->xfer_len - 1].flags |= - cpu_to_le16(DESC_FLAG_INT); + desc = async_desc->curr_desc; + avail = CIRC_SPACE(bchan->tail, bchan->head, + MAX_DESCRIPTORS + 1); + + if (async_desc->num_desc > avail) + async_desc->xfer_len = avail; + else + async_desc->xfer_len = async_desc->num_desc; + + /* set any special flags on the last descriptor */ + if (async_desc->num_desc == async_desc->xfer_len) + desc[async_desc->xfer_len - 1].flags = + cpu_to_le16(async_desc->flags); - if (bchan->tail + async_desc->xfer_len > MAX_DESCRIPTORS) { - u32 partial = MAX_DESCRIPTORS - bchan->tail; + vd = vchan_next_desc(&bchan->vc); - memcpy(&fifo[bchan->tail], desc, - partial * sizeof(struct bam_desc_hw)); - memcpy(fifo, &desc[partial], (async_desc->xfer_len - partial) * + dmaengine_desc_get_callback(&async_desc->vd.tx, &cb); + + /* + * An interrupt is generated at this desc, if + * - FIFO is FULL. + * - No more descriptors to add. + * - If a callback completion was requested for this DESC, + * In this case, BAM will deliver the completion callback + * for this desc and continue processing the next desc. + */ + if (((avail <= async_desc->xfer_len) || !vd || + dmaengine_desc_callback_valid(&cb)) && + !(async_desc->flags & DESC_FLAG_EOT)) + desc[async_desc->xfer_len - 1].flags |= + cpu_to_le16(DESC_FLAG_INT); + + if (bchan->tail + async_desc->xfer_len > MAX_DESCRIPTORS) { + u32 partial = MAX_DESCRIPTORS - bchan->tail; + + memcpy(&fifo[bchan->tail], desc, + partial * sizeof(struct bam_desc_hw)); + memcpy(fifo, &desc[partial], + (async_desc->xfer_len - partial) * sizeof(struct bam_desc_hw)); - } else { - memcpy(&fifo[bchan->tail], desc, - async_desc->xfer_len * sizeof(struct bam_desc_hw)); - } + } else { + memcpy(&fifo[bchan->tail], desc, + async_desc->xfer_len * + sizeof(struct bam_desc_hw)); + } - bchan->tail += async_desc->xfer_len; - bchan->tail %= MAX_DESCRIPTORS; + bchan->tail += async_desc->xfer_len; + bchan->tail %= MAX_DESCRIPTORS; + list_add_tail(&async_desc->desc_node, &bchan->desc_list); + } /* ensure descriptor writes and dma start not reordered */ wmb(); @@ -1008,7 +1056,7 @@ static void dma_tasklet(unsigned long data) bchan = &bdev->channels[i]; spin_lock_irqsave(&bchan->vc.lock, flags); - if (!list_empty(&bchan->vc.desc_issued) && !bchan->curr_txd) + if (!list_empty(&bchan->vc.desc_issued) && !IS_BUSY(bchan)) bam_start_dma(bchan); spin_unlock_irqrestore(&bchan->vc.lock, flags); } @@ -1029,7 +1077,7 @@ static void bam_issue_pending(struct dma_chan *chan) spin_lock_irqsave(&bchan->vc.lock, flags); /* if work pending and idle, start a transaction */ - if (vchan_issue_pending(&bchan->vc) && !bchan->curr_txd) + if (vchan_issue_pending(&bchan->vc) && !IS_BUSY(bchan)) bam_start_dma(bchan); spin_unlock_irqrestore(&bchan->vc.lock, flags); @@ -1129,6 +1177,7 @@ static void bam_channel_init(struct bam_device *bdev, struct bam_chan *bchan, vchan_init(&bchan->vc, &bdev->common); bchan->vc.desc_free = bam_dma_free_desc; + INIT_LIST_HEAD(&bchan->desc_list); } static const struct of_device_id bam_of_match[] = {