From patchwork Mon Sep 24 13:00:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 10612463 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 223E0157B for ; Mon, 24 Sep 2018 12:59:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1317D29C95 for ; Mon, 24 Sep 2018 12:59:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 06A5A29C9C; Mon, 24 Sep 2018 12:59:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98E0A29C95 for ; Mon, 24 Sep 2018 12:59:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728841AbeIXTBq (ORCPT ); Mon, 24 Sep 2018 15:01:46 -0400 Received: from fllv0015.ext.ti.com ([198.47.19.141]:55630 "EHLO fllv0015.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729283AbeIXTBq (ORCPT ); Mon, 24 Sep 2018 15:01:46 -0400 Received: from dflxv15.itg.ti.com ([128.247.5.124]) by fllv0015.ext.ti.com (8.15.2/8.15.2) with ESMTP id w8OCxdnK004410; Mon, 24 Sep 2018 07:59:39 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1537793979; bh=iGA0h29HimyR/dGnF9Na2CM5ts5rq1igp4wAbTLzK6s=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=gWd/oVYL4KKmcTv1TP6LVqcmybLGMmT3240J241I26gawCKzMa//+MBDZYY71EUtk ERWyqMaK7eKTtizqZ7f9Ux6dpnWNnHqQwtmslj1rHOfQblE9L8eCUW31OrA4AmHzDt mKJu3rLApofjutHNPCoa8LRSl3MjiEEV8GDDqrt8= Received: from DFLE115.ent.ti.com (dfle115.ent.ti.com [10.64.6.36]) by dflxv15.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxddC017076; Mon, 24 Sep 2018 07:59:39 -0500 Received: from DFLE105.ent.ti.com (10.64.6.26) by DFLE115.ent.ti.com (10.64.6.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 24 Sep 2018 07:59:39 -0500 Received: from dflp32.itg.ti.com (10.64.6.15) by DFLE105.ent.ti.com (10.64.6.26) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1466.3 via Frontend Transport; Mon, 24 Sep 2018 07:59:38 -0500 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by dflp32.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxZsE012211; Mon, 24 Sep 2018 07:59:37 -0500 From: Peter Ujfalusi To: , CC: , Subject: [RFC 1/5] dmaengine: Add support for reporting DMA cached data amount Date: Mon, 24 Sep 2018 16:00:17 +0300 Message-ID: <20180924130021.20530-2-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180924130021.20530-1-peter.ujfalusi@ti.com> References: <20180924130021.20530-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A DMA hardware can have big cache or FIFO and the amount of data sitting in the DMA fabric can be an interest for the clients. For example in audio we want to know the delay in the data flow and in case the DMA have significantly large FIFO/cache, it can affect the latenc/delay Signed-off-by: Peter Ujfalusi --- drivers/dma/dmaengine.h | 7 +++++++ include/linux/dmaengine.h | 2 ++ 2 files changed, 9 insertions(+) diff --git a/drivers/dma/dmaengine.h b/drivers/dma/dmaengine.h index 501c0b063f85..80214270b70e 100644 --- a/drivers/dma/dmaengine.h +++ b/drivers/dma/dmaengine.h @@ -77,6 +77,7 @@ static inline enum dma_status dma_cookie_status(struct dma_chan *chan, state->last = complete; state->used = used; state->residue = 0; + state->cached = 0; } return dma_async_is_complete(cookie, complete, used); } @@ -87,6 +88,12 @@ static inline void dma_set_residue(struct dma_tx_state *state, u32 residue) state->residue = residue; } +static inline void dma_set_cached(struct dma_tx_state *state, u32 cached) +{ + if (state) + state->cached = cached; +} + struct dmaengine_desc_callback { dma_async_tx_callback callback; dma_async_tx_callback_result callback_result; diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h index 10ff71b13eef..16c9d021988a 100644 --- a/include/linux/dmaengine.h +++ b/include/linux/dmaengine.h @@ -701,11 +701,13 @@ static inline struct dma_async_tx_descriptor *txd_next(struct dma_async_tx_descr * @residue: the remaining number of bytes left to transmit * on the selected transfer for states DMA_IN_PROGRESS and * DMA_PAUSED if this is implemented in the driver, else 0 + * @cached: amount of data in bytes cached by the DMA. */ struct dma_tx_state { dma_cookie_t last; dma_cookie_t used; u32 residue; + u32 cached; }; /** From patchwork Mon Sep 24 13:00:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 10612467 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 54A296CB for ; Mon, 24 Sep 2018 12:59:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 42BFC29C95 for ; Mon, 24 Sep 2018 12:59:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 333C829C9C; Mon, 24 Sep 2018 12:59:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CFAE629C95 for ; Mon, 24 Sep 2018 12:59:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729988AbeIXTBt (ORCPT ); Mon, 24 Sep 2018 15:01:49 -0400 Received: from lelv0142.ext.ti.com ([198.47.23.249]:37152 "EHLO lelv0142.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728863AbeIXTBt (ORCPT ); Mon, 24 Sep 2018 15:01:49 -0400 Received: from dlelxv90.itg.ti.com ([172.17.2.17]) by lelv0142.ext.ti.com (8.15.2/8.15.2) with ESMTP id w8OCxhOI130456; Mon, 24 Sep 2018 07:59:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1537793983; bh=UMle9PjzDQ347d02REkPEFIsAw5Sb4LjUN6kXOVQPJk=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=xgWtLJOSFsUhbi88+rFw3bPwU6wIpn2d1Q3KgQMiIxhr6aGDhqH2yUe9TQmYt9BbT 6xLTc7jabLdPItot/rZt7Eo2P3bUFzH1lH+3swQguO9Y1NmaeyVd16TcUGW0jXiNAj Z6dKgGSO/6Ft2GUmmSShSabp3FNBwoL3F0r7fNVQ= Received: from DLEE114.ent.ti.com (dlee114.ent.ti.com [157.170.170.25]) by dlelxv90.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxh71006487; Mon, 24 Sep 2018 07:59:43 -0500 Received: from DLEE115.ent.ti.com (157.170.170.26) by DLEE114.ent.ti.com (157.170.170.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 24 Sep 2018 07:59:41 -0500 Received: from dflp32.itg.ti.com (10.64.6.15) by DLEE115.ent.ti.com (157.170.170.26) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1466.3 via Frontend Transport; Mon, 24 Sep 2018 07:59:41 -0500 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by dflp32.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxZsF012211; Mon, 24 Sep 2018 07:59:39 -0500 From: Peter Ujfalusi To: , CC: , Subject: [RFC 2/5] dmaengine: Add function to request slave channel from a dma_device Date: Mon, 24 Sep 2018 16:00:18 +0300 Message-ID: <20180924130021.20530-3-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180924130021.20530-1-peter.ujfalusi@ti.com> References: <20180924130021.20530-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP dma_get_any_slave_channel() would skip using the filter function, which in some cases needed to be executed before the alloc_chan_resources callback to make sure that all parameters are provided for the slave channel. Signed-off-by: Peter Ujfalusi --- drivers/dma/dmaengine.c | 20 ++++++++++++++++++++ include/linux/dmaengine.h | 2 ++ 2 files changed, 22 insertions(+) diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c index 27b6d7c2d8a0..b80f1bdd8813 100644 --- a/drivers/dma/dmaengine.c +++ b/drivers/dma/dmaengine.c @@ -634,6 +634,26 @@ struct dma_chan *dma_get_any_slave_channel(struct dma_device *device) } EXPORT_SYMBOL_GPL(dma_get_any_slave_channel); +struct dma_chan *dmadev_get_slave_channel(struct dma_device *device, + dma_filter_fn fn, void *fn_param) +{ + dma_cap_mask_t mask; + struct dma_chan *chan; + + dma_cap_zero(mask); + dma_cap_set(DMA_SLAVE, mask); + + /* lock against __dma_request_channel */ + mutex_lock(&dma_list_mutex); + + chan = find_candidate(device, &mask, fn, fn_param); + + mutex_unlock(&dma_list_mutex); + + return IS_ERR(chan) ? NULL : chan; +} +EXPORT_SYMBOL_GPL(dmadev_get_slave_channel); + /** * __dma_request_channel - try to allocate an exclusive channel * @mask: capabilities that the channel must satisfy diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h index 16c9d021988a..32010c6ae13a 100644 --- a/include/linux/dmaengine.h +++ b/include/linux/dmaengine.h @@ -1542,6 +1542,8 @@ void dma_async_device_unregister(struct dma_device *device); void dma_run_dependencies(struct dma_async_tx_descriptor *tx); struct dma_chan *dma_get_slave_channel(struct dma_chan *chan); struct dma_chan *dma_get_any_slave_channel(struct dma_device *device); +struct dma_chan *dmadev_get_slave_channel(struct dma_device *device, + dma_filter_fn fn, void *fn_param); #define dma_request_channel(mask, x, y) __dma_request_channel(&(mask), x, y) #define dma_request_slave_channel_compat(mask, x, y, dev, name) \ __dma_request_slave_channel_compat(&(mask), x, y, dev, name) From patchwork Mon Sep 24 13:00:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 10612469 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 37FF06CB for ; Mon, 24 Sep 2018 12:59:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 278D629C95 for ; Mon, 24 Sep 2018 12:59:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C2AF29C9C; Mon, 24 Sep 2018 12:59:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EC6629C95 for ; Mon, 24 Sep 2018 12:59:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730675AbeIXTBv (ORCPT ); Mon, 24 Sep 2018 15:01:51 -0400 Received: from lelv0143.ext.ti.com ([198.47.23.248]:43610 "EHLO lelv0143.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729353AbeIXTBu (ORCPT ); Mon, 24 Sep 2018 15:01:50 -0400 Received: from dlelxv90.itg.ti.com ([172.17.2.17]) by lelv0143.ext.ti.com (8.15.2/8.15.2) with ESMTP id w8OCxjAl004241; Mon, 24 Sep 2018 07:59:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1537793985; bh=n/7xIcJLWKloFO+dFJtYLBeDdmrvzhwgJpuXY04F/lM=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=gxjL+Zfx8y/+UWeoWtMZYbBb6ER16YIN8stpkztJLUqruwmaWJhZexaAF8LNCmwXQ gwznIbmA1MAeJl5HMMkaQXn3hqqkjKFFHy5Zjbxrjx1PQhUEmO42wG5hQWJI9kEsfY L4LFMXb/uBk4tpgtP+9UdczuAsJL2T0UR2FtfHiw= Received: from DFLE110.ent.ti.com (dfle110.ent.ti.com [10.64.6.31]) by dlelxv90.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxiEq006506; Mon, 24 Sep 2018 07:59:44 -0500 Received: from DFLE104.ent.ti.com (10.64.6.25) by DFLE110.ent.ti.com (10.64.6.31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 24 Sep 2018 07:59:42 -0500 Received: from dflp32.itg.ti.com (10.64.6.15) by DFLE104.ent.ti.com (10.64.6.25) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1466.3 via Frontend Transport; Mon, 24 Sep 2018 07:59:42 -0500 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by dflp32.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxZsG012211; Mon, 24 Sep 2018 07:59:41 -0500 From: Peter Ujfalusi To: , CC: , Subject: [RFC 3/5] dt-bindings: dma: ti: Add document for K3 UDMA Date: Mon, 24 Sep 2018 16:00:19 +0300 Message-ID: <20180924130021.20530-4-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180924130021.20530-1-peter.ujfalusi@ti.com> References: <20180924130021.20530-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP New binding document for Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P). UDMA-P is introduced as part of the K3 architecture and can be found on AM65x SoC. Signed-off-by: Peter Ujfalusi --- .../devicetree/bindings/dma/ti/k3-udma.txt | 134 ++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 Documentation/devicetree/bindings/dma/ti/k3-udma.txt diff --git a/Documentation/devicetree/bindings/dma/ti/k3-udma.txt b/Documentation/devicetree/bindings/dma/ti/k3-udma.txt new file mode 100644 index 000000000000..4f79d2d30c5c --- /dev/null +++ b/Documentation/devicetree/bindings/dma/ti/k3-udma.txt @@ -0,0 +1,134 @@ +* Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P) + +The UDMA-P is intended to perform similar (but significantly upgraded) functions +as the packet-oriented DMA used on previous SoC devices. The UDMA-P module +supports the transmission and reception of various packet types. The UDMA-P is +architected to facilitate the segmentation and reassembly of SoC DMA data +structure compliant packets to/from smaller data blocks that are natively +compatible with the specific requirements of each connected peripheral. Multiple +Tx and Rx channels are provided within the DMA which allow multiple segmentation +or reassembly operations to be ongoing. The DMA controller maintains state +information for each of the channels which allows packet segmentation and +reassembly operations to be time division multiplexed between channels in order +to share the underlying DMA hardware. An external DMA scheduler is used to +control the ordering and rate at which this multiplexing occurs for Transmit +operations. The ordering and rate of Receive operations is indirectly controlled +by the order in which blocks are pushed into the DMA on the Rx PSI-L interface. + +The UDMA-P also supports acting as both a UTC and UDMA-C for its internal +channels. Channels in the UDMA-P can be configured to be either Packet-Based or +Third-Party channels on a channel by channel basis. + +Required properties: +-------------------- +- compatible: Should be + "ti,am654-navss-main-udmap" for am654 main NAVSS UDMAP + "ti,am654-navss-mcu-udmap" for am654 mcu NAVSS UDMAP +- #dma-cells: Should be set to <3>. + - The first parameter is a phandle to the remote PSI-L + endpoint + - The second parameter is the thread offset within the + remote thread ID range + - The third parameter is the channel direction. +- reg: Memory map of UDMAP +- reg-names: "gcfg", "rchanrt", "tchanrt" +- interrupt-parent: phandle for "ti,sci-irq" irqchip +- ti,ringacc: phandle for the ring accelerator node +- ti,psil-base: PSI-L thread ID base of the UDMAP channels +- ti,sci: phandle on TI-SCI compatible System controller node +- ti,sci-dev-id: TI-SCI device id +- ti,sci-rm-range-tchan: UDMA tchan resource list in pairs of type and subtype +- ti,sci-rm-range-rchan: UDMA rchan resource list in pairs of type and subtype +- ti,sci-rm-range-rflow: UDMA rflow resource list in pairs of type and subtype + +For PSI-L thread management the parent NAVSS node must have: +- ti,sci: phandle on TI-SCI compatible System controller node +- ti,sci-dev-id: TI-SCI device id of the NAVSS instance + +Remote PSI-L endpoint + +Required properties: +-------------------- +- ti,psil-base: PSI-L thread ID base of the endpoint + +Within the PSI-L endpoint node thread configuration subnodes must present with: +ti,psil-configX naming convention, where X is the thread ID offset. + +Configuration node Required properties: +-------------------- +- linux,udma-mode: Channel mode, can be: + - UDMA_PKT_MODE: for Packet mode channels (peripherals) + - UDMA_TR_MODE: for Third-Party mode + +Configuration node Optional properties: +-------------------- +- statictr-type: In case the remote endpoint requires StaticTR + configuration: + - PSIL_STATIC_TR_XY: XY type of StaticTR + - PSIL_STATIC_TR_MCAN: MCAN type of StaticTR +- ti,channel-tpl: Channel Throughput level: + 0 / or not present - normal channel + 1 - High Throughput channel +- ti,needs-epib: If the endpoint require EPIB to be present in the + descriptor. +- ti,psd-size: Size of the Protocol Specific Data section of the + descriptor. + +Example: + +main_navss: main_navss { + compatible = "simple-bus"; + #address-cells = <2>; + #size-cells = <2>; + dma-coherent; + dma-ranges; + ranges; + + ti,sci = <&dmsc>; + ti,sci-dev-id = <118>; + + main_udmap: udmap@31150000 { + compatible = "ti,am654-navss-main-udmap"; + reg = <0x0 0x31150000 0x0 0x100>, + <0x0 0x34000000 0x0 0x100000>, + <0x0 0x35000000 0x0 0x100000>; + reg-names = "gcfg", "rchanrt", "tchanrt"; + #dma-cells = <3>; + + ti,ringacc = <&ringacc>; + ti,psil-base = <0x1000>; + + interrupt-parent = <&main_udmass_inta>; + + ti,sci = <&dmsc>; + ti,sci-dev-id = <188>; + + ti,sci-rm-range-tchan = <0x6 0x1>, /* TX_HCHAN */ + <0x6 0x2>; /* TX_CHAN */ + ti,sci-rm-range-rchan = <0x6 0x4>, /* RX_HCHAN */ + <0x6 0x5>; /* RX_CHAN */ + ti,sci-rm-range-rflow = <0x6 0x6>; /* GP RFLOW */ + }; +}; + +pdma0: pdma@2a41000 { + compatible = "ti,am654-pdma"; + reg = <0x0 0x02A41000 0x0 0x400>; + reg-names = "eccaggr_cfg"; + + ti,psil-base = <0x4400>; + + /* ti,psil-config0-2 */ + UDMA_PDMA_TR_XY(0); + UDMA_PDMA_TR_XY(1); + UDMA_PDMA_TR_XY(2); +}; + +mcasp0: mcasp@02B00000 { +... + /* tx: pdma0-0, rx: pdma0-0 */ + dmas = <&main_udmap &pdma0 0 UDMA_DIR_TX>, + <&main_udmap &pdma0 0 UDMA_DIR_RX>; + dma-names = "tx", "rx"; +... +}; From patchwork Mon Sep 24 13:00:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 10612473 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 30A76157B for ; Mon, 24 Sep 2018 12:59:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16B4729C95 for ; Mon, 24 Sep 2018 12:59:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 09EC329C9C; Mon, 24 Sep 2018 12:59:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B2B529C99 for ; Mon, 24 Sep 2018 12:59:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728863AbeIXTBy (ORCPT ); Mon, 24 Sep 2018 15:01:54 -0400 Received: from lelv0142.ext.ti.com ([198.47.23.249]:37162 "EHLO lelv0142.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730361AbeIXTBy (ORCPT ); Mon, 24 Sep 2018 15:01:54 -0400 Received: from dlelxv90.itg.ti.com ([172.17.2.17]) by lelv0142.ext.ti.com (8.15.2/8.15.2) with ESMTP id w8OCxjoE130462; Mon, 24 Sep 2018 07:59:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1537793985; bh=DH2Q3Zz1kGTycu9eLzvm2xgn7kMaOY9jJ0q5yBw6Lo4=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=GAbBQS8dldtGKPdn7ihVYX75BUc9Q4odYRZMBv4agfgKa+e5AyGayUips2KX063dx YdJPIs096TI/AWbbmlgLxHHuCpWMTb1ctbUHTvrDgnsk+ITFMOj8wd2s2lyYeF62zf Bjx5ZR8tyDS0EhIq4TQG3u96yUDFi6gRC+tPC67Q= Received: from DLEE106.ent.ti.com (dlee106.ent.ti.com [157.170.170.36]) by dlelxv90.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxjNX006526; Mon, 24 Sep 2018 07:59:45 -0500 Received: from DLEE102.ent.ti.com (157.170.170.32) by DLEE106.ent.ti.com (157.170.170.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 24 Sep 2018 07:59:44 -0500 Received: from dflp32.itg.ti.com (10.64.6.15) by DLEE102.ent.ti.com (157.170.170.32) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1466.3 via Frontend Transport; Mon, 24 Sep 2018 07:59:44 -0500 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by dflp32.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxZsH012211; Mon, 24 Sep 2018 07:59:42 -0500 From: Peter Ujfalusi To: , CC: , Subject: [RFC 4/5] dmaengine: ti: New driver for K3 UDMA Date: Mon, 24 Sep 2018 16:00:20 +0300 Message-ID: <20180924130021.20530-5-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180924130021.20530-1-peter.ujfalusi@ti.com> References: <20180924130021.20530-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP DMA driver for Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P) The UDMA-P is intended to perform similar (but significantly upgraded) functions as the packet-oriented DMA used on previous SoC devices. The UDMA-P module supports the transmission and reception of various packet types. The UDMA-P is architected to facilitate the segmentation and reassembly of SoC DMA data structure compliant packets to/from smaller data blocks that are natively compatible with the specific requirements of each connected peripheral. Multiple Tx and Rx channels are provided within the DMA which allow multiple segmentation or reassembly operations to be ongoing. The DMA controller maintains state information for each of the channels which allows packet segmentation and reassembly operations to be time division multiplexed between channels in order to share the underlying DMA hardware. An external DMA scheduler is used to control the ordering and rate at which this multiplexing occurs for Transmit operations. The ordering and rate of Receive operations is indirectly controlled by the order in which blocks are pushed into the DMA on the Rx PSI-L interface. The UDMA-P also supports acting as both a UTC and UDMA-C for its internal channels. Channels in the UDMA-P can be configured to be either Packet-Based or Third-Party channels on a channel by channel basis. The initial driver supports: - MEM_TO_MEM (TR mode) - DEV_TO_MEM (Packet / TR mode) - MEM_TO_DEV (Packet / TR mode) - Cyclic (Packet / TR mode) - Metadata for descriptors Signed-off-by: Peter Ujfalusi --- drivers/dma/ti/Kconfig | 12 + drivers/dma/ti/Makefile | 1 + drivers/dma/ti/k3-udma.c | 3623 +++++++++++++++++++++++++++++ drivers/dma/ti/k3-udma.h | 132 ++ include/dt-bindings/dma/k3-udma.h | 26 + 5 files changed, 3794 insertions(+) create mode 100644 drivers/dma/ti/k3-udma.c create mode 100644 drivers/dma/ti/k3-udma.h create mode 100644 include/dt-bindings/dma/k3-udma.h diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig index e5e74e1361dc..db73371de206 100644 --- a/drivers/dma/ti/Kconfig +++ b/drivers/dma/ti/Kconfig @@ -33,5 +33,17 @@ config DMA_OMAP Enable support for the TI sDMA (System DMA or DMA4) controller. This DMA engine is found on OMAP and DRA7xx parts. +config TI_K3_NAVSS_UDMA + tristate "Texas Instruments UDMA support" + depends on ARCH_K3 || COMPILE_TEST + depends on TI_SCI_PROTOCOL + select DMA_ENGINE + select DMA_VIRTUAL_CHANNELS + select TI_K3_NAVSS_RINGACC + default y + help + Enable support for the TI UDMA (Unified DMA) controller. This + DMA engine is used in AM65x. + config TI_DMA_CROSSBAR bool diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile index 113e59ec9c32..911f7c723979 100644 --- a/drivers/dma/ti/Makefile +++ b/drivers/dma/ti/Makefile @@ -2,4 +2,5 @@ obj-$(CONFIG_TI_CPPI41) += cppi41.o obj-$(CONFIG_TI_EDMA) += edma.o obj-$(CONFIG_DMA_OMAP) += omap-dma.o +obj-$(CONFIG_TI_K3_NAVSS_UDMA) += k3-udma.o obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c new file mode 100644 index 000000000000..2a2a2c8b1fc2 --- /dev/null +++ b/drivers/dma/ti/k3-udma.c @@ -0,0 +1,3623 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com + * Author: Peter Ujfalusi + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../virt-dma.h" +#include "k3-udma.h" + +/* element sizes */ +#define UDMA_ELSIZE_8 0 +#define UDMA_ELSIZE_16 1 +#define UDMA_ELSIZE_24 2 +#define UDMA_ELSIZE_32 3 +#define UDMA_ELSIZE_64 4 + +static const u8 elsize_bytes[] = { + [UDMA_ELSIZE_8] = 1, + [UDMA_ELSIZE_16] = 2, + [UDMA_ELSIZE_24] = 3, + [UDMA_ELSIZE_32] = 4, + [UDMA_ELSIZE_64] = 8, +}; + +#define CPPI50_TR_FLAGS_TYPE(x) (x << 0) +#define CPPI50_TR_FLAGS_WAIT (1 << 5) +/* + * EVENT_SIZE: + * 0: Only generate event when the TR is completed + * 1: Event is generated when ICNT1 is decremented by 1 + * 2: Event is generated when ICNT2 is decremented by 1 + * 3: Event is generated when ICNT3 is decremented by 1 + */ +#define CPPI50_TR_FLAGS_EVENT_SIZE(x) (((x) & 0x3) << 6) +/* + * TRIGGERx: + * 0: No trigger + * 1: Global Trigger 0 for the channel + * 2: Global Trigger 1 for the channel + * 3: Local Event for the channel + * + * TRIGGER_TYPEx: + * 0: ICNT1 can be decremented by 1 on trigger + * 1: ICNT2 can be decremented by 1 on trigger + * 2: ICNT3 can be decremented by 1 on trigger + * 3: The entire TR is waiting for the trigger + */ +#define CPPI50_TR_FLAGS_TRIGGER0(x) (((x) & 0x3) << 8) +#define CPPI50_TR_FLAGS_TRIGGER0_TYPE(x) (((x) & 0x3) << 10) +#define CPPI50_TR_FLAGS_TRIGGER1(x) (((x) & 0x3) << 10) +#define CPPI50_TR_FLAGS_TRIGGER1_TYPE(x) (((x) & 0x3) << 14) + +#define CPPI50_TR_FLAGS_SUPR_EVT (1 << 26) +#define CPPI50_TR_FLAGS_EOP (1 << 31) + +/* Transfer Request Type 0: One Dimensional Transfer */ +struct cppi50_tr_req_type0 { + u32 flags; + u16 icnt0; + u16 unused; + u64 addr; +} __packed; + +/* Transfer Request Type 1: Two Dimensional Transfer */ +struct cppi50_tr_req_type1 { + u32 flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u32 padd[3]; /* full size is 32 bytes */ +} __packed; + +/* Transfer Request Type 2: Three Dimensional Transfer */ +struct cppi50_tr_req_type2 { + u32 flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u16 icnt2; + u16 unused; + s32 dim2; + u32 padd; /* full size is 32 bytes */ +} __packed; + +/* Transfer Request Type 9: Four Dimensional Block Copy with Repacking */ +struct cppi50_tr_req_type9 { + u32 flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u16 icnt2; + u16 icnt3; + s32 dim2; + s32 dim3; + u32 fmtflags; + s32 ddim1; + u64 daddr; + s32 ddim2; + s32 ddim3; + u16 dicnt0; + u16 dicnt1; + u16 dicnt2; + u16 dicnt3; +} __packed; + +/* Transfer Request Type 10: Two Dimensional BLock Copy */ +struct cppi50_tr_req_type10 { + u32 flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u32 unused[3]; + u32 fmtflags; + s32 ddim1; + u64 daddr; + u32 padd[4]; /* full size is 64 bytes */ +} __packed; + +/* + * Transfer Request Type 15: Four Dimensional Block Copy with Repacking + * and Indirection Support + */ +struct cppi50_tr_req_type15 { + u32 flags; + u16 icnt0; + u16 icnt1; + u64 addr; + s32 dim1; + u16 icnt2; + u16 icnt3; + s32 dim2; + s32 dim3; + u32 unused; + s32 ddim1; + u64 daddr; + s32 ddim2; + s32 ddim3; + u16 dicnt0; + u16 dicnt1; + u16 dicnt2; + u16 dicnt3; +} __packed; + +struct cppi50_tr_resp { + u8 status; + u8 reserved; + u8 cmd_id; + u8 flags; +} __packed; + +#define CPPI50_TRDESC_W0_LAST_ENTRY(x) (((x) & 0x3fff) << 0) +#define CPPI50_TRDESC_W0_RELOAD_IDX(x) (((x) & 0x3fff) << 14) +#define CPPI50_TRDESC_W0_RELOAD_CNT(x) (((x) & 0x1ff) << 20) +#define CPPI50_TRDESC_W0_TYPE (0x3 << 30) + +#define CPPI50_TRDESC_W1_FLOWID(x) (((x) & 0x3fff) << 0) +#define CPPI50_TRDESC_W1_PACKETID(x) (((x) & 0x3ff) << 14) +#define CPPI50_TRDESC_W1_TR_SIZE_16 (0 << 24) +#define CPPI50_TRDESC_W1_TR_SIZE_32 (1 << 24) +#define CPPI50_TRDESC_W1_TR_SIZE_64 (2 << 24) +#define CPPI50_TRDESC_W1_TR_SIZE_128 (3 << 24) + +/* CPPI 5.0 Transfer Request Descriptor */ +struct cppi50_tr_req_desc { + u32 packet_info[4]; +} __aligned(64) __packed; + +struct udma_static_tr { + u8 elsize; /* RPSTR0 */ + u16 elcnt; /* RPSTR0 */ + u16 bstcnt; /* RPSTR1 */ +}; + +#define K3_UDMA_MAX_RFLOWS 1024 + +struct udma_chan; + +enum udma_mmr { + MMR_GCFG = 0, + MMR_RCHANRT, + MMR_TCHANRT, + MMR_LAST, +}; + +static const char * const mmr_names[] = { "gcfg", "rchanrt", "tchanrt" }; + +struct udma_tchan { + void __iomem *reg_rt; + + int id; + struct k3_nav_ring *t_ring; /* Transmit ring */ + struct k3_nav_ring *tc_ring; /* Transmit Completion ring */ +}; + +struct udma_rchan { + void __iomem *reg_rt; + + int id; + struct k3_nav_ring *fd_ring; /* Free Descriptor ring */ + struct k3_nav_ring *r_ring; /* Receive ring*/ +}; + +struct udma_rflow { + void __iomem *reg_rflow; + + int id; +}; + +struct udma_match_data { + u8 tpl_levels; + u32 level_start_idx[]; +}; + +struct udma_dev { + struct dma_device ddev; + struct device *dev; + void __iomem *mmrs[MMR_LAST]; + const struct udma_match_data *match_data; + + struct udma_tisci_rm tisci_rm; + + struct k3_nav_ringacc *ringacc; + + struct irq_domain *irq_domain; + + struct work_struct purge_work; + struct list_head desc_to_purge; + spinlock_t lock; + + int tchan_cnt; + int echan_cnt; + int rchan_cnt; + int rflow_cnt; + unsigned long *tchan_map; + unsigned long *rchan_map; + unsigned long *rflow_map; + unsigned long *rflow_map_reserved; + + struct udma_tchan *tchans; + struct udma_rchan *rchans; + struct udma_rflow *rflows; + + struct udma_chan *channels; + u32 psil_base; +}; + +/* + * Slave RX scatter gather workaround: + * We need to use single continuous buffer if the original buffer is scattered + */ +struct udma_rx_sg_workaround { + bool in_use; + + struct scatterlist *sgl; + unsigned int sglen; + size_t total_len; + + struct scatterlist single_sg; +}; + +struct udma_hwdesc { + size_t cppi5_desc_size; + void *cppi5_desc_vaddr; + dma_addr_t cppi5_desc_paddr; + + /* TR descriptor internal pointers */ + void *tr_req_base; + struct cppi50_tr_resp *tr_resp_base; +}; + +struct udma_desc { + struct virt_dma_desc vd; + + bool terminated; + + enum dma_transfer_direction dir; + + struct udma_static_tr static_tr; + u32 residue; + + unsigned int sglen; + unsigned int desc_idx; /* Only used for cyclic in packet mode */ + unsigned int tr_idx; + + /* for slave_sg RX workaround */ + struct udma_rx_sg_workaround rx_sg_wa; + + u32 metadata_size; + void *metadata; /* pointer to provided metadata buffer (EPIP, PSdata) */ + + unsigned int hwdesc_count; + struct udma_hwdesc hwdesc[0]; +}; + +enum udma_chan_state { + UDMA_CHAN_IS_IDLE = 0, /* not active, no teardown is in progress */ + UDMA_CHAN_IS_ACTIVE, /* Normal operation */ + UDMA_CHAN_IS_ACTIVE_FLUSH, /* Flushing for delayed tx */ + UDMA_CHAN_IS_TERMINATING, /* channel is being terminated */ +}; + +struct udma_chan { + struct virt_dma_chan vc; + struct dma_slave_config cfg; + struct udma_dev *ud; + struct udma_desc *desc; + struct udma_desc *terminated_desc; + struct udma_static_tr static_tr; + char *name; + + struct udma_tchan *tchan; + struct udma_rchan *rchan; + struct udma_rflow *rflow; + + bool psil_paired; + u32 irq_ra_tisci; + u32 irq_ra_idx; + u32 irq_udma_idx; + + int irq_num_ring; + int irq_num_udma; + + bool cyclic; + bool paused; + + enum udma_chan_state state; + struct completion teardown_completed; + + u32 bcnt; /* number of bytes completed since the start of the channel */ + u32 in_ring_cnt; /* number of descriptors in flight */ + + bool pkt_mode; /* TR or packet */ + bool needs_epib; /* EPIB is needed for the communication or not */ + u32 psd_size; /* size of Protocol Specific Data */ + u32 metadata_size; /* (needs_epib ? 16:0) + psd_size */ + u32 hdesc_size; /* Size of a packet descriptor in packet mode */ + int desc_align; /* alignment to use for descriptors */ + int slave_thread_id; + u32 src_thread; + u32 dst_thread; + u32 static_tr_type; + enum udma_tp_level channel_tpl; /* Channel Throughput Level */ + + /* dmapool for packet mode descriptors */ + bool use_dma_pool; + struct dma_pool *hdesc_pool; + + u32 id; + enum dma_transfer_direction dir; +}; + +static inline struct udma_dev *to_udma_dev(struct dma_device *d) +{ + return container_of(d, struct udma_dev, ddev); +} + +static inline struct udma_chan *to_udma_chan(struct dma_chan *c) +{ + return container_of(c, struct udma_chan, vc.chan); +} + +static inline struct udma_desc *to_udma_desc(struct dma_async_tx_descriptor *t) +{ + return container_of(t, struct udma_desc, vd.tx); +} + +#define UDMA_CH_1000(ch) (ch * 0x1000) + +/* Generic register access functions */ +static inline u32 udma_read(void __iomem *base, int reg) +{ + return __raw_readl(base + reg); +} + +static inline void udma_write(void __iomem *base, int reg, u32 val) +{ + __raw_writel(val, base + reg); +} + +static inline void udma_update_bits(void __iomem *base, int reg, + u32 mask, u32 val) +{ + u32 tmp, orig; + + orig = __raw_readl(base + reg); + tmp = orig & ~mask; + tmp |= (val & mask); + + if (tmp != orig) + __raw_writel(tmp, base + reg); +} + +/* TCHANRT */ +static inline u32 udma_tchanrt_read(struct udma_tchan *tchan, int reg) +{ + if (!tchan) + return 0; + return udma_read(tchan->reg_rt, reg); +} + +static inline void udma_tchanrt_write(struct udma_tchan *tchan, int reg, + u32 val) +{ + if (!tchan) + return; + udma_write(tchan->reg_rt, reg, val); +} + +static inline void udma_tchanrt_update_bits(struct udma_tchan *tchan, int reg, + u32 mask, u32 val) +{ + if (!tchan) + return; + udma_update_bits(tchan->reg_rt, reg, mask, val); +} + +/* RCHANRT */ +static inline u32 udma_rchanrt_read(struct udma_rchan *rchan, int reg) +{ + if (!rchan) + return 0; + return udma_read(rchan->reg_rt, reg); +} + +static inline void udma_rchanrt_write(struct udma_rchan *rchan, int reg, + u32 val) +{ + if (!rchan) + return; + udma_write(rchan->reg_rt, reg, val); +} + +static inline void udma_rchanrt_update_bits(struct udma_rchan *rchan, int reg, + u32 mask, u32 val) +{ + if (!rchan) + return; + udma_update_bits(rchan->reg_rt, reg, mask, val); +} + +static inline int navss_psil_pair(struct udma_dev *ud, u32 src_thread, + u32 dst_thread) +{ + struct udma_tisci_rm *tisci_rm = &ud->tisci_rm; + + dst_thread |= UDMA_PSIL_DST_THREAD_ID_OFFSET; + return tisci_rm->tisci_psil_ops->pair(tisci_rm->tisci, + tisci_rm->tisci_navss_dev_id, + src_thread, dst_thread); +} + +static inline int navss_psil_unpair(struct udma_dev *ud, u32 src_thread, + u32 dst_thread) +{ + struct udma_tisci_rm *tisci_rm = &ud->tisci_rm; + + dst_thread |= UDMA_PSIL_DST_THREAD_ID_OFFSET; + return tisci_rm->tisci_psil_ops->unpair(tisci_rm->tisci, + tisci_rm->tisci_navss_dev_id, + src_thread, dst_thread); +} + +static inline char *udma_get_dir_text(enum dma_transfer_direction dir) +{ + switch (dir) { + case DMA_DEV_TO_MEM: + return "DEV_TO_MEM"; + case DMA_MEM_TO_DEV: + return "MEM_TO_DEV"; + case DMA_MEM_TO_MEM: + return "MEM_TO_MEM"; + case DMA_DEV_TO_DEV: + return "DEV_TO_DEV"; + default: + break; + } + + return "invalid"; +} + +static inline void udma_dump_chan_stdata(struct udma_chan *uc) +{ + struct device *dev = uc->ud->dev; + u32 offset; + int i; + + if (uc->dir == DMA_MEM_TO_DEV || uc->dir == DMA_MEM_TO_MEM) { + dev_dbg(dev, "TCHAN State data:\n"); + for (i = 0; i < 32; i++) { + offset = UDMA_TCHAN_RT_STDATA_REG + i * 4; + dev_dbg(dev, "TRT_STDATA[%02d]: 0x%08x\n", i, + udma_tchanrt_read(uc->tchan, offset)); + } + } + + if (uc->dir == DMA_DEV_TO_MEM || uc->dir == DMA_MEM_TO_MEM) { + dev_dbg(dev, "RCHAN State data:\n"); + for (i = 0; i < 32; i++) { + offset = UDMA_RCHAN_RT_STDATA_REG + i * 4; + dev_dbg(dev, "RRT_STDATA[%02d]: 0x%08x\n", i, + udma_rchanrt_read(uc->rchan, offset)); + } + } +} + +static inline dma_addr_t udma_curr_cppi5_desc_paddr(struct udma_desc *d, + int idx) +{ + return d->hwdesc[idx].cppi5_desc_paddr; +} + +static inline void *udma_curr_cppi5_desc_vaddr(struct udma_desc *d, int idx) +{ + return d->hwdesc[idx].cppi5_desc_vaddr; +} + +static inline struct udma_desc *udma_udma_desc_from_paddr(struct udma_chan *uc, + dma_addr_t paddr) +{ + struct udma_desc *d = uc->terminated_desc; + + if (d) { + dma_addr_t desc_paddr = udma_curr_cppi5_desc_paddr(d, + d->desc_idx); + + if (desc_paddr != paddr) + d = NULL; + } + + if (!d) { + d = uc->desc; + if (d) { + dma_addr_t desc_paddr = udma_curr_cppi5_desc_paddr(d, + d->desc_idx); + + if (desc_paddr != paddr) + d = NULL; + } + } + + return d; +} + +static inline size_t udma_calc_trdesc_size(size_t tr_size, int tr_count) +{ + size_t trdesc_size = sizeof(struct cppi50_tr_req_desc); + + trdesc_size += tr_size * (tr_count + 1); + trdesc_size += tr_count * sizeof(struct cppi50_tr_resp); + + return trdesc_size; +} + +static void udma_free_hwdesc(struct virt_dma_desc *vd) +{ + struct udma_chan *uc = to_udma_chan(vd->tx.chan); + struct udma_desc *d = to_udma_desc(&vd->tx); + + if (uc->use_dma_pool) { + int i; + + for (i = 0; i < d->hwdesc_count; i++) { + if (!d->hwdesc[i].cppi5_desc_vaddr) + continue; + + dma_pool_free(uc->hdesc_pool, + d->hwdesc[i].cppi5_desc_vaddr, + d->hwdesc[i].cppi5_desc_paddr); + + d->hwdesc[i].cppi5_desc_vaddr = NULL; + } + } else if (d->hwdesc[0].cppi5_desc_vaddr) { + struct udma_dev *ud = to_udma_dev(vd->tx.chan->device); + + dma_free_coherent(ud->dev, d->hwdesc[0].cppi5_desc_size, + d->hwdesc[0].cppi5_desc_vaddr, + d->hwdesc[0].cppi5_desc_paddr); + + d->hwdesc[0].cppi5_desc_vaddr = NULL; + } + + if (d->rx_sg_wa.in_use) { + dma_unmap_sg(uc->ud->dev, &d->rx_sg_wa.single_sg, 1, + DMA_FROM_DEVICE); + kfree(sg_virt(&d->rx_sg_wa.single_sg)); + } +} + +static void udma_purge_desc_work(struct work_struct *work) +{ + struct udma_dev *ud = container_of(work, typeof(*ud), purge_work); + struct virt_dma_desc *vd, *_vd; + unsigned long flags; + LIST_HEAD(head); + + spin_lock_irqsave(&ud->lock, flags); + list_splice_tail_init(&ud->desc_to_purge, &head); + spin_unlock_irqrestore(&ud->lock, flags); + + list_for_each_entry_safe(vd, _vd, &head, node) { + struct udma_desc *d; + + d = to_udma_desc(&vd->tx); + + udma_free_hwdesc(vd); + list_del(&vd->node); + kfree(d); + } + + /* If more to purge, schedule the work again */ + if (!list_empty(&ud->desc_to_purge)) + schedule_work(&ud->purge_work); +} + +static void udma_desc_free(struct virt_dma_desc *vd) +{ + struct udma_dev *ud = to_udma_dev(vd->tx.chan->device); + struct udma_chan *uc = to_udma_chan(vd->tx.chan); + struct udma_desc *d = to_udma_desc(&vd->tx); + unsigned long flags; + + if (uc->terminated_desc == d) + uc->terminated_desc = NULL; + + if (uc->use_dma_pool) { + udma_free_hwdesc(&d->vd); + kfree(d); + return; + } + + spin_lock_irqsave(&ud->lock, flags); + list_add_tail(&vd->node, &ud->desc_to_purge); + spin_unlock_irqrestore(&ud->lock, flags); + + schedule_work(&ud->purge_work); +} + +static inline bool udma_is_chan_running(struct udma_chan *uc) +{ + u32 trt_ctl = 0; + u32 rrt_ctl = 0; + + switch (uc->dir) { + case DMA_DEV_TO_MEM: + rrt_ctl = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_CTL_REG); + break; + case DMA_MEM_TO_DEV: + trt_ctl = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_CTL_REG); + break; + case DMA_MEM_TO_MEM: + trt_ctl = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_CTL_REG); + rrt_ctl = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_CTL_REG); + break; + default: + break; + } + + if (trt_ctl & UDMA_CHAN_RT_CTL_EN || rrt_ctl & UDMA_CHAN_RT_CTL_EN) + return true; + + return false; +} + +static void udma_sync_for_device(struct udma_chan *uc, int idx) +{ + struct udma_desc *d = uc->desc; + + if (uc->cyclic && uc->pkt_mode) { + dma_sync_single_for_device(uc->ud->dev, + d->hwdesc[idx].cppi5_desc_paddr, + d->hwdesc[idx].cppi5_desc_size, + DMA_TO_DEVICE); + } else { + int i; + + for (i = 0; i < d->hwdesc_count; i++) { + if (!d->hwdesc[i].cppi5_desc_vaddr) + continue; + + dma_sync_single_for_device(uc->ud->dev, + d->hwdesc[i].cppi5_desc_paddr, + d->hwdesc[i].cppi5_desc_size, + DMA_TO_DEVICE); + } + } +} + +static int udma_push_to_ring(struct udma_chan *uc, int idx) +{ + struct udma_desc *d = uc->desc; + + struct k3_nav_ring *ring = NULL; + int ret = -EINVAL; + + switch (uc->dir) { + case DMA_DEV_TO_MEM: + ring = uc->rchan->fd_ring; + break; + case DMA_MEM_TO_DEV: + ring = uc->tchan->t_ring; + break; + case DMA_MEM_TO_MEM: + ring = uc->tchan->t_ring; + break; + default: + break; + } + + if (ring) { + dma_addr_t desc_addr = udma_curr_cppi5_desc_paddr(d, idx); + + wmb(); /* Ensure that writes are not moved over this point */ + udma_sync_for_device(uc, idx); + ret = k3_nav_ringacc_ring_push(ring, &desc_addr); + uc->in_ring_cnt++; + } + + return ret; +} + +static int udma_pop_from_ring(struct udma_chan *uc, dma_addr_t *addr) +{ + struct k3_nav_ring *ring = NULL; + int ret = -ENOENT; + + switch (uc->dir) { + case DMA_DEV_TO_MEM: + ring = uc->rchan->r_ring; + break; + case DMA_MEM_TO_DEV: + ring = uc->tchan->tc_ring; + break; + case DMA_MEM_TO_MEM: + ring = uc->tchan->tc_ring; + break; + default: + break; + } + + if (ring && k3_nav_ringacc_ring_get_occ(ring)) { + struct udma_desc *d = NULL; + + ret = k3_nav_ringacc_ring_pop(ring, addr); + if (ret) + return ret; + + /* Teardown completion */ + if (*addr & 0x1) + return ret; + + d = udma_udma_desc_from_paddr(uc, *addr); + + if (d) + dma_sync_single_for_cpu(uc->ud->dev, *addr, + d->hwdesc[0].cppi5_desc_size, + DMA_FROM_DEVICE); + rmb(); /* Ensure that reads are not moved before this point */ + + if (!ret) + uc->in_ring_cnt--; + } + + return ret; +} + +static void udma_reset_rings(struct udma_chan *uc) +{ + struct k3_nav_ring *ring1 = NULL; + struct k3_nav_ring *ring2 = NULL; + + switch (uc->dir) { + case DMA_DEV_TO_MEM: + if (uc->rchan) { + ring1 = uc->rchan->fd_ring; + ring2 = uc->rchan->r_ring; + } + break; + case DMA_MEM_TO_DEV: + if (uc->tchan) { + ring1 = uc->tchan->t_ring; + ring2 = uc->tchan->tc_ring; + } + break; + case DMA_MEM_TO_MEM: + if (uc->tchan) { + ring1 = uc->tchan->t_ring; + ring2 = uc->tchan->tc_ring; + } + break; + default: + break; + } + + if (ring1) + k3_nav_ringacc_ring_reset_dma(ring1, 0); + if (ring2) + k3_nav_ringacc_ring_reset(ring2); + + /* make sure we are not leaking memory by stalled descriptor */ + if (uc->terminated_desc) { + udma_desc_free(&uc->terminated_desc->vd); + uc->terminated_desc = NULL; + } + + uc->in_ring_cnt = 0; +} + +static inline void udma_reset_counters(struct udma_chan *uc) +{ + u32 val; + + if (uc->tchan) { + val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_BCNT_REG); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_BCNT_REG, val); + + val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_SBCNT_REG); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_SBCNT_REG, val); + + val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_PCNT_REG); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PCNT_REG, val); + + val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_PEER_BCNT_REG); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_BCNT_REG, val); + } + + if (uc->rchan) { + val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_BCNT_REG); + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_BCNT_REG, val); + + val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_SBCNT_REG); + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_SBCNT_REG, val); + + val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_PCNT_REG); + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PCNT_REG, val); + + val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_PEER_BCNT_REG); + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_BCNT_REG, val); + } + + uc->bcnt = 0; +} + +static inline int udma_reset_chan(struct udma_chan *uc, bool hard) +{ + switch (uc->dir) { + case DMA_DEV_TO_MEM: + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_RT_EN_REG, 0); + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG, 0); + break; + case DMA_MEM_TO_DEV: + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, 0); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_RT_EN_REG, 0); + break; + case DMA_MEM_TO_MEM: + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG, 0); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, 0); + break; + default: + return -EINVAL; + } + + /* Reset all counters */ + udma_reset_counters(uc); + + /* Hard reset: re-initialize the channel to reset */ + if (hard) { + int ret; + + uc->ud->ddev.device_free_chan_resources(&uc->vc.chan); + ret = uc->ud->ddev.device_alloc_chan_resources(&uc->vc.chan); + if (ret) + return ret; + } + uc->state = UDMA_CHAN_IS_IDLE; + + return 0; +} + +static inline void udma_start_desc(struct udma_chan *uc) +{ + if (uc->cyclic && uc->pkt_mode) { + int i; + + /* Push all descriptors to ring for cyclic packet mode */ + for (i = 0; i < uc->desc->sglen; i++) + udma_push_to_ring(uc, i); + } else { + udma_push_to_ring(uc, 0); + } +} + +static inline bool udma_chan_needs_reconfiguration(struct udma_chan *uc) +{ + /* Only PDMAs have staticTR */ + if (!uc->static_tr_type) + return false; + + /* RX channels always need to be reset, reconfigured */ + if (uc->dir == DMA_DEV_TO_MEM) + return true; + + /* Check if the staticTR configuration has changed for TX */ + if (memcmp(&uc->static_tr, &uc->desc->static_tr, sizeof(uc->static_tr))) + return true; + + return false; +} + +static int udma_start(struct udma_chan *uc) +{ + struct virt_dma_desc *vd = vchan_next_desc(&uc->vc); + + if (!vd) { + uc->desc = NULL; + return -ENOENT; + } + + list_del(&vd->node); + + uc->desc = to_udma_desc(&vd->tx); + + /* Channel is already running and does not need reconfiguration */ + if (udma_is_chan_running(uc) && !udma_chan_needs_reconfiguration(uc)) { + udma_start_desc(uc); + goto out; + } + + /* Make sure that we clear the teardown bit, if it is set */ + udma_reset_chan(uc, false); + + /* Push descriptors before we start the channel */ + udma_start_desc(uc); + + switch (uc->desc->dir) { + case DMA_DEV_TO_MEM: + /* Config remote TR */ + if (uc->static_tr_type) { + udma_rchanrt_write(uc->rchan, + UDMA_RCHAN_RT_PEER_STATIC_TR_XY_REG, + PDMA_STATIC_TR_Y(uc->desc->static_tr.elcnt) | + PDMA_STATIC_TR_X(uc->desc->static_tr.elsize)); + udma_rchanrt_write(uc->rchan, + UDMA_RCHAN_RT_PEER_STATIC_TR_Z_REG, + PDMA_STATIC_TR_Z(uc->desc->static_tr.bstcnt)); + + /* save the current staticTR configuration */ + memcpy(&uc->static_tr, &uc->desc->static_tr, + sizeof(uc->static_tr)); + } + + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + + /* Enable remote */ + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE); + + break; + case DMA_MEM_TO_DEV: + /* Config remote TR */ + if (uc->static_tr_type) { + udma_tchanrt_write(uc->tchan, + UDMA_TCHAN_RT_PEER_STATIC_TR_XY_REG, + PDMA_STATIC_TR_Y(uc->desc->static_tr.elcnt) | + PDMA_STATIC_TR_X(uc->desc->static_tr.elsize)); + + /* save the current staticTR configuration */ + memcpy(&uc->static_tr, &uc->desc->static_tr, + sizeof(uc->static_tr)); + } + + /* Enable remote */ + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE); + + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + + break; + case DMA_MEM_TO_MEM: + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN); + + break; + default: + return -EINVAL; + } + + uc->state = UDMA_CHAN_IS_ACTIVE; +out: + + return 0; +} + +static inline int udma_stop(struct udma_chan *uc) +{ + enum udma_chan_state old_state = uc->state; + + uc->state = UDMA_CHAN_IS_TERMINATING; + reinit_completion(&uc->teardown_completed); + + switch (uc->dir) { + case DMA_DEV_TO_MEM: + udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE | + UDMA_PEER_RT_EN_TEARDOWN); + break; + case DMA_MEM_TO_DEV: + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE | + UDMA_PEER_RT_EN_FLUSH); + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | + UDMA_CHAN_RT_CTL_TDOWN); + break; + case DMA_MEM_TO_MEM: + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | + UDMA_CHAN_RT_CTL_TDOWN); + break; + default: + uc->state = old_state; + complete_all(&uc->teardown_completed); + return -EINVAL; + } + + return 0; +} + +static void udma_cyclic_packet_elapsed(struct udma_chan *uc) +{ + struct udma_desc *d = uc->desc; + struct knav_udmap_host_desc_t *h_desc; + + h_desc = d->hwdesc[d->desc_idx].cppi5_desc_vaddr; + knav_udmap_hdesc_reset_to_original(h_desc); + udma_push_to_ring(uc, d->desc_idx); + d->desc_idx = (d->desc_idx + 1) % d->sglen; +} + +static inline void udma_fetch_epib(struct udma_chan *uc, struct udma_desc *d) +{ + struct knav_udmap_host_desc_t *h_desc = d->hwdesc[0].cppi5_desc_vaddr; + + memcpy(d->metadata, h_desc->epib, d->metadata_size); +} + +static inline bool udma_is_desc_really_done(struct udma_chan *uc, + struct udma_desc *d) +{ + u32 peer_bcnt, bcnt; + + /* Only TX towards PDMA is affected */ + if (!uc->static_tr_type || uc->dir != DMA_MEM_TO_DEV) + return true; + + peer_bcnt = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_PEER_BCNT_REG); + bcnt = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_BCNT_REG); + + if (peer_bcnt < bcnt) + return false; + + return true; +} + +static void udma_flush_tx(struct udma_chan *uc) +{ + if (uc->dir != DMA_MEM_TO_DEV) + return; + + uc->state = UDMA_CHAN_IS_ACTIVE_FLUSH; + + udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | + UDMA_CHAN_RT_CTL_TDOWN); +} + +static void udma_ring_callback(struct udma_chan *uc, dma_addr_t paddr) +{ + struct udma_desc *d; + unsigned long flags; + + if (!paddr) + return; + + spin_lock_irqsave(&uc->vc.lock, flags); + + /* Teardown completion message */ + if (paddr & 0x1) { + /* Compensate our internal pop/push counter */ + uc->in_ring_cnt++; + + complete_all(&uc->teardown_completed); + + if (uc->terminated_desc) { + udma_desc_free(&uc->terminated_desc->vd); + uc->terminated_desc = NULL; + } + + if (!uc->desc) + udma_start(uc); + + if (uc->state != UDMA_CHAN_IS_ACTIVE_FLUSH) + goto out; + else if (uc->desc) + paddr = udma_curr_cppi5_desc_paddr(uc->desc, + uc->desc->desc_idx); + } + + d = udma_udma_desc_from_paddr(uc, paddr); + + if (d) { + dma_addr_t desc_paddr = udma_curr_cppi5_desc_paddr(d, + d->desc_idx); + if (desc_paddr != paddr) { + dev_err(uc->ud->dev, "not matching descriptors!\n"); + goto out; + } + + if (uc->cyclic) { + /* push the descriptor back to the ring */ + if (d == uc->desc) { + udma_cyclic_packet_elapsed(uc); + vchan_cyclic_callback(&d->vd); + } + } else { + bool desc_done = true; + + if (d == uc->desc) { + desc_done = udma_is_desc_really_done(uc, d); + + if (desc_done) { + uc->bcnt += d->residue; + udma_start(uc); + } else { + udma_flush_tx(uc); + } + } else if (d == uc->terminated_desc) { + uc->terminated_desc = NULL; + } + + if (desc_done) + vchan_cookie_complete(&d->vd); + } + } +out: + spin_unlock_irqrestore(&uc->vc.lock, flags); +} + +static void udma_tr_event_callback(struct udma_chan *uc) +{ + struct udma_desc *d; + unsigned long flags; + + spin_lock_irqsave(&uc->vc.lock, flags); + d = uc->desc; + if (d) { + d->tr_idx = (d->tr_idx + 1) % d->sglen; + + if (uc->cyclic) { + vchan_cyclic_callback(&d->vd); + } else { + /* TODO: figure out the real amount of data */ + uc->bcnt += d->residue; + udma_start(uc); + vchan_cookie_complete(&d->vd); + } + } + + spin_unlock_irqrestore(&uc->vc.lock, flags); +} + +static irqreturn_t udma_ring_irq_handler(int irq, void *data) +{ + struct udma_chan *uc = data; + dma_addr_t paddr = 0; + + if (!udma_pop_from_ring(uc, &paddr)) + udma_ring_callback(uc, paddr); + + return IRQ_HANDLED; +} + +static irqreturn_t udma_udma_irq_handler(int irq, void *data) +{ + struct udma_chan *uc = data; + struct udma_tisci_rm *tisci_rm = &uc->ud->tisci_rm; + + ti_sci_inta_ack_event(uc->ud->irq_domain, tisci_rm->tisci_dev_id, + uc->irq_udma_idx, uc->irq_num_udma); + + udma_tr_event_callback(uc); + + return IRQ_HANDLED; +} + +/** + * __udma_reserve_rflow_range - reserve range of flow ids + * @ud: UDMA device + * @from: Start the search from this flow id number + * @cnt: Number of consecutive flow ids to allocate + * + * Reserve range of flow ids for future use, those flows can be allocated + * only using explicit flow id number. if @from is set to -1 it will try to find + * first free range. if @from is positive value it will force allocation only + * of the specified range of flows. + * + * Returns -ENOMEM if can't find free range. + * -EEXIST if requested range is busy. + * -EINVAL if wrong input values passed. + * Returns flow id on success. + */ +static int __udma_reserve_rflow_range(struct udma_dev *ud, int from, int cnt) +{ + int start, tmp_from; + DECLARE_BITMAP(tmp, K3_UDMA_MAX_RFLOWS); + + tmp_from = from; + if (tmp_from < 0) + tmp_from = ud->rchan_cnt; + /* default flows can't be reserved and accessible only by id */ + if (tmp_from < ud->rchan_cnt) + return -EINVAL; + + if (tmp_from + cnt > ud->rflow_cnt) + return -EINVAL; + + bitmap_or(tmp, ud->rflow_map, ud->rflow_map_reserved, + ud->rflow_cnt); + + start = bitmap_find_next_zero_area(tmp, + ud->rflow_cnt, + tmp_from, cnt, 0); + if (start >= ud->rflow_cnt) + return -ENOMEM; + + if (from >= 0 && start != from) + return -EEXIST; + + bitmap_set(ud->rflow_map_reserved, start, cnt); + return start; +} + +static int __udma_free_rflow_range(struct udma_dev *ud, int from, int cnt) +{ + if (from < ud->rchan_cnt) + return -EINVAL; + if (from + cnt > ud->rflow_cnt) + return -EINVAL; + + bitmap_clear(ud->rflow_map_reserved, from, cnt); + return 0; +} + +static struct udma_rflow *__udma_reserve_rflow(struct udma_dev *ud, + enum udma_tp_level tpl, int id) +{ + DECLARE_BITMAP(tmp, K3_UDMA_MAX_RFLOWS); + + if (id >= 0) { + if (test_bit(id, ud->rflow_map)) { + dev_err(ud->dev, "rflow%d is in use\n", id); + return ERR_PTR(-ENOENT); + } + } else { + bitmap_or(tmp, ud->rflow_map, ud->rflow_map_reserved, + ud->rflow_cnt); + + id = find_next_zero_bit(tmp, ud->rflow_cnt, ud->rchan_cnt); + if (id >= ud->rflow_cnt) + return ERR_PTR(-ENOENT); + } + + set_bit(id, ud->rflow_map); + return &ud->rflows[id]; +} + +#define UDMA_RESERVE_RESOURCE(res) \ +static struct udma_##res *__udma_reserve_##res(struct udma_dev *ud, \ + enum udma_tp_level tpl, \ + int id) \ +{ \ + if (id >= 0) { \ + if (test_bit(id, ud->res##_map)) { \ + dev_err(ud->dev, "res##%d is in use\n", id); \ + return ERR_PTR(-ENOENT); \ + } \ + } else { \ + int start; \ + \ + if (tpl >= ud->match_data->tpl_levels) \ + tpl = ud->match_data->tpl_levels - 1; \ + \ + start = ud->match_data->level_start_idx[tpl]; \ + \ + id = find_next_zero_bit(ud->res##_map, ud->res##_cnt, \ + start); \ + if (id == ud->res##_cnt) { \ + return ERR_PTR(-ENOENT); \ + } \ + } \ + \ + set_bit(id, ud->res##_map); \ + return &ud->res##s[id]; \ +} + +UDMA_RESERVE_RESOURCE(tchan); +UDMA_RESERVE_RESOURCE(rchan); + +static int udma_get_tchan(struct udma_chan *uc) +{ + struct udma_dev *ud = uc->ud; + + if (uc->tchan) { + dev_dbg(ud->dev, "chan%d: already have tchan%d allocated\n", + uc->id, uc->tchan->id); + return 0; + } + + uc->tchan = __udma_reserve_tchan(ud, uc->channel_tpl, -1); + if (IS_ERR(uc->tchan)) + return PTR_ERR(uc->tchan); + + if (udma_is_chan_running(uc)) { + dev_warn(ud->dev, "chan%d: tchan%d is running!\n", uc->id, + uc->tchan->id); + udma_stop(uc); + if (udma_is_chan_running(uc)) + dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); + } + + return 0; +} + +static int udma_get_rchan(struct udma_chan *uc) +{ + struct udma_dev *ud = uc->ud; + + if (uc->rchan) { + dev_dbg(ud->dev, "chan%d: already have rchan%d allocated\n", + uc->id, uc->rchan->id); + return 0; + } + + uc->rchan = __udma_reserve_rchan(ud, uc->channel_tpl, -1); + if (IS_ERR(uc->rchan)) + return PTR_ERR(uc->rchan); + + if (udma_is_chan_running(uc)) { + dev_warn(ud->dev, "chan%d: rchan%d is running!\n", uc->id, + uc->rchan->id); + udma_stop(uc); + if (udma_is_chan_running(uc)) + dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); + } + + return 0; +} + +static int udma_get_chan_pair(struct udma_chan *uc) +{ + struct udma_dev *ud = uc->ud; + int chan_id, end; + + if ((uc->tchan && uc->rchan) && uc->tchan->id == uc->rchan->id) { + dev_info(ud->dev, "chan%d: already have %d pair allocated\n", + uc->id, uc->tchan->id); + return 0; + } + + if (uc->tchan) { + dev_err(ud->dev, "chan%d: already have tchan%d allocated\n", + uc->id, uc->tchan->id); + return -EBUSY; + } else if (uc->rchan) { + dev_err(ud->dev, "chan%d: already have rchan%d allocated\n", + uc->id, uc->rchan->id); + return -EBUSY; + } + + /* Can be optimized, but let's have it like this for now */ + end = min(ud->tchan_cnt, ud->rchan_cnt); + for (chan_id = ud->match_data->level_start_idx[UDMA_TP_NORMAL]; + chan_id < end; chan_id++) { + if (!test_bit(chan_id, ud->tchan_map) && + !test_bit(chan_id, ud->rchan_map)) + break; + } + + if (chan_id == end) + return -ENOENT; + + set_bit(chan_id, ud->tchan_map); + set_bit(chan_id, ud->rchan_map); + uc->tchan = &ud->tchans[chan_id]; + uc->rchan = &ud->rchans[chan_id]; + + if (udma_is_chan_running(uc)) { + dev_warn(ud->dev, "chan%d: t/rchan%d pair is running!\n", + uc->id, chan_id); + udma_stop(uc); + if (udma_is_chan_running(uc)) + dev_err(ud->dev, "chan%d: won't stop!\n", uc->id); + } + + return 0; +} + +static int udma_get_rflow(struct udma_chan *uc, int flow_id) +{ + struct udma_dev *ud = uc->ud; + + if (uc->rflow) { + dev_dbg(ud->dev, "chan%d: already have rflow%d allocated\n", + uc->id, uc->rflow->id); + return 0; + } + + if (!uc->rchan) + dev_warn(ud->dev, "chan%d: does not have rchan??\n", uc->id); + + uc->rflow = __udma_reserve_rflow(ud, uc->channel_tpl, flow_id); + if (IS_ERR(uc->rflow)) + return PTR_ERR(uc->rflow); + + return 0; +} + +static void udma_put_rchan(struct udma_chan *uc) +{ + struct udma_dev *ud = uc->ud; + + if (uc->rchan) { + dev_dbg(ud->dev, "chan%d: put rchan%d\n", uc->id, + uc->rchan->id); + clear_bit(uc->rchan->id, ud->rchan_map); + uc->rchan = NULL; + } +} + +static void udma_put_tchan(struct udma_chan *uc) +{ + struct udma_dev *ud = uc->ud; + + if (uc->tchan) { + dev_dbg(ud->dev, "chan%d: put tchan%d\n", uc->id, + uc->tchan->id); + clear_bit(uc->tchan->id, ud->tchan_map); + uc->tchan = NULL; + } +} + +static void udma_put_rflow(struct udma_chan *uc) +{ + struct udma_dev *ud = uc->ud; + + if (uc->rflow) { + dev_dbg(ud->dev, "chan%d: put rflow%d\n", uc->id, + uc->rflow->id); + clear_bit(uc->rflow->id, ud->rflow_map); + uc->rflow = NULL; + } +} + +static void udma_free_tx_resources(struct udma_chan *uc) +{ + if (!uc->tchan) + return; + + k3_nav_ringacc_ring_free(uc->tchan->t_ring); + k3_nav_ringacc_ring_free(uc->tchan->tc_ring); + uc->tchan->t_ring = NULL; + uc->tchan->tc_ring = NULL; + + udma_put_tchan(uc); +} + +static int udma_alloc_tx_resources(struct udma_chan *uc) +{ + struct k3_nav_ring_cfg ring_cfg; + struct udma_dev *ud = uc->ud; + int ret; + + ret = udma_get_tchan(uc); + if (ret) + return ret; + + uc->tchan->t_ring = k3_nav_ringacc_request_ring(ud->ringacc, + uc->tchan->id, 0); + if (!uc->tchan->t_ring) { + ret = -EBUSY; + goto err_tx_ring; + } + + uc->tchan->tc_ring = k3_nav_ringacc_request_ring(ud->ringacc, -1, 0); + if (!uc->tchan->tc_ring) { + ret = -EBUSY; + goto err_txc_ring; + } + + memset(&ring_cfg, 0, sizeof(ring_cfg)); + ring_cfg.size = 16; + ring_cfg.elm_size = K3_NAV_RINGACC_RING_ELSIZE_8; + ring_cfg.mode = K3_NAV_RINGACC_RING_MODE_MESSAGE; + + ret = k3_nav_ringacc_ring_cfg(uc->tchan->t_ring, &ring_cfg); + ret |= k3_nav_ringacc_ring_cfg(uc->tchan->tc_ring, &ring_cfg); + + if (ret) + goto err_ringcfg; + + return 0; + +err_ringcfg: + k3_nav_ringacc_ring_free(uc->tchan->tc_ring); + uc->tchan->tc_ring = NULL; +err_txc_ring: + k3_nav_ringacc_ring_free(uc->tchan->t_ring); + uc->tchan->t_ring = NULL; +err_tx_ring: + udma_put_tchan(uc); + + return ret; +} + +static void udma_free_rx_resources(struct udma_chan *uc) +{ + if (!uc->rchan) + return; + + if (uc->dir != DMA_MEM_TO_MEM) { + k3_nav_ringacc_ring_free(uc->rchan->fd_ring); + k3_nav_ringacc_ring_free(uc->rchan->r_ring); + uc->rchan->fd_ring = NULL; + uc->rchan->r_ring = NULL; + + udma_put_rflow(uc); + } + + udma_put_rchan(uc); +} + +static int udma_alloc_rx_resources(struct udma_chan *uc) +{ + struct k3_nav_ring_cfg ring_cfg; + struct udma_dev *ud = uc->ud; + int fd_ring_id; + int ret; + + ret = udma_get_rchan(uc); + if (ret) + return ret; + + /* For MEM_TO_MEM we don't need rflow or rings */ + if (uc->dir == DMA_MEM_TO_MEM) + return 0; + + ret = udma_get_rflow(uc, uc->rchan->id); + if (ret) { + ret = -EBUSY; + goto err_rflow; + } + + fd_ring_id = ud->tchan_cnt + ud->echan_cnt + uc->rchan->id; + uc->rchan->fd_ring = k3_nav_ringacc_request_ring(ud->ringacc, + fd_ring_id, 0); + if (!uc->rchan->fd_ring) { + ret = -EBUSY; + goto err_rx_ring; + } + + uc->rchan->r_ring = k3_nav_ringacc_request_ring(ud->ringacc, -1, 0); + if (!uc->rchan->r_ring) { + ret = -EBUSY; + goto err_rxc_ring; + } + + memset(&ring_cfg, 0, sizeof(ring_cfg)); + ring_cfg.size = 16; + ring_cfg.elm_size = K3_NAV_RINGACC_RING_ELSIZE_8; + ring_cfg.mode = K3_NAV_RINGACC_RING_MODE_MESSAGE; + + ret = k3_nav_ringacc_ring_cfg(uc->rchan->fd_ring, &ring_cfg); + ret |= k3_nav_ringacc_ring_cfg(uc->rchan->r_ring, &ring_cfg); + + if (ret) + goto err_ringcfg; + + return 0; + +err_ringcfg: + k3_nav_ringacc_ring_free(uc->rchan->r_ring); + uc->rchan->r_ring = NULL; +err_rxc_ring: + k3_nav_ringacc_ring_free(uc->rchan->fd_ring); + uc->rchan->fd_ring = NULL; +err_rx_ring: + udma_put_rflow(uc); +err_rflow: + udma_put_rchan(uc); + + return ret; +} + +static int udma_alloc_chan_resources(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + struct udma_dev *ud = to_udma_dev(chan->device); + struct udma_tisci_rm *tisci_rm = &ud->tisci_rm; + const struct ti_sci_rm_udmap_ops *tisci_ops = tisci_rm->tisci_udmap_ops; + struct udma_tchan *tchan; + struct udma_rchan *rchan; + int ret; + + if (uc->pkt_mode || uc->dir == DMA_MEM_TO_MEM) { + uc->use_dma_pool = true; + /* in case of MEM_TO_MEM we have maximum of two TRs */ + if (uc->dir == DMA_MEM_TO_MEM) + uc->hdesc_size = udma_calc_trdesc_size( + sizeof(struct cppi50_tr_req_type15), 2); + } + + if (uc->use_dma_pool) { + uc->hdesc_pool = dma_pool_create(uc->name, ud->ddev.dev, + uc->hdesc_size, uc->desc_align, + 0); + if (!uc->hdesc_pool) { + dev_err(ud->ddev.dev, + "Descriptor pool allocation failed\n"); + uc->use_dma_pool = false; + return -ENOMEM; + } + } + + pm_runtime_get_sync(ud->ddev.dev); + + /* + * Make sure that the completion is in a known state: + * No teardown, the channel is idle + */ + reinit_completion(&uc->teardown_completed); + complete_all(&uc->teardown_completed); + uc->state = UDMA_CHAN_IS_IDLE; + + switch (uc->dir) { + case DMA_MEM_TO_MEM: + /* Non synchronized - mem to mem type of transfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-MEM\n", __func__, + uc->id); + + ret = udma_get_chan_pair(uc); + if (ret) + return ret; + + ret = udma_alloc_tx_resources(uc); + if (ret) + return ret; + + ret = udma_alloc_rx_resources(uc); + if (ret) { + udma_free_tx_resources(uc); + return ret; + } + + uc->src_thread = ud->psil_base + uc->tchan->id; + uc->dst_thread = (ud->psil_base + uc->rchan->id) | 0x8000; + + break; + case DMA_MEM_TO_DEV: + /* Slave transfer synchronized - mem to dev (TX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-DEV\n", __func__, + uc->id); + + ret = udma_alloc_tx_resources(uc); + if (ret) { + uc->slave_thread_id = -1; + return ret; + } + + uc->src_thread = ud->psil_base + uc->tchan->id; + uc->dst_thread = uc->slave_thread_id; + if (!(uc->dst_thread & 0x8000)) + uc->dst_thread |= 0x8000; + + break; + case DMA_DEV_TO_MEM: + /* Slave transfer synchronized - dev to mem (RX) trasnfer */ + dev_dbg(uc->ud->dev, "%s: chan%d as DEV-to-MEM\n", __func__, + uc->id); + + ret = udma_alloc_rx_resources(uc); + if (ret) { + uc->slave_thread_id = -1; + return ret; + } + + uc->src_thread = uc->slave_thread_id; + uc->dst_thread = (ud->psil_base + uc->rchan->id) | 0x8000; + + break; + default: + /* Can not happen */ + dev_err(uc->ud->dev, "%s: chan%d invalid direction (%u)\n", + __func__, uc->id, uc->dir); + return -EINVAL; + } + + tchan = uc->tchan; + rchan = uc->rchan; + + /* + * Configure Tx and Rx channel type to: + * Third Party DMA control transfers using pass by reference rings + */ + if (uc->dir == DMA_MEM_TO_MEM) { + /* Non synchronized - mem to mem type of transfer */ + int tc_ring = k3_nav_ringacc_get_ring_id(tchan->tc_ring); + struct ti_sci_msg_rm_udmap_tx_ch_cfg req_tx = { 0 }; + struct ti_sci_msg_rm_udmap_rx_ch_cfg req_rx = { 0 }; + + req_tx.valid_params = + TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_EINFO_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_PSWORDS_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_SUPR_TDPKT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID; + + req_tx.nav_id = tisci_rm->tisci_dev_id; + req_tx.index = tchan->id; + req_tx.tx_pause_on_err = 0; + req_tx.tx_filt_einfo = 0; + req_tx.tx_filt_pswords = 0; + req_tx.tx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_3RDP_BCOPY_PBRR; + req_tx.tx_supr_tdpkt = 0; + req_tx.tx_fetch_size = sizeof(struct cppi50_tr_req_desc) >> 2; + req_tx.txcq_qnum = tc_ring; + + ret = tisci_ops->tx_ch_cfg(tisci_rm->tisci, &req_tx); + if (ret) { + dev_err(ud->dev, "tchan%d cfg failed %d\n", + tchan->id, ret); + goto err_res_free; + } + + req_rx.valid_params = + TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_SHORT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_LONG_VALID; + + req_rx.nav_id = tisci_rm->tisci_dev_id; + req_rx.index = rchan->id; + req_rx.rx_fetch_size = sizeof(struct cppi50_tr_req_desc) >> 2; + req_rx.rxcq_qnum = tc_ring; + req_rx.rx_pause_on_err = 0; + req_rx.rx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_3RDP_BCOPY_PBRR; + req_rx.rx_ignore_short = 0; + req_rx.rx_ignore_long = 0; + + ret = tisci_ops->rx_ch_cfg(tisci_rm->tisci, &req_rx); + if (ret) { + dev_err(ud->dev, "rchan%d alloc failed %d\n", + rchan->id, ret); + goto err_res_free; + } + + uc->irq_ra_tisci = k3_nav_ringacc_get_tisci_dev_id( + tchan->tc_ring); + uc->irq_ra_idx = tc_ring; + uc->irq_udma_idx = tchan->id; + } else { + /* Slave transfer */ + u32 mode, fetch_size; + + if (uc->pkt_mode) { + mode = TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR; + fetch_size = knav_udmap_hdesc_calc_size(uc->needs_epib, + uc->psd_size, + 0); + } else { + mode = TI_SCI_RM_UDMAP_CHAN_TYPE_3RDP_PBRR; + fetch_size = sizeof(struct cppi50_tr_req_desc); + } + + if (uc->dir == DMA_MEM_TO_DEV) { + /* TX */ + int tc_ring = k3_nav_ringacc_get_ring_id( + tchan->tc_ring); + struct ti_sci_msg_rm_udmap_tx_ch_cfg req_tx = { 0 }; + + req_tx.valid_params = + TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_EINFO_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_PSWORDS_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_SUPR_TDPKT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID; + + req_tx.nav_id = tisci_rm->tisci_dev_id; + req_tx.index = tchan->id; + req_tx.tx_pause_on_err = 0; + req_tx.tx_filt_einfo = 0; + req_tx.tx_filt_pswords = 0; + req_tx.tx_chan_type = mode; + req_tx.tx_supr_tdpkt = 0; + req_tx.tx_fetch_size = fetch_size >> 2; + req_tx.txcq_qnum = tc_ring; + + ret = tisci_ops->tx_ch_cfg(tisci_rm->tisci, &req_tx); + if (ret) { + dev_err(ud->dev, "tchan%d cfg failed %d\n", + tchan->id, ret); + goto err_res_free; + } + + uc->irq_ra_tisci = k3_nav_ringacc_get_tisci_dev_id( + tchan->tc_ring); + uc->irq_ra_idx = tc_ring; + uc->irq_udma_idx = tchan->id; + } else { + /* RX */ + int fd_ring = k3_nav_ringacc_get_ring_id( + rchan->fd_ring); + int rx_ring = k3_nav_ringacc_get_ring_id(rchan->r_ring); + struct ti_sci_msg_rm_udmap_rx_ch_cfg req_rx = { 0 }; + struct ti_sci_msg_rm_udmap_flow_cfg flow_req = { 0 }; + + req_rx.valid_params = + TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_SHORT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_LONG_VALID; + + req_rx.nav_id = tisci_rm->tisci_dev_id; + req_rx.index = rchan->id; + req_rx.rx_fetch_size = fetch_size >> 2; + req_rx.rxcq_qnum = rx_ring; + req_rx.rx_pause_on_err = 0; + req_rx.rx_chan_type = mode; + req_rx.rx_ignore_short = 0; + req_rx.rx_ignore_long = 0; + + ret = tisci_ops->rx_ch_cfg(tisci_rm->tisci, &req_rx); + if (ret) { + dev_err(ud->dev, "rchan%d cfg failed %d\n", + rchan->id, ret); + goto err_res_free; + } + + flow_req.valid_params = + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_EINFO_PRESENT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_PSINFO_PRESENT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_ERROR_HANDLING_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DESC_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_HI_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_LO_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_HI_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_LO_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ0_SZ0_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ1_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ2_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ3_QNUM_VALID; + + flow_req.nav_id = tisci_rm->tisci_dev_id; + flow_req.flow_index = rchan->id; + + if (uc->needs_epib) + flow_req.rx_einfo_present = 1; + else + flow_req.rx_einfo_present = 0; + if (uc->psd_size) + flow_req.rx_psinfo_present = 1; + else + flow_req.rx_psinfo_present = 0; + flow_req.rx_error_handling = 1; + flow_req.rx_desc_type = 0; + flow_req.rx_dest_qnum = rx_ring; + flow_req.rx_src_tag_hi_sel = 2; + flow_req.rx_src_tag_lo_sel = 4; + flow_req.rx_dest_tag_hi_sel = 5; + flow_req.rx_dest_tag_lo_sel = 4; + flow_req.rx_fdq0_sz0_qnum = fd_ring; + flow_req.rx_fdq1_qnum = fd_ring; + flow_req.rx_fdq2_qnum = fd_ring; + flow_req.rx_fdq3_qnum = fd_ring; + + ret = tisci_ops->rx_flow_cfg(tisci_rm->tisci, + &flow_req); + + if (ret) { + dev_err(ud->dev, "flow%d config failed: %d\n", + rchan->id, ret); + goto err_chan_free; + } + + uc->irq_ra_tisci = k3_nav_ringacc_get_tisci_dev_id( + rchan->r_ring); + uc->irq_ra_idx = rx_ring; + uc->irq_udma_idx = 0x2000 + rchan->id; + } + } + + /* PSI-L pairing */ + ret = navss_psil_pair(ud, uc->src_thread, uc->dst_thread); + if (ret) + goto err_chan_free; + + uc->psil_paired = true; + + /* Get the interrupts... */ + uc->irq_num_ring = ti_sci_inta_register_event(ud->dev, uc->irq_ra_tisci, + uc->irq_ra_idx, 0, + IRQF_TRIGGER_HIGH); + if (uc->irq_num_ring <= 0) { + dev_err(ud->dev, "Failed to get ring irq (index: %u) %d\n", + uc->irq_ra_idx, uc->irq_num_ring); + ret = -EINVAL; + goto err_psi_free; + } + + uc->irq_num_udma = ti_sci_inta_register_event(ud->dev, + tisci_rm->tisci_dev_id, + uc->irq_udma_idx, 0, + IRQF_TRIGGER_HIGH); + if (uc->irq_num_udma <= 0) { + dev_err(ud->dev, "Failed to get udma irq (index: %u) %d\n", + uc->irq_udma_idx, uc->irq_num_udma); + + ti_sci_inta_unregister_event(ud->dev, uc->irq_ra_tisci, + uc->irq_ra_idx, uc->irq_num_ring); + + ret = -EINVAL; + goto err_psi_free; + } + + ret = request_irq(uc->irq_num_ring, udma_ring_irq_handler, 0, uc->name, + uc); + if (ret) { + dev_err(ud->dev, "%s: chan%d: Failed to request ring irq\n", + __func__, uc->id); + goto err_irq_free; + } + + ret = request_irq(uc->irq_num_udma, udma_udma_irq_handler, 0, uc->name, + uc); + if (ret) { + dev_err(ud->dev, "%s: chan%d: Failed to request UDMA irq\n", + __func__, uc->id); + free_irq(uc->irq_num_ring, uc); + goto err_irq_free; + } + + udma_reset_rings(uc); + + return 0; + +err_irq_free: + ti_sci_inta_unregister_event(ud->dev, uc->irq_ra_tisci, uc->irq_ra_idx, + uc->irq_num_ring); + uc->irq_num_ring = 0; + + ti_sci_inta_unregister_event(ud->dev, tisci_rm->tisci_dev_id, + uc->irq_udma_idx, uc->irq_num_udma); + uc->irq_num_udma = 0; +err_psi_free: + navss_psil_unpair(ud, uc->src_thread, uc->dst_thread); + uc->psil_paired = false; +err_chan_free: +err_res_free: + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); + uc->slave_thread_id = -1; + + if (uc->use_dma_pool) { + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool = false; + } + + return ret; +} + +static int udma_slave_config(struct dma_chan *chan, + struct dma_slave_config *cfg) +{ + struct udma_chan *uc = to_udma_chan(chan); + + memcpy(&uc->cfg, cfg, sizeof(uc->cfg)); + + return 0; +} + +static struct udma_desc *udma_alloc_tr_desc(struct udma_chan *uc, + size_t tr_size, int tr_count, + enum dma_transfer_direction dir) +{ + struct udma_hwdesc *hwdesc; + struct cppi50_tr_req_desc *tr_desc; + struct udma_desc *d; + u32 tr_nominal_size; + + switch (tr_size) { + case 16: + tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_16; + break; + case 32: + tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_32; + break; + case 64: + tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_64; + break; + case 128: + tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_128; + break; + default: + dev_err(uc->ud->dev, "Unsupported TR size of %zu\n", tr_size); + return NULL; + } + + /* We have only one descriptor containing multiple TRs */ + d = kzalloc(sizeof(*d) + sizeof(d->hwdesc[0]), GFP_ATOMIC); + if (!d) + return NULL; + + d->sglen = tr_count; + + d->hwdesc_count = 1; + hwdesc = &d->hwdesc[0]; + + /* Allocate memory for DMA ring descriptor */ + if (uc->use_dma_pool) { + hwdesc->cppi5_desc_size = uc->hdesc_size; + hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool, + GFP_ATOMIC, + &hwdesc->cppi5_desc_paddr); + } else { + hwdesc->cppi5_desc_size = udma_calc_trdesc_size(tr_size, + tr_count); + hwdesc->cppi5_desc_vaddr = dma_zalloc_coherent(uc->ud->dev, + hwdesc->cppi5_desc_size, + &hwdesc->cppi5_desc_paddr, + GFP_ATOMIC); + } + + if (!hwdesc->cppi5_desc_vaddr) { + kfree(d); + return NULL; + } + + /* Start of the TR req records */ + hwdesc->tr_req_base = hwdesc->cppi5_desc_vaddr + tr_size; + /* Start address of the TR response array */ + hwdesc->tr_resp_base = hwdesc->tr_req_base + tr_size * tr_count; + + tr_desc = hwdesc->cppi5_desc_vaddr; + tr_desc->packet_info[0] = CPPI50_TRDESC_W0_LAST_ENTRY(tr_count - 1) | + CPPI50_TRDESC_W0_TYPE; + if (uc->cyclic) + tr_desc->packet_info[0] |= CPPI50_TRDESC_W0_RELOAD_CNT(0x1ff); + + /* Flow and Packed ID */ + tr_desc->packet_info[1] = tr_nominal_size | + CPPI50_TRDESC_W1_PACKETID(uc->id) | + CPPI50_TRDESC_W1_FLOWID(0x3fff); + + if (dir == DMA_DEV_TO_MEM) + tr_desc->packet_info[2] = k3_nav_ringacc_get_ring_id( + uc->rchan->r_ring); + else + tr_desc->packet_info[2] = k3_nav_ringacc_get_ring_id( + uc->tchan->tc_ring); + + return d; +} + +static struct udma_desc *udma_prep_slave_sg_tr( + struct udma_chan *uc, struct scatterlist *sgl, unsigned int sglen, + enum dma_transfer_direction dir, unsigned long tx_flags, void *context) +{ + enum dma_slave_buswidth dev_width; + struct scatterlist *sgent; + struct udma_desc *d; + size_t tr_size; + struct cppi50_tr_req_type1 *tr_req = NULL; + unsigned int i; + u8 elsize; + u32 burst; + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + /* Bus width translates to the element size (ES) */ + switch (dev_width) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + elsize = UDMA_ELSIZE_8; + break; + case DMA_SLAVE_BUSWIDTH_2_BYTES: + elsize = UDMA_ELSIZE_16; + break; + case DMA_SLAVE_BUSWIDTH_3_BYTES: + elsize = UDMA_ELSIZE_24; + break; + case DMA_SLAVE_BUSWIDTH_4_BYTES: + elsize = UDMA_ELSIZE_32; + break; + case DMA_SLAVE_BUSWIDTH_8_BYTES: + elsize = UDMA_ELSIZE_64; + break; + default: /* not reached */ + return NULL; + } + + if (!burst) + burst = 1; + + /* Now allocate and setup the descriptor. */ + tr_size = sizeof(struct cppi50_tr_req_type1); + d = udma_alloc_tr_desc(uc, tr_size, sglen, dir); + if (!d) + return NULL; + + d->sglen = sglen; + + tr_req = (struct cppi50_tr_req_type1 *)d->hwdesc[0].tr_req_base; + for_each_sg(sgl, sgent, sglen, i) { + d->residue += sg_dma_len(sgent); + tr_req[i].flags = CPPI50_TR_FLAGS_TYPE(1); + tr_req[i].addr = sg_dma_address(sgent); + tr_req[i].icnt0 = burst * elsize_bytes[elsize]; + tr_req[i].dim1 = burst * elsize_bytes[elsize]; + tr_req[i].icnt1 = sg_dma_len(sgent) / tr_req[i].icnt0; + + tr_req[i].flags |= CPPI50_TR_FLAGS_SUPR_EVT; + } + + tr_req[i - 1].flags |= CPPI50_TR_FLAGS_EOP; + + return d; +} + +static inline int udma_configure_statictr(struct udma_chan *uc, + struct udma_desc *d, u8 elsize, + u16 elcnt) +{ + if (!uc->static_tr_type) + return 0; + + d->static_tr.elsize = elsize; + d->static_tr.elcnt = elcnt; + if (uc->pkt_mode) { + unsigned int div = elsize_bytes[elsize] * elcnt; + + if (uc->cyclic) + d->static_tr.bstcnt = d->residue / d->sglen / div; + else + d->static_tr.bstcnt = d->residue / div; + + if (uc->dir == DMA_DEV_TO_MEM && d->static_tr.bstcnt > 0xfff) + return -EINVAL; + } else { + d->static_tr.bstcnt = 0; + } + + return 0; +} + +static struct udma_desc *udma_prep_slave_sg_pkt( + struct udma_chan *uc, struct scatterlist *sgl, unsigned int sglen, + enum dma_transfer_direction dir, unsigned long tx_flags, void *context) +{ + struct scatterlist *sgent; + struct knav_udmap_host_desc_t *h_desc = NULL; + struct udma_desc *d; + u32 ring_id; + unsigned int i; + + d = kzalloc(sizeof(*d) + sglen * sizeof(d->hwdesc[0]), GFP_ATOMIC); + if (!d) + return NULL; + + if (dir == DMA_DEV_TO_MEM && sglen > 1) { + void *buffer; + int ret; + size_t total_len = 0; + + /* Count the total length of the receive SG buffer */ + for_each_sg(sgl, sgent, sglen, i) + total_len += sg_dma_len(sgent); + + buffer = kzalloc(total_len, GFP_ATOMIC); + if (!buffer) { + kfree(d); + return NULL; + } + + sg_init_table(&d->rx_sg_wa.single_sg, 1); + sg_set_buf(&d->rx_sg_wa.single_sg, buffer, total_len); + ret = dma_map_sg(uc->ud->dev, &d->rx_sg_wa.single_sg, 1, + DMA_FROM_DEVICE); + if (ret != 1) { + dev_err(uc->ud->dev, + "mapping of temp buffer error (%d)\n", ret); + kfree(buffer); + kfree(d); + return NULL; + } + + d->rx_sg_wa.in_use = true; + + d->rx_sg_wa.sgl = sgl; + d->rx_sg_wa.sglen = sglen; + d->rx_sg_wa.total_len = total_len; + + sgl = &d->rx_sg_wa.single_sg; + sglen = 1; + } + + d->sglen = sglen; + d->hwdesc_count = sglen; + + if (dir == DMA_DEV_TO_MEM) + ring_id = k3_nav_ringacc_get_ring_id(uc->rchan->r_ring); + else + ring_id = k3_nav_ringacc_get_ring_id(uc->tchan->tc_ring); + + for_each_sg(sgl, sgent, sglen, i) { + struct udma_hwdesc *hwdesc = &d->hwdesc[i]; + dma_addr_t sg_addr = sg_dma_address(sgent); + struct knav_udmap_host_desc_t *desc; + size_t sg_len = sg_dma_len(sgent); + + hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool, + GFP_ATOMIC, + &hwdesc->cppi5_desc_paddr); + if (!hwdesc->cppi5_desc_vaddr) { + dev_err(uc->ud->dev, + "descriptor%d allocation failed\n", i); + + udma_free_hwdesc(&d->vd); + kfree(d); + return NULL; + + } + + d->residue += sg_len; + hwdesc->cppi5_desc_size = uc->hdesc_size; + desc = hwdesc->cppi5_desc_vaddr; + + if (i == 0) { + knav_udmap_hdesc_init(desc, 0, 0); + /* Flow and Packed ID ??? */ + knav_udmap_hdesc_set_pktids(&desc->hdr, uc->id, 0x3fff); + knav_udmap_desc_set_retpolicy(&desc->hdr, 0, ring_id); + } else { + knav_udmap_hdesc_reset_hbdesc(desc); + knav_udmap_desc_set_retpolicy(&desc->hdr, 0, 0xffff); + } + + /* attach the sg buffer to the descriptor */ + knav_udmap_hdesc_attach_buf(desc, + sg_addr, sg_len, + sg_addr, sg_len); + + /* Attach link as host buffer descriptor */ + if (h_desc) + knav_udmap_hdesc_link_hbdesc(h_desc, + hwdesc->cppi5_desc_paddr); + + h_desc = desc; + } + + if (d->residue >= SZ_4M) { + dev_err(uc->ud->dev, + "%s: Transfer size %u is over the supported 4M range\n", + __func__, d->residue); + udma_free_hwdesc(&d->vd); + kfree(d); + return NULL; + } + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + knav_udmap_hdesc_set_pktlen(h_desc, d->residue); + + return d; +} + +static int udma_attach_metadata(struct dma_async_tx_descriptor *desc, + void *data, size_t len) +{ + struct udma_desc *d = to_udma_desc(desc); + struct udma_chan *uc = to_udma_chan(desc->chan); + struct knav_udmap_host_desc_t *h_desc; + u32 psd_size = len; + u32 flags = 0; + + if (!uc->pkt_mode || !uc->metadata_size) + return -ENOTSUPP; + + if (!data || len > uc->metadata_size) + return -EINVAL; + + if (uc->needs_epib && len < 16) + return -EINVAL; + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + if (d->dir == DMA_MEM_TO_DEV) + memcpy(h_desc->epib, data, len); + + if (uc->needs_epib) + psd_size -= 16; + + d->metadata = data; + d->metadata_size = len; + if (uc->needs_epib) + flags |= KNAV_UDMAP_INFO0_HDESC_EPIB_PRESENT; + + knav_udmap_hdesc_update_flags(h_desc, flags); + knav_udmap_hdesc_update_psdata_size(h_desc, psd_size); + + return 0; +} + +static void *udma_get_metadata_ptr(struct dma_async_tx_descriptor *desc, + size_t *payload_len, size_t *max_len) +{ + struct udma_desc *d = to_udma_desc(desc); + struct udma_chan *uc = to_udma_chan(desc->chan); + struct knav_udmap_host_desc_t *h_desc; + + if (!uc->pkt_mode || !uc->metadata_size) + return ERR_PTR(-ENOTSUPP); + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + + *max_len = uc->metadata_size; + + *payload_len = knav_udmap_desc_is_epib_present(&h_desc->hdr) ? 16 : 0; + *payload_len += knav_udmap_hdesc_get_psdata_size(h_desc); + + return h_desc->epib; +} + +static int udma_set_metadata_len(struct dma_async_tx_descriptor *desc, + size_t payload_len) +{ + struct udma_desc *d = to_udma_desc(desc); + struct udma_chan *uc = to_udma_chan(desc->chan); + struct knav_udmap_host_desc_t *h_desc; + u32 psd_size = payload_len; + u32 flags = 0; + + if (!uc->pkt_mode || !uc->metadata_size) + return -ENOTSUPP; + + if (payload_len > uc->metadata_size) + return -EINVAL; + + if (uc->needs_epib && payload_len < 16) + return -EINVAL; + + h_desc = d->hwdesc[0].cppi5_desc_vaddr; + + if (uc->needs_epib) { + psd_size -= 16; + flags |= KNAV_UDMAP_INFO0_HDESC_EPIB_PRESENT; + } + + knav_udmap_hdesc_update_flags(h_desc, flags); + knav_udmap_hdesc_update_psdata_size(h_desc, psd_size); + + return 0; +} + +static struct dma_descriptor_metadata_ops metadata_ops = { + .attach = udma_attach_metadata, + .get_ptr = udma_get_metadata_ptr, + .set_len = udma_set_metadata_len, +}; + +static struct dma_async_tx_descriptor *udma_prep_slave_sg( + struct dma_chan *chan, struct scatterlist *sgl, unsigned int sglen, + enum dma_transfer_direction dir, unsigned long tx_flags, void *context) +{ + struct udma_chan *uc = to_udma_chan(chan); + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + u8 elsize; + u32 burst; + + if (dir != uc->dir) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, udma_get_dir_text(uc->dir), + udma_get_dir_text(dir)); + return NULL; + } + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(chan->device->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + /* Bus width translates to the element size (ES) */ + switch (dev_width) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + elsize = UDMA_ELSIZE_8; + break; + case DMA_SLAVE_BUSWIDTH_2_BYTES: + elsize = UDMA_ELSIZE_16; + break; + case DMA_SLAVE_BUSWIDTH_3_BYTES: + elsize = UDMA_ELSIZE_24; + break; + case DMA_SLAVE_BUSWIDTH_4_BYTES: + elsize = UDMA_ELSIZE_32; + break; + case DMA_SLAVE_BUSWIDTH_8_BYTES: + elsize = UDMA_ELSIZE_64; + break; + default: /* not reached */ + return NULL; + } + + if (!burst) + burst = 1; + + if (uc->pkt_mode) + d = udma_prep_slave_sg_pkt(uc, sgl, sglen, dir, tx_flags, + context); + else + d = udma_prep_slave_sg_tr(uc, sgl, sglen, dir, tx_flags, + context); + + if (!d) + return NULL; + + d->dir = dir; + d->desc_idx = 0; + d->tr_idx = 0; + + /* static TR for remote PDMA */ + if (udma_configure_statictr(uc, d, elsize, burst)) { + dev_err(uc->ud->dev, + "%s: StaticTR Z is limted to maximum 4095 (%u)\n", + __func__, d->static_tr.bstcnt); + + udma_free_hwdesc(&d->vd); + kfree(d); + return NULL; + } + + if (uc->metadata_size) + d->vd.tx.metadata_ops = &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); +} + +static struct udma_desc *udma_prep_dma_cyclic_tr( + struct udma_chan *uc, dma_addr_t buf_addr, size_t buf_len, + size_t period_len, enum dma_transfer_direction dir, unsigned long flags) +{ + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + size_t tr_size; + struct cppi50_tr_req_type1 *tr_req; + unsigned int i; + unsigned int periods = buf_len / period_len; + u32 burst; + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst = 1; + + /* Now allocate and setup the descriptor. */ + tr_size = sizeof(struct cppi50_tr_req_type1); + d = udma_alloc_tr_desc(uc, tr_size, periods, dir); + if (!d) + return NULL; + + tr_req = (struct cppi50_tr_req_type1 *)d->hwdesc[0].tr_req_base; + for (i = 0; i < periods; i++) { + tr_req[i].flags = CPPI50_TR_FLAGS_TYPE(1); + tr_req[i].addr = buf_addr + period_len * i; + tr_req[i].icnt0 = dev_width; + tr_req[i].icnt1 = period_len / dev_width; + tr_req[i].dim1 = dev_width; + + if (!(flags & DMA_PREP_INTERRUPT)) + tr_req[i].flags |= CPPI50_TR_FLAGS_SUPR_EVT; + } + + return d; +} + +static struct udma_desc *udma_prep_dma_cyclic_pkt( + struct udma_chan *uc, dma_addr_t buf_addr, size_t buf_len, + size_t period_len, enum dma_transfer_direction dir, unsigned long flags) +{ + struct udma_desc *d; + u32 ring_id; + int i; + int periods = buf_len / period_len; + + if (periods > 15) + return NULL; + + if (period_len > 0x3FFFFF) + return NULL; + + d = kzalloc(sizeof(*d) + periods * sizeof(d->hwdesc[0]), GFP_ATOMIC); + if (!d) + return NULL; + + d->hwdesc_count = periods; + + /* TODO: re-check this... */ + if (dir == DMA_DEV_TO_MEM) + ring_id = k3_nav_ringacc_get_ring_id(uc->rchan->r_ring); + else + ring_id = k3_nav_ringacc_get_ring_id(uc->tchan->tc_ring); + + for (i = 0; i < periods; i++) { + struct udma_hwdesc *hwdesc = &d->hwdesc[i]; + dma_addr_t period_addr = buf_addr + (period_len * i); + struct knav_udmap_host_desc_t *h_desc; + + hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool, + GFP_ATOMIC, + &hwdesc->cppi5_desc_paddr); + if (!hwdesc->cppi5_desc_vaddr) { + dev_err(uc->ud->dev, + "descriptor%d allocation failed\n", i); + + udma_free_hwdesc(&d->vd); + kfree(d); + return NULL; + } + + hwdesc->cppi5_desc_size = uc->hdesc_size; + h_desc = hwdesc->cppi5_desc_vaddr; + + knav_udmap_hdesc_init(h_desc, 0, 0); + knav_udmap_hdesc_set_pktlen(h_desc, period_len); + + /* Flow and Packed ID ??? */ + knav_udmap_hdesc_set_pktids(&h_desc->hdr, uc->id, 0x3fff); + knav_udmap_desc_set_retpolicy(&h_desc->hdr, 0, ring_id); + + /* attach each period to a new descriptor */ + knav_udmap_hdesc_attach_buf(h_desc, + period_addr, period_len, + period_addr, period_len); + } + + return d; +} + +static struct dma_async_tx_descriptor *udma_prep_dma_cyclic( + struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len, + size_t period_len, enum dma_transfer_direction dir, unsigned long flags) +{ + struct udma_chan *uc = to_udma_chan(chan); + enum dma_slave_buswidth dev_width; + struct udma_desc *d; + u8 elsize; + u32 burst; + + if (dir != uc->dir) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, udma_get_dir_text(uc->dir), + udma_get_dir_text(dir)); + return NULL; + } + + uc->cyclic = true; + + if (dir == DMA_DEV_TO_MEM) { + dev_width = uc->cfg.src_addr_width; + burst = uc->cfg.src_maxburst; + } else if (dir == DMA_MEM_TO_DEV) { + dev_width = uc->cfg.dst_addr_width; + burst = uc->cfg.dst_maxburst; + } else { + dev_err(uc->ud->dev, "%s: bad direction?\n", __func__); + return NULL; + } + + if (!burst) + burst = 1; + + /* Bus width translates to the element size (ES) */ + switch (dev_width) { + case DMA_SLAVE_BUSWIDTH_1_BYTE: + elsize = UDMA_ELSIZE_8; + break; + case DMA_SLAVE_BUSWIDTH_2_BYTES: + elsize = UDMA_ELSIZE_16; + break; + case DMA_SLAVE_BUSWIDTH_3_BYTES: + elsize = UDMA_ELSIZE_24; + break; + case DMA_SLAVE_BUSWIDTH_4_BYTES: + elsize = UDMA_ELSIZE_32; + break; + case DMA_SLAVE_BUSWIDTH_8_BYTES: + elsize = UDMA_ELSIZE_64; + break; + default: /* not reached */ + return NULL; + } + + if (uc->pkt_mode) + d = udma_prep_dma_cyclic_pkt(uc, buf_addr, buf_len, period_len, + dir, flags); + else + d = udma_prep_dma_cyclic_tr(uc, buf_addr, buf_len, period_len, + dir, flags); + + if (!d) + return NULL; + + d->sglen = buf_len / period_len; + + d->dir = dir; + d->residue = buf_len; + + /* static TR for remote PDMA */ + if (udma_configure_statictr(uc, d, elsize, burst)) { + dev_err(uc->ud->dev, + "%s: StaticTR Z is limted to maximum 4095 (%u)\n", + __func__, d->static_tr.bstcnt); + + udma_free_hwdesc(&d->vd); + kfree(d); + return NULL; + } + + if (uc->metadata_size) + d->vd.tx.metadata_ops = &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, flags); +} + +static struct dma_async_tx_descriptor *udma_prep_dma_memcpy( + struct dma_chan *chan, dma_addr_t dest, dma_addr_t src, + size_t len, unsigned long tx_flags) +{ + struct udma_chan *uc = to_udma_chan(chan); + struct udma_desc *d; + struct cppi50_tr_req_type15 *tr_req; + int num_tr; + size_t tr_size = sizeof(struct cppi50_tr_req_type15); + u16 tr0_cnt0, tr0_cnt1, tr1_cnt0; + + if (uc->dir != DMA_MEM_TO_MEM) { + dev_err(chan->device->dev, + "%s: chan%d is for %s, not supporting %s\n", + __func__, uc->id, udma_get_dir_text(uc->dir), + udma_get_dir_text(DMA_MEM_TO_MEM)); + return NULL; + } + + if (len < SZ_64K) { + num_tr = 1; + tr0_cnt0 = len; + tr0_cnt1 = 1; + } else { + unsigned long align_to = __ffs(src | dest); + + if (align_to > 3) + align_to = 3; + /* + * Keep simple: tr0: SZ_64K-alignment blocks, + * tr1: the remaining + */ + num_tr = 2; + tr0_cnt0 = (SZ_64K - BIT(align_to)); + if (len / tr0_cnt0 >= SZ_64K) { + dev_err(uc->ud->dev, "size %zu is not supported\n", + len); + return NULL; + } + + tr0_cnt1 = len / tr0_cnt0; + tr1_cnt0 = len % tr0_cnt0; + } + + d = udma_alloc_tr_desc(uc, tr_size, num_tr, DMA_MEM_TO_MEM); + if (!d) + return NULL; + + d->dir = DMA_MEM_TO_MEM; + d->desc_idx = 0; + d->tr_idx = 0; + d->residue = len; + + tr_req = (struct cppi50_tr_req_type15 *)d->hwdesc[0].tr_req_base; + + tr_req[0].flags = CPPI50_TR_FLAGS_TYPE(15); + tr_req[0].addr = src; + tr_req[0].icnt0 = tr0_cnt0; + tr_req[0].icnt1 = tr0_cnt1; + tr_req[0].icnt2 = 1; + tr_req[0].icnt3 = 1; + tr_req[0].dim1 = tr0_cnt0; + + tr_req[0].daddr = dest; + tr_req[0].dicnt0 = tr0_cnt0; + tr_req[0].dicnt1 = tr0_cnt1; + tr_req[0].dicnt2 = 1; + tr_req[0].dicnt3 = 1; + tr_req[0].ddim1 = tr0_cnt0; + + tr_req[0].flags |= CPPI50_TR_FLAGS_WAIT; + + tr_req[0].flags |= CPPI50_TR_FLAGS_SUPR_EVT; + + if (num_tr == 2) { + tr_req[1].flags = CPPI50_TR_FLAGS_TYPE(15); + tr_req[1].addr = src + tr0_cnt1 * tr0_cnt0; + tr_req[1].icnt0 = tr1_cnt0; + tr_req[1].icnt1 = 1; + tr_req[1].icnt2 = 1; + tr_req[1].icnt3 = 1; + + tr_req[1].daddr = dest + tr0_cnt1 * tr0_cnt0; + tr_req[1].dicnt0 = tr1_cnt0; + tr_req[1].dicnt1 = 1; + tr_req[1].dicnt2 = 1; + tr_req[1].dicnt3 = 1; + + tr_req[1].flags |= CPPI50_TR_FLAGS_WAIT; + + tr_req[1].flags |= CPPI50_TR_FLAGS_SUPR_EVT; + } + + tr_req[num_tr - 1].flags |= CPPI50_TR_FLAGS_EOP; + + if (uc->metadata_size) + d->vd.tx.metadata_ops = &metadata_ops; + + return vchan_tx_prep(&uc->vc, &d->vd, tx_flags); +} + +static void udma_issue_pending(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + unsigned long flags; + + spin_lock_irqsave(&uc->vc.lock, flags); + + /* If we have something pending and no active descriptor, then */ + if (vchan_issue_pending(&uc->vc) && !uc->desc) { + /* + * start a descriptor if the channel is NOT [marked as + * terminating _and_ it is still running (teardown has not + * completed yet)]. + */ + if (!(uc->state == UDMA_CHAN_IS_TERMINATING && + udma_is_chan_running(uc))) + udma_start(uc); + } + + spin_unlock_irqrestore(&uc->vc.lock, flags); +} + +/* Not much yet */ +static enum dma_status udma_tx_status(struct dma_chan *chan, + dma_cookie_t cookie, + struct dma_tx_state *txstate) +{ + struct udma_chan *uc = to_udma_chan(chan); + enum dma_status ret; + + ret = dma_cookie_status(chan, cookie, txstate); + + if (!udma_is_chan_running(uc)) + ret = DMA_COMPLETE; + + if (ret == DMA_COMPLETE || !txstate) + return ret; + + if (uc->desc && uc->desc->vd.tx.cookie == cookie) { + u32 pdma_bcnt = 0; + u32 bcnt = 0; + u32 pcnt = 0; + u32 residue = uc->desc->residue; + u32 delay = 0; + + if (uc->desc->dir == DMA_MEM_TO_DEV) { + bcnt = udma_tchanrt_read(uc->tchan, + UDMA_TCHAN_RT_BCNT_REG); + pdma_bcnt = udma_tchanrt_read(uc->tchan, + UDMA_TCHAN_RT_PEER_BCNT_REG); + pcnt = udma_tchanrt_read(uc->tchan, + UDMA_TCHAN_RT_PCNT_REG); + + if (bcnt > pdma_bcnt) + delay = bcnt - pdma_bcnt; + } else if (uc->desc->dir == DMA_DEV_TO_MEM) { + bcnt = udma_rchanrt_read(uc->rchan, + UDMA_RCHAN_RT_SBCNT_REG); + pdma_bcnt = udma_rchanrt_read(uc->rchan, + UDMA_RCHAN_RT_PEER_BCNT_REG); + pcnt = udma_rchanrt_read(uc->rchan, + UDMA_RCHAN_RT_PCNT_REG); + + if (pdma_bcnt > bcnt) + delay = pdma_bcnt - bcnt; + } else { + u32 sbcnt; + + sbcnt = udma_tchanrt_read(uc->tchan, + UDMA_TCHAN_RT_SBCNT_REG); + bcnt = udma_tchanrt_read(uc->tchan, + UDMA_TCHAN_RT_PEER_BCNT_REG); + pcnt = udma_tchanrt_read(uc->tchan, + UDMA_TCHAN_RT_PCNT_REG); + + if (sbcnt > bcnt) + delay = sbcnt - bcnt; + } + + residue -= ((bcnt - uc->bcnt) % uc->desc->residue); + dma_set_residue(txstate, residue); + dma_set_cached(txstate, delay); + + } else { + ret = DMA_COMPLETE; + } + + return ret; +} + +static int udma_pause(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + + if (!uc->desc) + return -EINVAL; + + /* pause the channel */ + switch (uc->desc->dir) { + case DMA_DEV_TO_MEM: + udma_rchanrt_update_bits(uc->rchan, + UDMA_RCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_PAUSE, + UDMA_PEER_RT_EN_PAUSE); + break; + case DMA_MEM_TO_DEV: + udma_tchanrt_update_bits(uc->tchan, + UDMA_TCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_PAUSE, + UDMA_PEER_RT_EN_PAUSE); + break; + case DMA_MEM_TO_MEM: + udma_tchanrt_update_bits(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_PAUSE, + UDMA_CHAN_RT_CTL_PAUSE); + break; + default: + return -EINVAL; + } + + return 0; +} + +static int udma_resume(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + + if (!uc->desc) + return -EINVAL; + + /* resume the channel */ + switch (uc->desc->dir) { + case DMA_DEV_TO_MEM: + udma_rchanrt_update_bits(uc->rchan, + UDMA_RCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_PAUSE, 0); + + break; + case DMA_MEM_TO_DEV: + udma_tchanrt_update_bits(uc->tchan, + UDMA_TCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_PAUSE, 0); + break; + case DMA_MEM_TO_MEM: + udma_tchanrt_update_bits(uc->tchan, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_PAUSE, 0); + break; + default: + return -EINVAL; + } + + return 0; +} + +static int udma_terminate_all(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + unsigned long flags; + LIST_HEAD(head); + + spin_lock_irqsave(&uc->vc.lock, flags); + + if (udma_is_chan_running(uc)) + udma_stop(uc); + + if (uc->desc) { + uc->terminated_desc = uc->desc; + uc->desc = NULL; + uc->terminated_desc->terminated = true; + } + + uc->paused = false; + + vchan_get_all_descriptors(&uc->vc, &head); + spin_unlock_irqrestore(&uc->vc.lock, flags); + vchan_dma_desc_free_list(&uc->vc, &head); + + return 0; +} + +static void udma_synchronize(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + unsigned long timeout = msecs_to_jiffies(1000); + + vchan_synchronize(&uc->vc); + + if (uc->state == UDMA_CHAN_IS_TERMINATING) { + timeout = wait_for_completion_timeout(&uc->teardown_completed, + timeout); + if (!timeout) { + dev_warn(uc->ud->dev, "chan%d teardown timeout!\n", + uc->id); + udma_dump_chan_stdata(uc); + udma_reset_chan(uc, true); + } + } + + udma_reset_chan(uc, false); + if (udma_is_chan_running(uc)) + dev_warn(uc->ud->dev, "chan%d refused to stop!\n", uc->id); + + udma_reset_rings(uc); +} + +static void udma_desc_pre_callback(struct virt_dma_chan *vc, + struct virt_dma_desc *vd, + struct dmaengine_result *result) +{ + struct udma_chan *uc = to_udma_chan(&vc->chan); + struct udma_desc *d; + + if (!vd) + return; + + d = to_udma_desc(&vd->tx); + + if (d->metadata_size) + udma_fetch_epib(uc, d); + + /* TODO: peek into the desc to know the real length */ + if (d->rx_sg_wa.in_use) { + void *src = sg_virt(&d->rx_sg_wa.single_sg); + + dma_sync_sg_for_cpu(uc->ud->dev, &d->rx_sg_wa.single_sg, 1, + DMA_FROM_DEVICE); + /* Ensure that reads are not moved before this point */ + rmb(); + + sg_copy_from_buffer(d->rx_sg_wa.sgl, d->rx_sg_wa.sglen, src, + d->rx_sg_wa.total_len); + } + + /* Provide residue information for the client */ + if (result) { + void *desc_vaddr = udma_curr_cppi5_desc_vaddr(d, d->desc_idx); + + if (knav_udmap_desc_get_type(desc_vaddr) == + KNAV_UDMAP_INFO0_DESC_TYPE_VAL_HOST) { + result->residue = knav_udmap_hdesc_get_pktlen(desc_vaddr); + if (result->residue == d->residue) + result->result = DMA_TRANS_NOERROR; + else + result->result = DMA_TRANS_ABORTED; + } else { + result->residue = d->residue; + result->result = DMA_TRANS_NOERROR; + } + } +} + +/* + * This tasklet handles the completion of a DMA descriptor by + * calling its callback and freeing it. + */ +static void udma_vchan_complete(unsigned long arg) +{ + struct virt_dma_chan *vc = (struct virt_dma_chan *)arg; + struct virt_dma_desc *vd, *_vd; + struct dmaengine_desc_callback cb; + LIST_HEAD(head); + + spin_lock_irq(&vc->lock); + list_splice_tail_init(&vc->desc_completed, &head); + vd = vc->cyclic; + if (vd) { + vc->cyclic = NULL; + dmaengine_desc_get_callback(&vd->tx, &cb); + } else { + memset(&cb, 0, sizeof(cb)); + } + spin_unlock_irq(&vc->lock); + + udma_desc_pre_callback(vc, vd, NULL); + dmaengine_desc_callback_invoke(&cb, NULL); + + list_for_each_entry_safe(vd, _vd, &head, node) { + struct dmaengine_result result; + + dmaengine_desc_get_callback(&vd->tx, &cb); + + list_del(&vd->node); + + udma_desc_pre_callback(vc, vd, &result); + dmaengine_desc_callback_invoke(&cb, &result); + + vchan_vdesc_fini(vd); + } +} + +static void udma_free_chan_resources(struct dma_chan *chan) +{ + struct udma_chan *uc = to_udma_chan(chan); + struct udma_dev *ud = to_udma_dev(chan->device); + struct udma_tisci_rm *tisci_rm = &ud->tisci_rm; + + udma_terminate_all(chan); + + if (uc->irq_num_ring > 0) { + free_irq(uc->irq_num_ring, uc); + + ti_sci_inta_unregister_event(ud->dev, uc->irq_ra_tisci, + uc->irq_ra_idx, uc->irq_num_ring); + uc->irq_num_ring = 0; + } + if (uc->irq_num_udma > 0) { + free_irq(uc->irq_num_udma, uc); + + ti_sci_inta_unregister_event(ud->dev, tisci_rm->tisci_dev_id, + uc->irq_udma_idx, + uc->irq_num_udma); + uc->irq_num_udma = 0; + } + + /* Release PSI-L pairing */ + if (uc->psil_paired) { + navss_psil_unpair(ud, uc->src_thread, uc->dst_thread); + uc->psil_paired = false; + } + + vchan_free_chan_resources(&uc->vc); + tasklet_kill(&uc->vc.task); + + pm_runtime_put(ud->ddev.dev); + + udma_free_tx_resources(uc); + udma_free_rx_resources(uc); + + uc->slave_thread_id = -1; + uc->dir = DMA_MEM_TO_MEM; + + if (uc->use_dma_pool) { + dma_pool_destroy(uc->hdesc_pool); + uc->use_dma_pool = false; + } +} + +static struct platform_driver udma_driver; + +static bool udma_dma_filter_fn(struct dma_chan *chan, void *param) +{ + u32 *args; + struct udma_chan *uc; + struct udma_dev *ud; + struct device_node *chconf_node, *slave_node; + char prop[50]; + u32 val; + + if (chan->device->dev->driver != &udma_driver.driver) + return false; + + uc = to_udma_chan(chan); + ud = uc->ud; + args = param; + + if (args[2] == UDMA_DIR_TX) { + uc->dir = DMA_MEM_TO_DEV; + } else if (args[2] == UDMA_DIR_RX) { + uc->dir = DMA_DEV_TO_MEM; + } else { + dev_err(ud->dev, "Invalid direction (%u)\n", args[2]); + return false; + } + + slave_node = of_find_node_by_phandle(args[0]); + if (!slave_node) { + dev_err(ud->dev, "Slave node is missing\n"); + return false; + } + + snprintf(prop, sizeof(prop), "ti,psil-config%u", args[1]); + chconf_node = of_find_node_by_name(slave_node, prop); + if (!chconf_node) { + dev_err(ud->dev, "Channel configuration node is missing\n"); + of_node_put(slave_node); + return false; + } + + if (!of_property_read_u32(chconf_node, "linux,udma-mode", &val)) { + if (val == UDMA_PKT_MODE) + uc->pkt_mode = true; + } + + if (!of_property_read_u32(chconf_node, "statictr-type", &val)) + uc->static_tr_type = val; + + if (!of_property_read_u32(chconf_node, "ti,channel-tpl", &val)) + uc->channel_tpl = val; + + uc->needs_epib = of_property_read_bool(chconf_node, "ti,needs-epib"); + if (!of_property_read_u32(chconf_node, "ti,psd-size", &val)) + uc->psd_size = val; + uc->metadata_size = (uc->needs_epib ? 16 : 0) + uc->psd_size; + + uc->desc_align = 64; + if (uc->desc_align < dma_get_cache_alignment()) + uc->desc_align = dma_get_cache_alignment(); + + if (uc->pkt_mode) + uc->hdesc_size = ALIGN(sizeof(struct knav_udmap_host_desc_t) + + uc->metadata_size, uc->desc_align); + + of_node_put(chconf_node); + + if (of_property_read_u32(slave_node, "ti,psil-base", &val)) { + dev_err(ud->dev, "ti,psil-base is missing\n"); + of_node_put(slave_node); + return false; + } + + uc->slave_thread_id = val + args[1]; + + of_node_put(slave_node); + + dev_dbg(ud->dev, "%s: Slave %s thread%d will be handled by vchan %d\n", + __func__, udma_get_dir_text(uc->dir), uc->slave_thread_id, + uc->id); + + return true; +} + +static struct dma_chan *udma_of_xlate(struct of_phandle_args *dma_spec, + struct of_dma *ofdma) +{ + struct udma_dev *ud = ofdma->of_dma_data; + struct dma_chan *chan; + + if (dma_spec->args_count != 3) + return NULL; + + chan = dmadev_get_slave_channel(&ud->ddev, udma_dma_filter_fn, + &dma_spec->args[0]); + if (!chan) { + dev_err(ud->dev, "get channel fail in %s.\n", __func__); + return ERR_PTR(-EINVAL); + } + + return chan; +} + +struct udma_match_data am654_main_data = { + .tpl_levels = 2, + .level_start_idx = { + [0] = 8, /* Normal channels */ + [1] = 0, /* High Throughput channels */ + }, +}; + +struct udma_match_data am654_mcu_data = { + .tpl_levels = 2, + .level_start_idx = { + [0] = 2, /* Normal channels */ + [1] = 0, /* High Throughput channels */ + }, +}; + +static const struct of_device_id udma_of_match[] = { + { .compatible = "ti,am654-navss-main-udmap", .data = &am654_main_data, }, + { .compatible = "ti,am654-navss-mcu-udmap", .data = &am654_mcu_data, }, + {}, +}; +MODULE_DEVICE_TABLE(of, udma_of_match); + +static int udma_get_mmrs(struct platform_device *pdev, struct udma_dev *ud) +{ + struct resource *res; + int i; + + for (i = 0; i < MMR_LAST; i++) { + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, + mmr_names[i]); + ud->mmrs[i] = devm_ioremap_resource(&pdev->dev, res); + if (IS_ERR(ud->mmrs[i])) + return PTR_ERR(ud->mmrs[i]); + } + + return 0; +} + +static int udma_setup_resources(struct udma_dev *ud) +{ + struct device *dev = ud->dev; + int ch_count, i; + u32 cap2, cap3; + struct ti_sci_resource_desc *rm_desc; + struct ti_sci_resource *rm_res; + struct udma_tisci_rm *tisci_rm = &ud->tisci_rm; + char *range_names[] = { "ti,sci-rm-range-tchan", + "ti,sci-rm-range-rchan", + "ti,sci-rm-range-rflow" }; + + cap2 = udma_read(ud->mmrs[MMR_GCFG], 0x28); + cap3 = udma_read(ud->mmrs[MMR_GCFG], 0x2c); + + ud->rflow_cnt = cap3 & 0x3fff; + ud->tchan_cnt = cap2 & 0x1ff; + ud->echan_cnt = (cap2 >> 9) & 0x1ff; + ud->rchan_cnt = (cap2 >> 18) & 0x1ff; + ch_count = ud->tchan_cnt + ud->rchan_cnt; + + ud->tchan_map = devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->tchans = devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans), + GFP_KERNEL); + ud->rchan_map = devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->rchans = devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans), + GFP_KERNEL); + ud->rflow_map = devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rflow_cnt), + sizeof(unsigned long), GFP_KERNEL); + ud->rflow_map_reserved = devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt), + sizeof(unsigned long), + GFP_KERNEL); + ud->rflows = devm_kcalloc(dev, ud->rflow_cnt, sizeof(*ud->rflows), + GFP_KERNEL); + + if (!ud->tchan_map || !ud->rchan_map || !ud->rflow_map || + !ud->rflow_map_reserved || !ud->tchans || !ud->rchans || + !ud->rflows) + return -ENOMEM; + + /* + * RX flows with the same Ids as RX channels are reserved to be used + * as default flows if remote HW can't generate flow_ids. Those + * RX flows can be requested only explicitly by id. + */ + bitmap_set(ud->rflow_map_reserved, 0, ud->rchan_cnt); + + /* Get resource ranges from tisci */ + for (i = 0; i < RM_RANGE_LAST; i++) + tisci_rm->rm_ranges[i] = devm_ti_sci_get_of_resource( + tisci_rm->tisci, dev, + range_names[i]); + + /* tchan ranges */ + rm_res = tisci_rm->rm_ranges[RM_RANGE_TCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->tchan_map, ud->tchan_cnt); + } else { + bitmap_fill(ud->tchan_map, ud->tchan_cnt); + for (i = 0; i < rm_res->sets; i++) { + rm_desc = &rm_res->desc[i]; + bitmap_clear(ud->tchan_map, rm_desc->start, + rm_desc->num); + } + } + + /* rchan and matching default flow ranges */ + rm_res = tisci_rm->rm_ranges[RM_RANGE_RCHAN]; + if (IS_ERR(rm_res)) { + bitmap_zero(ud->rchan_map, ud->rchan_cnt); + bitmap_zero(ud->rflow_map, ud->rchan_cnt); + } else { + bitmap_fill(ud->rchan_map, ud->rchan_cnt); + bitmap_fill(ud->rflow_map, ud->rchan_cnt); + for (i = 0; i < rm_res->sets; i++) { + rm_desc = &rm_res->desc[i]; + bitmap_clear(ud->rchan_map, rm_desc->start, + rm_desc->num); + bitmap_clear(ud->rflow_map, rm_desc->start, + rm_desc->num); + } + } + + /* GP rflow ranges */ + rm_res = tisci_rm->rm_ranges[RM_RANGE_RFLOW]; + if (IS_ERR(rm_res)) { + bitmap_clear(ud->rflow_map, ud->rchan_cnt, + ud->rflow_cnt - ud->rchan_cnt); + } else { + bitmap_set(ud->rflow_map, ud->rchan_cnt, + ud->rflow_cnt - ud->rchan_cnt); + for (i = 0; i < rm_res->sets; i++) { + rm_desc = &rm_res->desc[i]; + bitmap_clear(ud->rflow_map, rm_desc->start, + rm_desc->num); + } + } + + /* + * HACK: tchan0, rchan0,1 and rflow0,1 on main_navss is dedicated to + * sysfw. + * Only UDMAP on main_navss have echan, use it as a hint for now. + */ + if (ud->echan_cnt) { + set_bit(0, ud->tchan_map); + set_bit(0, ud->rchan_map); + set_bit(1, ud->rchan_map); + set_bit(0, ud->rflow_map); + set_bit(1, ud->rflow_map); + } + + ch_count -= bitmap_weight(ud->tchan_map, ud->tchan_cnt); + ch_count -= bitmap_weight(ud->rchan_map, ud->rchan_cnt); + if (!ch_count) + return -ENODEV; + + ud->channels = devm_kcalloc(dev, ch_count, sizeof(*ud->channels), + GFP_KERNEL); + if (!ud->channels) + return -ENOMEM; + + dev_info(dev, + "Channels: %d (tchan: %u, echan: %u, rchan: %u, rflow: %u)\n", + ch_count, ud->tchan_cnt, ud->echan_cnt, ud->rchan_cnt, + ud->rflow_cnt); + + return ch_count; +} + +#define TI_UDMAC_BUSWIDTHS (BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) | \ + BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) | \ + BIT(DMA_SLAVE_BUSWIDTH_3_BYTES) | \ + BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) | \ + BIT(DMA_SLAVE_BUSWIDTH_8_BYTES)) + +static int udma_probe(struct platform_device *pdev) +{ + struct device_node *parent_irq_node; + struct device_node *navss_node = pdev->dev.parent->of_node; + struct device *dev = &pdev->dev; + struct udma_dev *ud; + const struct of_device_id *match; + int i, ret; + int ch_count; + + ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(48)); + if (ret) + dev_err(dev, "failed to set dma mask stuff\n"); + + ud = devm_kzalloc(dev, sizeof(*ud), GFP_KERNEL); + if (!ud) + return -ENOMEM; + + ret = udma_get_mmrs(pdev, ud); + if (ret) + return ret; + + ud->tisci_rm.tisci = ti_sci_get_by_phandle(dev->of_node, "ti,sci"); + if (IS_ERR(ud->tisci_rm.tisci)) + return PTR_ERR(ud->tisci_rm.tisci); + + ret = of_property_read_u32(dev->of_node, "ti,sci-dev-id", + &ud->tisci_rm.tisci_dev_id); + if (ret) { + dev_err(dev, "ti,sci-dev-id read failure %d\n", ret); + return ret; + } + + ret = of_property_read_u32(navss_node, "ti,sci-dev-id", + &ud->tisci_rm.tisci_navss_dev_id); + if (ret) { + dev_err(dev, "NAVSS ti,sci-dev-id read failure %d\n", ret); + return ret; + } + + ud->tisci_rm.tisci_udmap_ops = &ud->tisci_rm.tisci->ops.rm_udmap_ops; + ud->tisci_rm.tisci_psil_ops = &ud->tisci_rm.tisci->ops.rm_psil_ops; + + ud->ringacc = of_k3_nav_ringacc_get_by_phandle(dev->of_node, + "ti,ringacc"); + if (IS_ERR(ud->ringacc)) + return PTR_ERR(ud->ringacc); + + parent_irq_node = of_irq_find_parent(dev->of_node); + if (!parent_irq_node) { + dev_err(dev, "Failed to get IRQ parent node\n"); + return -ENODEV; + } + + ud->irq_domain = irq_find_host(parent_irq_node); + if (!ud->irq_domain) + return -EPROBE_DEFER; + + match = of_match_node(udma_of_match, dev->of_node); + if (!match) { + dev_err(dev, "No compatible match found\n"); + return -ENODEV; + } + ud->match_data = match->data; + + dma_cap_set(DMA_SLAVE, ud->ddev.cap_mask); + dma_cap_set(DMA_CYCLIC, ud->ddev.cap_mask); + dma_cap_set(DMA_MEMCPY, ud->ddev.cap_mask); + + ud->ddev.device_alloc_chan_resources = udma_alloc_chan_resources; + ud->ddev.device_config = udma_slave_config; + ud->ddev.device_prep_slave_sg = udma_prep_slave_sg; + ud->ddev.device_prep_dma_cyclic = udma_prep_dma_cyclic; + ud->ddev.device_prep_dma_memcpy = udma_prep_dma_memcpy; + ud->ddev.device_issue_pending = udma_issue_pending; + ud->ddev.device_tx_status = udma_tx_status; + ud->ddev.device_pause = udma_pause; + ud->ddev.device_resume = udma_resume; + ud->ddev.device_terminate_all = udma_terminate_all; + ud->ddev.device_synchronize = udma_synchronize; + + ud->ddev.device_free_chan_resources = udma_free_chan_resources; + ud->ddev.src_addr_widths = TI_UDMAC_BUSWIDTHS; + ud->ddev.dst_addr_widths = TI_UDMAC_BUSWIDTHS; + ud->ddev.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV) | + BIT(DMA_MEM_TO_MEM); + ud->ddev.residue_granularity = DMA_RESIDUE_GRANULARITY_BURST; + ud->ddev.copy_align = DMAENGINE_ALIGN_8_BYTES; + ud->ddev.desc_metadata_modes = DESC_METADATA_CLIENT | + DESC_METADATA_ENGINE; + ud->ddev.dev = dev; + ud->dev = dev; + + INIT_LIST_HEAD(&ud->ddev.channels); + INIT_LIST_HEAD(&ud->desc_to_purge); + + ret = of_property_read_u32(dev->of_node, "ti,psil-base", + &ud->psil_base); + if (ret) { + dev_info(dev, "Missing ti,psil-base property, using %d.\n", + ret); + return ret; + } + + ch_count = udma_setup_resources(ud); + if (ch_count <= 0) + return ch_count; + + spin_lock_init(&ud->lock); + INIT_WORK(&ud->purge_work, udma_purge_desc_work); + + for (i = 0; i < ud->tchan_cnt; i++) { + struct udma_tchan *tchan = &ud->tchans[i]; + + tchan->id = i; + tchan->reg_rt = ud->mmrs[MMR_TCHANRT] + UDMA_CH_1000(i); + } + + for (i = 0; i < ud->rchan_cnt; i++) { + struct udma_rchan *rchan = &ud->rchans[i]; + + rchan->id = i; + rchan->reg_rt = ud->mmrs[MMR_RCHANRT] + UDMA_CH_1000(i); + } + + for (i = 0; i < ud->rflow_cnt; i++) { + struct udma_rflow *rflow = &ud->rflows[i]; + + rflow->id = i; + } + + for (i = 0; i < ch_count; i++) { + struct udma_chan *uc = &ud->channels[i]; + + uc->ud = ud; + uc->vc.desc_free = udma_desc_free; + uc->id = i; + uc->slave_thread_id = -1; + uc->tchan = NULL; + uc->rchan = NULL; + uc->dir = DMA_MEM_TO_MEM; + uc->name = devm_kasprintf(dev, GFP_KERNEL, "UDMA chan%d", i); + + vchan_init(&uc->vc, &ud->ddev); + /* Use custom vchan completion handling */ + tasklet_init(&uc->vc.task, udma_vchan_complete, + (unsigned long)&uc->vc); + init_completion(&uc->teardown_completed); + } + + ret = dma_async_device_register(&ud->ddev); + if (ret) { + dev_err(dev, "failed to register slave DMA engine: %d\n", ret); + return ret; + } + + platform_set_drvdata(pdev, ud); + + ret = of_dma_controller_register(dev->of_node, udma_of_xlate, ud); + if (ret) { + dev_err(dev, "failed to register of_dma controller\n"); + dma_async_device_unregister(&ud->ddev); + } + + return ret; +} + +static int udma_remove(struct platform_device *pdev) +{ + struct udma_dev *ud = platform_get_drvdata(pdev); + + of_dma_controller_free(pdev->dev.of_node); + dma_async_device_unregister(&ud->ddev); + + /* Make sure that we did proper cleanup */ + cancel_work_sync(&ud->purge_work); + udma_purge_desc_work(&ud->purge_work); + + pm_runtime_put_sync(&pdev->dev); + pm_runtime_disable(&pdev->dev); + + return 0; +} + +static struct platform_driver udma_driver = { + .driver = { + .name = "ti-udma", + .of_match_table = udma_of_match, + }, + .probe = udma_probe, + .remove = udma_remove, +}; + +module_platform_driver(udma_driver); + +MODULE_ALIAS("platform:ti-udma"); +MODULE_DESCRIPTION("TI K3 DMA driver for CPPI 5.0 compliant devices"); +MODULE_AUTHOR("Peter Ujfalusi "); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h new file mode 100644 index 000000000000..787b9feefe5d --- /dev/null +++ b/drivers/dma/ti/k3-udma.h @@ -0,0 +1,132 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com + */ + +#ifndef K3_UDMA_H_ +#define K3_UDMA_H_ + +#define UDMA_PSIL_DST_THREAD_ID_OFFSET 0x8000 + +/* Global registers */ +#define UDMA_REV_REG 0x0 +#define UDMA_PERF_CTL_REG 0x4 +#define UDMA_EMU_CTL_REG 0x8 +#define UDMA_PSIL_TO_REG 0x10 +#define UDMA_UTC_CTL_REG 0x1c +#define UDMA_CAP_REG(i) (0x20 + (i * 4)) +#define UDMA_RX_FLOW_ID_FW_OES_REG 0x80 +#define UDMA_RX_FLOW_ID_FW_STATUS_REG 0x88 + +/* TX chan RT regs */ +#define UDMA_TCHAN_RT_CTL_REG 0x0 +#define UDMA_TCHAN_RT_SWTRIG_REG 0x8 +#define UDMA_TCHAN_RT_STDATA_REG 0x80 + +#define UDMA_TCHAN_RT_PEERn_REG(i) (0x200 + (i * 0x4)) +#define UDMA_TCHAN_RT_PEER_STATIC_TR_XY_REG \ + UDMA_TCHAN_RT_PEERn_REG(0) /* PSI-L: 0x400 */ +#define UDMA_TCHAN_RT_PEER_STATIC_TR_Z_REG \ + UDMA_TCHAN_RT_PEERn_REG(1) /* PSI-L: 0x401 */ +#define UDMA_TCHAN_RT_PEER_BCNT_REG \ + UDMA_TCHAN_RT_PEERn_REG(4) /* PSI-L: 0x404 */ +#define UDMA_TCHAN_RT_PEER_RT_EN_REG \ + UDMA_TCHAN_RT_PEERn_REG(8) /* PSI-L: 0x408 */ + +#define UDMA_TCHAN_RT_PCNT_REG 0x400 +#define UDMA_TCHAN_RT_BCNT_REG 0x408 +#define UDMA_TCHAN_RT_SBCNT_REG 0x410 + +/* RX chan RT regs */ +#define UDMA_RCHAN_RT_CTL_REG 0x0 +#define UDMA_RCHAN_RT_SWTRIG_REG 0x8 +#define UDMA_RCHAN_RT_STDATA_REG 0x80 + +#define UDMA_RCHAN_RT_PEERn_REG(i) (0x200 + (i * 0x4)) +#define UDMA_RCHAN_RT_PEER_STATIC_TR_XY_REG \ + UDMA_RCHAN_RT_PEERn_REG(0) /* PSI-L: 0x400 */ +#define UDMA_RCHAN_RT_PEER_STATIC_TR_Z_REG \ + UDMA_RCHAN_RT_PEERn_REG(1) /* PSI-L: 0x401 */ +#define UDMA_RCHAN_RT_PEER_BCNT_REG \ + UDMA_RCHAN_RT_PEERn_REG(4) /* PSI-L: 0x404 */ +#define UDMA_RCHAN_RT_PEER_RT_EN_REG \ + UDMA_RCHAN_RT_PEERn_REG(8) /* PSI-L: 0x408 */ + +#define UDMA_RCHAN_RT_PCNT_REG 0x400 +#define UDMA_RCHAN_RT_BCNT_REG 0x408 +#define UDMA_RCHAN_RT_SBCNT_REG 0x410 + +/* UDMA_TCHAN_RT_CTL_REG/UDMA_RCHAN_RT_CTL_REG */ +#define UDMA_CHAN_RT_CTL_EN BIT(31) +#define UDMA_CHAN_RT_CTL_TDOWN BIT(30) +#define UDMA_CHAN_RT_CTL_PAUSE BIT(29) +#define UDMA_CHAN_RT_CTL_FTDOWN BIT(28) +#define UDMA_CHAN_RT_CTL_ERROR BIT(0) + +/* UDMA_TCHAN_RT_PEER_RT_EN_REG/UDMA_RCHAN_RT_PEER_RT_EN_REG (PSI-L: 0x408) */ +#define UDMA_PEER_RT_EN_ENABLE BIT(31) +#define UDMA_PEER_RT_EN_TEARDOWN BIT(30) +#define UDMA_PEER_RT_EN_PAUSE BIT(29) +#define UDMA_PEER_RT_EN_FLUSH BIT(28) +#define UDMA_PEER_RT_EN_IDLE BIT(1) + +/* + * UDMA_TCHAN_RT_PEER_STATIC_TR_XY_REG / + * UDMA_RCHAN_RT_PEER_STATIC_TR_XY_REG + */ +#define PDMA_STATIC_TR_X_MASK GENMASK(26, 24) +#define PDMA_STATIC_TR_X_SHIFT (24) +#define PDMA_STATIC_TR_Y_MASK GENMASK(11, 0) +#define PDMA_STATIC_TR_Y_SHIFT (0) + +#define PDMA_STATIC_TR_Y(x) \ + (((x) << PDMA_STATIC_TR_Y_SHIFT) & PDMA_STATIC_TR_Y_MASK) +#define PDMA_STATIC_TR_X(x) \ + (((x) << PDMA_STATIC_TR_X_SHIFT) & PDMA_STATIC_TR_X_MASK) + +/* + * UDMA_TCHAN_RT_PEER_STATIC_TR_Z_REG / + * UDMA_RCHAN_RT_PEER_STATIC_TR_Z_REG + */ +#define PDMA_STATIC_TR_Z_MASK GENMASK(11, 0) +#define PDMA_STATIC_TR_Z_SHIFT (0) +#define PDMA_STATIC_TR_Z(x) \ + (((x) << PDMA_STATIC_TR_Z_SHIFT) & PDMA_STATIC_TR_Z_MASK) + +/* Private API to UDMA via k3-udma-private */ +#include +#include +#include + +struct udma_dev; +struct udma_tchan; +struct udma_rchan; +struct udma_rflow; + +enum udma_rm_range { + RM_RANGE_TCHAN = 0, + RM_RANGE_RCHAN, + RM_RANGE_RFLOW, + RM_RANGE_LAST, +}; + +/* Channel Throughput Levels */ +enum udma_tp_level { + UDMA_TP_NORMAL = 0, + UDMA_TP_HIGH = 1, + UDMA_TP_LAST, +}; + +struct udma_tisci_rm { + const struct ti_sci_handle *tisci; + const struct ti_sci_rm_udmap_ops *tisci_udmap_ops; + u32 tisci_dev_id; + + /* tisci information for PSI-L thread pairing/unpairing */ + const struct ti_sci_rm_psil_ops *tisci_psil_ops; + u32 tisci_navss_dev_id; + + struct ti_sci_resource *rm_ranges[RM_RANGE_LAST]; +}; + +#endif /* K3_UDMA_H_ */ diff --git a/include/dt-bindings/dma/k3-udma.h b/include/dt-bindings/dma/k3-udma.h new file mode 100644 index 000000000000..89ba6a9d4a8f --- /dev/null +++ b/include/dt-bindings/dma/k3-udma.h @@ -0,0 +1,26 @@ +#ifndef __DT_TI_UDMA_H +#define __DT_TI_UDMA_H + +#define UDMA_TR_MODE 0 +#define UDMA_PKT_MODE 1 + +#define UDMA_DIR_TX 0 +#define UDMA_DIR_RX 1 + +#define PSIL_STATIC_TR_NONE 0 +#define PSIL_STATIC_TR_XY 1 +#define PSIL_STATIC_TR_MCAN 2 + +#define UDMA_PDMA_TR_XY(id) \ + ti,psil-config##id { \ + linux,udma-mode = ; \ + statictr-type = ; \ + } + +#define UDMA_PDMA_PKT_XY(id) \ + ti,psil-config##id { \ + linux,udma-mode = ; \ + statictr-type = ; \ + } + +#endif /* __DT_TI_UDMA_H */ From patchwork Mon Sep 24 13:00:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Ujfalusi X-Patchwork-Id: 10612471 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 77BEC157B for ; Mon, 24 Sep 2018 12:59:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6357529C95 for ; Mon, 24 Sep 2018 12:59:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5779F29CCF; Mon, 24 Sep 2018 12:59:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 189F729C95 for ; Mon, 24 Sep 2018 12:59:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730121AbeIXTBy (ORCPT ); Mon, 24 Sep 2018 15:01:54 -0400 Received: from fllv0015.ext.ti.com ([198.47.19.141]:55644 "EHLO fllv0015.ext.ti.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728863AbeIXTBx (ORCPT ); Mon, 24 Sep 2018 15:01:53 -0400 Received: from dlelxv90.itg.ti.com ([172.17.2.17]) by fllv0015.ext.ti.com (8.15.2/8.15.2) with ESMTP id w8OCxkgV004472; Mon, 24 Sep 2018 07:59:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ti.com; s=ti-com-17Q1; t=1537793986; bh=geYkqJsvJLuIhMzJmZfoai4R9cOeyxXBw6ZFK2jfMWw=; h=From:To:CC:Subject:Date:In-Reply-To:References; b=MBTSVDmlua0R0ULbplifyAv9J0AYnWjMLGjunZzHjxk440ZOUmkcT3sVPPsPD2sB1 wuYvZ3gEkM5UTrJ3fJum9ijFisHE8H3MRD5wLnx/TiTthBD3BEOi/IJQaH25rpgxrR HcZQL2WtnKVNUn/zfkKlU6zWkPeEbsVPIOjA2oY8= Received: from DLEE102.ent.ti.com (dlee102.ent.ti.com [157.170.170.32]) by dlelxv90.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxksK006575; Mon, 24 Sep 2018 07:59:46 -0500 Received: from DLEE100.ent.ti.com (157.170.170.30) by DLEE102.ent.ti.com (157.170.170.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1466.3; Mon, 24 Sep 2018 07:59:46 -0500 Received: from dflp32.itg.ti.com (10.64.6.15) by DLEE100.ent.ti.com (157.170.170.30) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_RSA_WITH_AES_256_CBC_SHA) id 15.1.1466.3 via Frontend Transport; Mon, 24 Sep 2018 07:59:46 -0500 Received: from feketebors.ti.com (ileax41-snat.itg.ti.com [10.172.224.153]) by dflp32.itg.ti.com (8.14.3/8.13.8) with ESMTP id w8OCxZsI012211; Mon, 24 Sep 2018 07:59:44 -0500 From: Peter Ujfalusi To: , CC: , Subject: [RFC 5/5] dmaengine: ti: k3-udma: Add glue layer for non DMAengine users Date: Mon, 24 Sep 2018 16:00:21 +0300 Message-ID: <20180924130021.20530-6-peter.ujfalusi@ti.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180924130021.20530-1-peter.ujfalusi@ti.com> References: <20180924130021.20530-1-peter.ujfalusi@ti.com> MIME-Version: 1.0 X-EXCLAIMER-MD-CONFIG: e1e8a2fd-e40a-4ac6-ac9b-f7e9cc9ee180 Sender: dmaengine-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: dmaengine@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Grygorii Strashko Certain users can not use right now the DMAengine API due to missing features in the core. Prime example is Networking. These users can use the glue layer interface to avoid misuse of DMAengine API and when the core gains the needed features they can be converted to use generic API. Signed-off-by: Grygorii Strashko Signed-off-by: Peter Ujfalusi --- drivers/dma/ti/Kconfig | 9 + drivers/dma/ti/Makefile | 1 + drivers/dma/ti/k3-navss-udma.c | 1133 +++++++++++++++++++++++++++++ drivers/dma/ti/k3-udma-private.c | 123 ++++ drivers/dma/ti/k3-udma.c | 3 + drivers/dma/ti/k3-udma.h | 30 + include/linux/dma/k3-navss-udma.h | 152 ++++ 7 files changed, 1451 insertions(+) create mode 100644 drivers/dma/ti/k3-navss-udma.c create mode 100644 drivers/dma/ti/k3-udma-private.c create mode 100644 include/linux/dma/k3-navss-udma.h diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig index db73371de206..23df9e0a1597 100644 --- a/drivers/dma/ti/Kconfig +++ b/drivers/dma/ti/Kconfig @@ -45,5 +45,14 @@ config TI_K3_NAVSS_UDMA Enable support for the TI UDMA (Unified DMA) controller. This DMA engine is used in AM65x. +config TI_K3_NAVSS_UDMA_GLUE_LAYER + tristate "Texas Instruments UDMA Glue layer for non DMAengine users" + depends on ARCH_K3 || COMPILE_TEST + depends on TI_K3_NAVSS_UDMA + default y + help + Say y here to support the K3 NAVSS DMA glue interface + If unsure, say N. + config TI_DMA_CROSSBAR bool diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile index 911f7c723979..c2533360c0de 100644 --- a/drivers/dma/ti/Makefile +++ b/drivers/dma/ti/Makefile @@ -3,4 +3,5 @@ obj-$(CONFIG_TI_CPPI41) += cppi41.o obj-$(CONFIG_TI_EDMA) += edma.o obj-$(CONFIG_DMA_OMAP) += omap-dma.o obj-$(CONFIG_TI_K3_NAVSS_UDMA) += k3-udma.o +obj-$(CONFIG_TI_K3_NAVSS_UDMA_GLUE_LAYER) += k3-navss-udma.o obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o diff --git a/drivers/dma/ti/k3-navss-udma.c b/drivers/dma/ti/k3-navss-udma.c new file mode 100644 index 000000000000..d1613febc86b --- /dev/null +++ b/drivers/dma/ti/k3-navss-udma.c @@ -0,0 +1,1133 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * K3 NAVSS DMA glue interface + * + * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "k3-udma.h" + +struct k3_nav_udmax_common { + struct device *dev; + struct udma_dev *udmax; + const struct udma_tisci_rm *tisci_rm; + struct k3_nav_ringacc *ringacc; + u32 src_thread; + u32 dst_thread; + + u32 hdesc_size; + bool epib; + u32 psdata_size; + u32 swdata_size; +}; + +struct k3_nav_udmax_tx_channel { + struct k3_nav_udmax_common common; + + struct udma_tchan *udma_tchanx; + int udma_tchan_id; + bool need_tisci_free; + + struct k3_nav_ring *ringtx; + struct k3_nav_ring *ringtxcq; + + bool psil_paired; + + unsigned int virq; + + atomic_t free_pkts; + bool tx_pause_on_err; + bool tx_filt_einfo; + bool tx_filt_pswords; + bool tx_supr_tdpkt; +}; + +/** + * k3_nav_udmax_rx_flow - UDMA RX flow context data + * + */ +struct k3_nav_udmax_rx_flow { + struct udma_rflow *udma_rflow; + int udma_rflow_id; + struct k3_nav_ring *ringrx; + struct k3_nav_ring *ringrxfdq; + + unsigned int virq; +}; + +struct k3_nav_udmax_rx_channel { + struct k3_nav_udmax_common common; + + struct udma_rchan *udma_rchanx; + int udma_rchan_id; + bool need_tisci_free; + + bool psil_paired; + + u32 swdata_size; + int flow_id_base; + + struct k3_nav_udmax_rx_flow *flows; + u32 flow_num; + u32 flows_ready; +}; + +#define K3_UDMAX_TDOWN_TIMEOUT_US 1000 + +static int of_k3_nav_udmax_parse(struct device_node *udmax_np, + struct k3_nav_udmax_common *common) +{ + common->ringacc = of_k3_nav_ringacc_get_by_phandle( + udmax_np, "ti,ringacc"); + if (IS_ERR(common->ringacc)) + return PTR_ERR(common->ringacc); + + common->udmax = of_xudma_dev_get(udmax_np, NULL); + if (IS_ERR(common->udmax)) + return PTR_ERR(common->udmax); + + common->tisci_rm = xudma_dev_get_tisci_rm(common->udmax); + + return 0; +} + +static int of_k3_nav_udmax_parse_chn( + struct device_node *chn_np, + const char *name, + struct k3_nav_udmax_common *common, + bool tx_chn) +{ + struct device_node *psil_cfg_node; + struct device_node *ch_cfg_node; + struct of_phandle_args dma_spec; + int index, ret = 0; + char prop[50]; + u32 val; + + if (unlikely(!name)) + return -EINVAL; + + index = of_property_match_string(chn_np, "dma-names", name); + if (index < 0) + return index; + + if (of_parse_phandle_with_args(chn_np, "dmas", "#dma-cells", index, + &dma_spec)) + return -ENOENT; + + if (tx_chn && dma_spec.args[2] != UDMA_DIR_TX) { + ret = -EINVAL; + goto out_put_spec; + } + + if (!tx_chn && dma_spec.args[2] != UDMA_DIR_RX) { + ret = -EINVAL; + goto out_put_spec; + } + + /* get psil cfg node */ + psil_cfg_node = of_find_node_by_phandle(dma_spec.args[0]); + if (!psil_cfg_node) { + ret = -ENOENT; + goto out_put_spec; + } + + snprintf(prop, sizeof(prop), "ti,psil-config%u", dma_spec.args[1]); + ch_cfg_node = of_find_node_by_name(psil_cfg_node, prop); + if (!ch_cfg_node) { + dev_err(common->dev, + "Channel %u configuration node is missing\n", + dma_spec.args[1]); + goto out_put_psil_cfg; + } + + common->epib = of_property_read_bool(ch_cfg_node, "ti,needs-epib"); + + if (!of_property_read_u32(ch_cfg_node, "ti,psd-size", &val)) + common->psdata_size = val; + + ret = of_property_read_u32(psil_cfg_node, "ti,psil-base", &val); + if (ret) { + dev_err(common->dev, "ti,psil-base is missing %d\n", ret); + goto out_ch_cfg; + } + + if (tx_chn) + common->dst_thread = val + dma_spec.args[1]; + else + common->src_thread = val + dma_spec.args[1]; + ret = of_k3_nav_udmax_parse(dma_spec.np, common); + +out_ch_cfg: + of_node_put(ch_cfg_node); +out_put_psil_cfg: + of_node_put(psil_cfg_node); +out_put_spec: + of_node_put(dma_spec.np); + return ret; +}; + +static void k3_nav_udmax_dump_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn) +{ + struct device *dev = tx_chn->common.dev; + + dev_dbg(dev, "dump_tx_chn:\n" + "udma_tchan_id: %d\n" + "src_thread: %08x\n" + "dst_thread: %08x\n", + tx_chn->udma_tchan_id, + tx_chn->common.src_thread, + tx_chn->common.dst_thread); +} + +static void k3_nav_udmax_dump_tx_rt_chn(struct k3_nav_udmax_tx_channel *chn, + char *mark) +{ + struct device *dev = chn->common.dev; + + dev_dbg(dev, "=== dump ===> %s\n", mark); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_CTL_REG, + xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_PEER_RT_EN_REG, + xudma_tchanrt_read(chn->udma_tchanx, + UDMA_TCHAN_RT_PEER_RT_EN_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_PCNT_REG, + xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_PCNT_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_BCNT_REG, + xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_BCNT_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_SBCNT_REG, + xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_SBCNT_REG)); +} + +static int k3_nav_udmax_cfg_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn) +{ + const struct udma_tisci_rm *tisci_rm = tx_chn->common.tisci_rm; + struct ti_sci_msg_rm_udmap_tx_ch_cfg req; + + memset(&req, 0, sizeof(req)); + + req.valid_params = TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_EINFO_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_PSWORDS_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_SUPR_TDPKT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID; + req.nav_id = tisci_rm->tisci_dev_id; + req.index = tx_chn->udma_tchan_id; + if (tx_chn->tx_pause_on_err) + req.tx_pause_on_err = 1; + if (tx_chn->tx_filt_einfo) + req.tx_filt_einfo = 1; + if (tx_chn->tx_filt_pswords) + req.tx_filt_pswords = 1; + req.tx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR; + if (tx_chn->tx_supr_tdpkt) + req.tx_supr_tdpkt = 1; + req.tx_fetch_size = tx_chn->common.hdesc_size >> 2; + req.txcq_qnum = k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq); + + return tisci_rm->tisci_udmap_ops->tx_ch_cfg(tisci_rm->tisci, &req); +} + +struct k3_nav_udmax_tx_channel *k3_nav_udmax_request_tx_chn( + struct device *dev, + const char *name, + struct k3_nav_udmax_tx_channel_cfg *cfg) +{ + struct k3_nav_udmax_tx_channel *tx_chn; + int ret; + + tx_chn = devm_kzalloc(dev, sizeof(*tx_chn), GFP_KERNEL); + if (!tx_chn) + return ERR_PTR(-ENOMEM); + + tx_chn->common.dev = dev; + tx_chn->common.swdata_size = cfg->swdata_size; + tx_chn->tx_pause_on_err = cfg->tx_pause_on_err; + tx_chn->tx_filt_einfo = cfg->tx_filt_einfo; + tx_chn->tx_filt_pswords = cfg->tx_filt_pswords; + tx_chn->tx_supr_tdpkt = cfg->tx_supr_tdpkt; + + /* parse of udmap channel */ + ret = of_k3_nav_udmax_parse_chn(dev->of_node, name, + &tx_chn->common, true); + if (ret) + goto err; + + tx_chn->common.hdesc_size = knav_udmap_hdesc_calc_size( + tx_chn->common.epib, + tx_chn->common.psdata_size, + tx_chn->common.swdata_size); + + /* request and cfg UDMAP TX channel */ + tx_chn->udma_tchanx = xudma_tchan_get(tx_chn->common.udmax, -1); + if (IS_ERR(tx_chn->udma_tchanx)) { + ret = PTR_ERR(tx_chn->udma_tchanx); + dev_err(dev, "UDMAX tchanx get err %d\n", ret); + goto err; + } + tx_chn->udma_tchan_id = xudma_tchan_get_id(tx_chn->udma_tchanx); + + atomic_set(&tx_chn->free_pkts, cfg->txcq_cfg.size); + + /* request and cfg rings */ + tx_chn->ringtx = k3_nav_ringacc_request_ring(tx_chn->common.ringacc, + tx_chn->udma_tchan_id, 0); + if (!tx_chn->ringtx) { + ret = -ENODEV; + dev_err(dev, "Failed to get TX ring %u\n", + tx_chn->udma_tchan_id); + goto err; + } + + tx_chn->ringtxcq = k3_nav_ringacc_request_ring(tx_chn->common.ringacc, + -1, 0); + if (!tx_chn->ringtxcq) { + ret = -ENODEV; + dev_err(dev, "Failed to get TXCQ ring\n"); + goto err; + } + + ret = k3_nav_ringacc_ring_cfg(tx_chn->ringtx, &cfg->tx_cfg); + if (ret) { + dev_err(dev, "Failed to cfg ringtx %d\n", ret); + goto err; + } + + ret = k3_nav_ringacc_ring_cfg(tx_chn->ringtxcq, &cfg->txcq_cfg); + if (ret) { + dev_err(dev, "Failed to cfg ringtx %d\n", ret); + goto err; + } + + /* request and cfg psi-l */ + tx_chn->common.src_thread = + xudma_dev_get_psil_base(tx_chn->common.udmax) + + tx_chn->udma_tchan_id; + + tx_chn->need_tisci_free = false; + ret = k3_nav_udmax_cfg_tx_chn(tx_chn); + if (ret) { + dev_err(dev, "Failed to cfg tchan %d\n", ret); + goto err; + } + + tx_chn->need_tisci_free = true; + + ret = xudma_navss_psil_pair(tx_chn->common.udmax, + tx_chn->common.src_thread, + tx_chn->common.dst_thread); + if (ret) { + dev_err(dev, "PSI-L request err %d\n", ret); + goto err; + } + + tx_chn->psil_paired = true; + + k3_nav_udmax_dump_tx_chn(tx_chn); + + return tx_chn; + +err: + k3_nav_udmax_release_tx_chn(tx_chn); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_request_tx_chn); + +void k3_nav_udmax_release_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn) +{ + if (tx_chn->psil_paired) { + xudma_navss_psil_unpair(tx_chn->common.udmax, + tx_chn->common.src_thread, + tx_chn->common.dst_thread); + tx_chn->psil_paired = false; + } + + if (!IS_ERR_OR_NULL(tx_chn->common.udmax)) { + if (tx_chn->need_tisci_free) { + tx_chn->need_tisci_free = false; + } + + if (!IS_ERR_OR_NULL(tx_chn->udma_tchanx)) + xudma_tchan_put(tx_chn->common.udmax, + tx_chn->udma_tchanx); + + xudma_dev_put(tx_chn->common.udmax); + } + + if (tx_chn->ringtxcq) + k3_nav_ringacc_ring_free(tx_chn->ringtxcq); + + if (tx_chn->ringtx) + k3_nav_ringacc_ring_free(tx_chn->ringtx); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_release_tx_chn); + +int k3_nav_udmax_push_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn, + struct knav_udmap_host_desc_t *desc_tx, + dma_addr_t desc_dma) +{ + u32 ringtxcq_id; + + if (!atomic_add_unless(&tx_chn->free_pkts, -1, 0)) + return -ENOMEM; + + ringtxcq_id = k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq); + knav_udmap_desc_set_retpolicy(&desc_tx->hdr, 0, + ringtxcq_id); + + return k3_nav_ringacc_ring_push(tx_chn->ringtx, &desc_dma); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_push_tx_chn); + +int k3_nav_udmax_pop_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn, + dma_addr_t *desc_dma) +{ + int ret; + + ret = k3_nav_ringacc_ring_pop(tx_chn->ringtxcq, desc_dma); + if (!ret) + atomic_inc(&tx_chn->free_pkts); + + return ret; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_pop_tx_chn); + +int k3_nav_udmax_enable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn) +{ + u32 txrt_ctl; + + txrt_ctl = UDMA_PEER_RT_EN_ENABLE; + xudma_tchanrt_write(tx_chn->udma_tchanx, + UDMA_TCHAN_RT_PEER_RT_EN_REG, + txrt_ctl); + + txrt_ctl = xudma_tchanrt_read(tx_chn->udma_tchanx, + UDMA_TCHAN_RT_CTL_REG); + txrt_ctl |= UDMA_CHAN_RT_CTL_EN; + xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG, + txrt_ctl); + + k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn en"); + return 0; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_enable_tx_chn); + +void k3_nav_udmax_disable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn) +{ + k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn dis1"); + + xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG, 0); + + xudma_tchanrt_write(tx_chn->udma_tchanx, + UDMA_TCHAN_RT_PEER_RT_EN_REG, 0); + k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn dis2"); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_disable_tx_chn); + +void k3_nav_udmax_tdown_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn, + bool sync) +{ + int i = 0; + u32 val; + + k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn tdown1"); + + xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG, + UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_TDOWN); + + val = xudma_tchanrt_read(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG); + + while (sync && (val & UDMA_CHAN_RT_CTL_EN)) { + val = xudma_tchanrt_read(tx_chn->udma_tchanx, + UDMA_TCHAN_RT_CTL_REG); + udelay(1); + if (i > K3_UDMAX_TDOWN_TIMEOUT_US) { + dev_err(tx_chn->common.dev, "TX tdown timeout\n"); + break; + } + i++; + } + + val = xudma_tchanrt_read(tx_chn->udma_tchanx, + UDMA_TCHAN_RT_PEER_RT_EN_REG); + if (sync && (val & UDMA_PEER_RT_EN_ENABLE)) + dev_err(tx_chn->common.dev, "TX tdown peer not stopped\n"); + k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn tdown2"); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_tdown_tx_chn); + +void k3_nav_udmax_reset_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn, void *data, + void (*cleanup)(void *data, dma_addr_t desc_dma)) +{ + dma_addr_t desc_dma; + int occ_tx, i, ret; + + /* reset TXCQ as it is not input for udma - expected to be empty */ + if (tx_chn->ringtxcq) + k3_nav_ringacc_ring_reset(tx_chn->ringtxcq); + + /* + * TXQ reset need to be special way as it is input for udma and its + * state cached by udma, so: + * 1) save TXQ occ + * 2) clean up TXQ and call callback .cleanup() for each desc + * 3) reset TXQ in a special way + */ + occ_tx = k3_nav_ringacc_ring_get_occ(tx_chn->ringtx); + dev_dbg(tx_chn->common.dev, "TX reset occ_tx %u\n", occ_tx); + + for (i = 0; i < occ_tx; i++) { + ret = k3_nav_ringacc_ring_pop(tx_chn->ringtx, &desc_dma); + if (ret) { + dev_err(tx_chn->common.dev, "TX reset pop %d\n", ret); + break; + } + cleanup(data, desc_dma); + } + + k3_nav_ringacc_ring_reset_dma(tx_chn->ringtx, occ_tx); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_reset_tx_chn); + +u32 k3_nav_udmax_tx_get_hdesc_size(struct k3_nav_udmax_tx_channel *tx_chn) +{ + return tx_chn->common.hdesc_size; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_get_hdesc_size); + +u32 k3_nav_udmax_tx_get_txcq_id(struct k3_nav_udmax_tx_channel *tx_chn) +{ + return k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_get_txcq_id); + +int k3_nav_udmax_tx_get_irq(struct k3_nav_udmax_tx_channel *tx_chn, + unsigned int *irq, u32 flags, bool share, + struct k3_nav_udmax_tx_channel *tx_chn_share) +{ + unsigned int virq = 0; + + if (share && tx_chn_share) + virq = tx_chn_share->virq; + + tx_chn->virq = ti_sci_inta_register_event( + tx_chn->common.dev, + k3_nav_ringacc_get_tisci_dev_id(tx_chn->ringtxcq), + k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq), + virq, flags); + if (tx_chn->virq <= 0) + return -ENODEV; + + *irq = tx_chn->virq; + + return 0; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_get_irq); + +void k3_nav_udmax_tx_put_irq(struct k3_nav_udmax_tx_channel *tx_chn) +{ + if (tx_chn->virq <= 0) + return; + + ti_sci_inta_unregister_event( + tx_chn->common.dev, + k3_nav_ringacc_get_tisci_dev_id(tx_chn->ringtxcq), + k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq), + tx_chn->virq); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_put_irq); + +static int k3_nav_udmax_cfg_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn) +{ + const struct udma_tisci_rm *tisci_rm = rx_chn->common.tisci_rm; + struct ti_sci_msg_rm_udmap_rx_ch_cfg req; + int ret; + + memset(&req, 0, sizeof(req)); + + req.valid_params = TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID; + + req.nav_id = tisci_rm->tisci_dev_id; + req.index = rx_chn->udma_rchan_id; + req.rx_fetch_size = rx_chn->common.hdesc_size >> 2; + /* + * TODO: we can't support rxcq_qnum/RCHAN[a]_RCQ cfg with current sysfw + * and udmax impl, so just configure it to invalid value. + * req.rxcq_qnum = k3_nav_ringacc_get_ring_id(rx_chn->flows[0].ringrx); + */ + req.rxcq_qnum = 0xFFFF; + if (rx_chn->flow_num && rx_chn->flow_id_base != rx_chn->udma_rchan_id) { + /* Default flow + extra ones */ + req.flowid_start = rx_chn->flow_id_base; + req.flowid_cnt = rx_chn->flow_num; + req.valid_params |= + TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_FLOWID_START_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_FLOWID_CNT_VALID; + } + req.rx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR; + + ret = tisci_rm->tisci_udmap_ops->rx_ch_cfg(tisci_rm->tisci, &req); + if (ret) + dev_err(rx_chn->common.dev, "rchan%d cfg failed %d\n", + rx_chn->udma_rchan_id, ret); + + return ret; +} + +static void k3_nav_udmax_release_rx_flow( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num) +{ + struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num]; + + if (IS_ERR_OR_NULL(flow->udma_rflow)) + return; + + if (flow->ringrxfdq) + k3_nav_ringacc_ring_free(flow->ringrxfdq); + + if (flow->ringrx) + k3_nav_ringacc_ring_free(flow->ringrx); + + xudma_rflow_put(rx_chn->common.udmax, flow->udma_rflow); + rx_chn->flows_ready--; +} + +static int k3_nav_udmax_cfg_rx_flow( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_idx, + struct k3_nav_udmax_rx_flow_cfg *flow_cfg) +{ + struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_idx]; + const struct udma_tisci_rm *tisci_rm = rx_chn->common.tisci_rm; + struct device *dev = rx_chn->common.dev; + struct ti_sci_msg_rm_udmap_flow_cfg req; + int rx_ring_id; + int rx_ringfdq_id; + int ret = 0; + + flow->udma_rflow = xudma_rflow_get(rx_chn->common.udmax, + flow->udma_rflow_id); + if (IS_ERR(flow->udma_rflow)) { + ret = PTR_ERR(flow->udma_rflow); + dev_err(dev, "UDMAX rflow get err %d\n", ret); + goto err; + } + + if (flow->udma_rflow_id != xudma_rflow_get_id(flow->udma_rflow)) { + xudma_rflow_put(rx_chn->common.udmax, flow->udma_rflow); + return -ENODEV; + } + + /* request and cfg rings */ + flow->ringrx = k3_nav_ringacc_request_ring(rx_chn->common.ringacc, + flow_cfg->ring_rxq_id, 0); + if (!flow->ringrx) { + ret = -ENODEV; + dev_err(dev, "Failed to get RX ring\n"); + goto err; + } + + flow->ringrxfdq = k3_nav_ringacc_request_ring(rx_chn->common.ringacc, + flow_cfg->ring_rxfdq0_id, + 0); + if (!flow->ringrxfdq) { + ret = -ENODEV; + dev_err(dev, "Failed to get RXFDQ ring\n"); + goto err; + } + + ret = k3_nav_ringacc_ring_cfg(flow->ringrx, &flow_cfg->rx_cfg); + if (ret) { + dev_err(dev, "Failed to cfg ringrx %d\n", ret); + goto err; + } + + ret = k3_nav_ringacc_ring_cfg(flow->ringrxfdq, &flow_cfg->rxfdq_cfg); + if (ret) { + dev_err(dev, "Failed to cfg ringrxfdq %d\n", ret); + goto err; + } + + rx_ring_id = k3_nav_ringacc_get_ring_id(flow->ringrx); + rx_ringfdq_id = k3_nav_ringacc_get_ring_id(flow->ringrxfdq); + + memset(&req, 0, sizeof(req)); + + req.valid_params = + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_EINFO_PRESENT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_PSINFO_PRESENT_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_ERROR_HANDLING_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DESC_TYPE_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_HI_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_LO_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_HI_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_LO_SEL_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ0_SZ0_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ1_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ2_QNUM_VALID | + TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ3_QNUM_VALID; + req.nav_id = tisci_rm->tisci_dev_id; + req.flow_index = flow->udma_rflow_id; + if (rx_chn->common.epib) + req.rx_einfo_present = 1; + if (rx_chn->common.psdata_size) + req.rx_psinfo_present = 1; + if (flow_cfg->rx_error_handling) + req.rx_error_handling = 1; + req.rx_desc_type = 0; + req.rx_dest_qnum = rx_ring_id; + req.rx_src_tag_hi_sel = 0; + req.rx_src_tag_lo_sel = flow_cfg->src_tag_lo_sel; + req.rx_dest_tag_hi_sel = 0; + req.rx_dest_tag_lo_sel = 0; + req.rx_fdq0_sz0_qnum = rx_ringfdq_id; + req.rx_fdq1_qnum = rx_ringfdq_id; + req.rx_fdq2_qnum = rx_ringfdq_id; + req.rx_fdq3_qnum = rx_ringfdq_id; + + ret = tisci_rm->tisci_udmap_ops->rx_flow_cfg(tisci_rm->tisci, &req); + if (ret) { + dev_err(dev, "flow%d config failed: %d\n", flow->udma_rflow_id, + ret); + goto err; + } + + rx_chn->flows_ready++; + + return 0; +err: + k3_nav_udmax_release_rx_flow(rx_chn, flow_idx); + return ret; +} + +static void k3_nav_udmax_dump_rx_chn(struct k3_nav_udmax_rx_channel *chn) +{ + struct device *dev = chn->common.dev; + + dev_dbg(dev, "dump_rx_chn:\n" + "udma_rchan_id: %d\n" + "src_thread: %08x\n" + "dst_thread: %08x\n" + "epib: %d\n" + "hdesc_size: %u\n" + "psdata_size: %u\n" + "swdata_size: %u\n" + "flow_id_base: %d\n" + "flow_num: %d\n", + chn->udma_rchan_id, + chn->common.src_thread, + chn->common.dst_thread, + chn->common.epib, + chn->common.hdesc_size, + chn->common.psdata_size, + chn->common.swdata_size, + chn->flow_id_base, + chn->flow_num); +} + +static void k3_nav_udmax_dump_rx_rt_chn(struct k3_nav_udmax_rx_channel *chn, + char *mark) +{ + struct device *dev = chn->common.dev; + + dev_dbg(dev, "=== dump ===> %s\n", mark); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_CTL_REG, + xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_PEER_RT_EN_REG, + xudma_rchanrt_read(chn->udma_rchanx, + UDMA_RCHAN_RT_PEER_RT_EN_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_PCNT_REG, + xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_PCNT_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_BCNT_REG, + xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_BCNT_REG)); + dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_SBCNT_REG, + xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_SBCNT_REG)); +} + +struct k3_nav_udmax_rx_channel *k3_nav_udmax_request_rx_chn( + struct device *dev, + const char *name, + struct k3_nav_udmax_rx_channel_cfg *cfg) +{ + struct k3_nav_udmax_rx_channel *rx_chn; + int ret, i; + + if (cfg->flow_id_num <= 0) + return ERR_PTR(-EINVAL); + + if (cfg->flow_id_num != 1 && + (cfg->def_flow_cfg || cfg->flow_id_use_rxchan_id)) + return ERR_PTR(-EINVAL); + + rx_chn = devm_kzalloc(dev, sizeof(*rx_chn), GFP_KERNEL); + if (!rx_chn) + return ERR_PTR(-ENOMEM); + + rx_chn->common.dev = dev; + rx_chn->common.swdata_size = cfg->swdata_size; + + /* parse of udmap channel */ + ret = of_k3_nav_udmax_parse_chn(dev->of_node, name, + &rx_chn->common, false); + if (ret) + goto err; + + rx_chn->common.hdesc_size = knav_udmap_hdesc_calc_size( + rx_chn->common.epib, + rx_chn->common.psdata_size, + rx_chn->common.swdata_size); + + /* request and cfg UDMAP RX channel */ + rx_chn->udma_rchanx = xudma_rchan_get(rx_chn->common.udmax, -1); + if (IS_ERR(rx_chn->udma_rchanx)) { + ret = PTR_ERR(rx_chn->udma_rchanx); + dev_err(dev, "UDMAX rchanx get err %d\n", ret); + goto err; + } + rx_chn->udma_rchan_id = xudma_rchan_get_id(rx_chn->udma_rchanx); + + rx_chn->flow_num = cfg->flow_id_num; + rx_chn->flow_id_base = cfg->flow_id_base; + + /* Use RX channel id as flow id: target dev can't generate flow_id */ + if (cfg->flow_id_use_rxchan_id) + rx_chn->flow_id_base = rx_chn->udma_rchan_id; + + rx_chn->flows = devm_kcalloc(dev, rx_chn->flow_num, + sizeof(*rx_chn->flows), GFP_KERNEL); + if (!rx_chn->flows) { + ret = -ENOMEM; + goto err; + } + + /* Reserve range of RX flows */ + if (!cfg->flow_id_use_rxchan_id) { + ret = xudma_reserve_rflow_range(rx_chn->common.udmax, + rx_chn->flow_id_base, + rx_chn->flow_num); + if (ret < 0) { + dev_err(dev, "UDMAX reserve_rflow get err %d\n", ret); + goto err; + } + rx_chn->flow_id_base = ret; + } + + for (i = 0; i < rx_chn->flow_num; i++) + rx_chn->flows[i].udma_rflow_id = rx_chn->flow_id_base + i; + + /* request and cfg psi-l */ + rx_chn->common.dst_thread = + xudma_dev_get_psil_base(rx_chn->common.udmax) + + rx_chn->udma_rchan_id; + + rx_chn->need_tisci_free = false; + ret = k3_nav_udmax_cfg_rx_chn(rx_chn); + if (ret) { + dev_err(dev, "Failed to cfg rchan %d\n", ret); + goto err; + } + rx_chn->need_tisci_free = true; + + /* init default RX flow only if flow_num = 1 */ + if (cfg->def_flow_cfg) { + ret = k3_nav_udmax_cfg_rx_flow(rx_chn, 0, cfg->def_flow_cfg); + if (ret) + goto err; + } + + if (!cfg->skip_psil) { + ret = xudma_navss_psil_pair(rx_chn->common.udmax, + rx_chn->common.src_thread, + rx_chn->common.dst_thread); + if (ret) { + dev_err(dev, "PSI-L request err %d\n", ret); + goto err; + } + + rx_chn->psil_paired = true; + } + + k3_nav_udmax_dump_rx_chn(rx_chn); + + return rx_chn; + +err: + k3_nav_udmax_release_rx_chn(rx_chn); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_request_rx_chn); + +void k3_nav_udmax_release_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn) +{ + int i; + + if (rx_chn->psil_paired) { + xudma_navss_psil_unpair(rx_chn->common.udmax, + rx_chn->common.src_thread, + rx_chn->common.dst_thread); + rx_chn->psil_paired = false; + } + + if (IS_ERR_OR_NULL(rx_chn->common.udmax)) + return; + + for (i = 0; i < rx_chn->flow_num; i++) + k3_nav_udmax_release_rx_flow(rx_chn, i); + + if (rx_chn->need_tisci_free) { + rx_chn->need_tisci_free = false; + } + + xudma_free_rflow_range(rx_chn->common.udmax, + rx_chn->flow_id_base, rx_chn->flow_num); + + if (!IS_ERR_OR_NULL(rx_chn->udma_rchanx)) + xudma_rchan_put(rx_chn->common.udmax, + rx_chn->udma_rchanx); + + xudma_dev_put(rx_chn->common.udmax); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_release_rx_chn); + +int k3_nav_udmax_rx_flow_init( + struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx, + struct k3_nav_udmax_rx_flow_cfg *flow_cfg) +{ + if (flow_idx >= rx_chn->flow_num) + return -EINVAL; + + return k3_nav_udmax_cfg_rx_flow(rx_chn, flow_idx, flow_cfg); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_flow_init); + +u32 k3_nav_udmax_rx_flow_get_fdq_id( + struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx) +{ + struct k3_nav_udmax_rx_flow *flow; + + if (flow_idx >= rx_chn->flow_num) + return -EINVAL; + + flow = &rx_chn->flows[flow_idx]; + + return k3_nav_ringacc_get_ring_id(flow->ringrxfdq); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_flow_get_fdq_id); + +u32 k3_nav_udmax_rx_get_flow_id_base( + struct k3_nav_udmax_rx_channel *rx_chn) +{ + return rx_chn->flow_id_base; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_get_flow_id_base); + +int k3_nav_udmax_enable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn) +{ + u32 rxrt_ctl; + + if (rx_chn->flows_ready < rx_chn->flow_num) + return -EINVAL; + + rxrt_ctl = xudma_rchanrt_read(rx_chn->udma_rchanx, + UDMA_RCHAN_RT_CTL_REG); + rxrt_ctl |= UDMA_CHAN_RT_CTL_EN; + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG, + rxrt_ctl); + + xudma_rchanrt_write(rx_chn->udma_rchanx, + UDMA_RCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE); + + k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt en"); + return 0; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_enable_rx_chn); + +void k3_nav_udmax_disable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn) +{ + k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt dis1"); + + xudma_rchanrt_write(rx_chn->udma_rchanx, + UDMA_RCHAN_RT_PEER_RT_EN_REG, + 0); + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG, 0); + + k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt dis2"); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_disable_rx_chn); + +void k3_nav_udmax_tdown_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn, + bool sync) +{ + int i = 0; + u32 val; + + k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt tdown1"); + + xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_RCHAN_RT_PEER_RT_EN_REG, + UDMA_PEER_RT_EN_ENABLE | UDMA_PEER_RT_EN_TEARDOWN); + + val = xudma_rchanrt_read(rx_chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG); + + while (sync && (val & UDMA_CHAN_RT_CTL_EN)) { + val = xudma_rchanrt_read(rx_chn->udma_rchanx, + UDMA_RCHAN_RT_CTL_REG); + udelay(1); + if (i > K3_UDMAX_TDOWN_TIMEOUT_US) { + dev_err(rx_chn->common.dev, "RX tdown timeout\n"); + break; + } + i++; + } + + val = xudma_rchanrt_read(rx_chn->udma_rchanx, + UDMA_RCHAN_RT_PEER_RT_EN_REG); + if (sync && (val & UDMA_PEER_RT_EN_ENABLE)) + dev_err(rx_chn->common.dev, "TX tdown peer not stopped\n"); + k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt tdown2"); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_tdown_rx_chn); + +void k3_nav_udmax_reset_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, void *data, + void (*cleanup)(void *data, dma_addr_t desc_dma), + bool skip_fdq) +{ + struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num]; + struct device *dev = rx_chn->common.dev; + dma_addr_t desc_dma; + int occ_rx, i, ret; + + /* reset RXCQ as it is not input for udma - expected to be empty */ + if (flow->ringrx) + k3_nav_ringacc_ring_reset(flow->ringrx); + + /* Skip RX FDQ in case one FDQ is used for the set of flows */ + if (skip_fdq) + return; + + /* + * RX FDQ reset need to be special way as it is input for udma and its + * state cached by udma, so: + * 1) save RX FDQ occ + * 2) clean up RX FDQ and call callback .cleanup() for each desc + * 3) reset RX FDQ in a special way + */ + occ_rx = k3_nav_ringacc_ring_get_occ(flow->ringrxfdq); + dev_dbg(dev, "RX reset flow %u occ_tx %u\n", flow_num, occ_rx); + + for (i = 0; i < occ_rx; i++) { + ret = k3_nav_ringacc_ring_pop(flow->ringrxfdq, &desc_dma); + if (ret) { + dev_err(dev, "RX reset pop %d\n", ret); + break; + } + cleanup(data, desc_dma); + } + + k3_nav_ringacc_ring_reset_dma(flow->ringrxfdq, occ_rx); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_reset_rx_chn); + +int k3_nav_udmax_push_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, + struct knav_udmap_host_desc_t *desc_rx, + dma_addr_t desc_dma) +{ + struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num]; + + return k3_nav_ringacc_ring_push(flow->ringrxfdq, &desc_dma); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_push_rx_chn); + +int k3_nav_udmax_pop_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, + dma_addr_t *desc_dma) +{ + struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num]; + + return k3_nav_ringacc_ring_pop(flow->ringrx, desc_dma); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_pop_rx_chn); + +int k3_nav_udmax_rx_get_irq(struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, + unsigned int *irq, u32 flags, bool share, + u32 flow_num_share) +{ + struct k3_nav_udmax_rx_flow *flow, *flow_share; + unsigned int virq = 0; + + if (flow_num >= rx_chn->flow_num || + (flow_num_share != -1 && flow_num_share >= rx_chn->flow_num)) + return -EINVAL; + + flow = &rx_chn->flows[flow_num]; + + if (share && flow_num_share != -1) { + flow_share = &rx_chn->flows[flow_num_share]; + virq = flow_share->virq; + } + + flow->virq = ti_sci_inta_register_event( + rx_chn->common.dev, + k3_nav_ringacc_get_tisci_dev_id(flow->ringrx), + k3_nav_ringacc_get_ring_id(flow->ringrx), + virq, flags); + if (flow->virq <= 0) + return -ENODEV; + + *irq = flow->virq; + + return 0; +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_get_irq); + +void k3_nav_udmax_rx_put_irq(struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num) +{ + struct k3_nav_udmax_rx_flow *flow; + + if (flow_num >= rx_chn->flow_num) + return; + + flow = &rx_chn->flows[flow_num]; + + if (flow->virq <= 0) + return; + + ti_sci_inta_unregister_event( + rx_chn->common.dev, + k3_nav_ringacc_get_tisci_dev_id(flow->ringrx), + k3_nav_ringacc_get_ring_id(flow->ringrx), flow->virq); +} +EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_put_irq); diff --git a/drivers/dma/ti/k3-udma-private.c b/drivers/dma/ti/k3-udma-private.c new file mode 100644 index 000000000000..21bae700f7a2 --- /dev/null +++ b/drivers/dma/ti/k3-udma-private.c @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com + * Author: Peter Ujfalusi + */ + +int xudma_navss_psil_pair(struct udma_dev *ud, u32 src_thread,u32 dst_thread) +{ + return navss_psil_pair(ud, src_thread, dst_thread); +} +EXPORT_SYMBOL(xudma_navss_psil_pair); + +int xudma_navss_psil_unpair(struct udma_dev *ud, u32 src_thread, u32 dst_thread) +{ + return navss_psil_unpair(ud, src_thread, dst_thread); +} +EXPORT_SYMBOL(xudma_navss_psil_unpair); + +struct udma_dev *of_xudma_dev_get(struct device_node *np, const char *property) +{ + struct device_node *udma_node = np; + struct platform_device *pdev; + struct udma_dev *ud; + + if (property) { + udma_node = of_parse_phandle(np, property, 0); + if (!udma_node) { + pr_err("UDMA node is not found\n"); + return ERR_PTR(-ENODEV); + } + } + + pdev = of_find_device_by_node(udma_node); + if (!pdev) { + pr_err("UDMA device not found\n"); + return ERR_PTR(-EPROBE_DEFER); + } + + of_node_put(udma_node); + + ud = platform_get_drvdata(pdev); + if (!ud) { + pr_err("UDMA has not been proped\n"); + return ERR_PTR(-EPROBE_DEFER); + } + + pm_runtime_get_sync(&pdev->dev); + + return ud; +} +EXPORT_SYMBOL(of_xudma_dev_get); + +void xudma_dev_put(struct udma_dev *ud) +{ + pm_runtime_put_sync(ud->ddev.dev); +} +EXPORT_SYMBOL(xudma_dev_put); + +u32 xudma_dev_get_psil_base(struct udma_dev *ud) +{ + return ud->psil_base; +} +EXPORT_SYMBOL(xudma_dev_get_psil_base); + +struct udma_tisci_rm *xudma_dev_get_tisci_rm(struct udma_dev *ud) +{ + return &ud->tisci_rm; +} +EXPORT_SYMBOL(xudma_dev_get_tisci_rm); + +int xudma_reserve_rflow_range(struct udma_dev *ud, int from, int cnt) +{ + return __udma_reserve_rflow_range(ud, from, cnt); +} +EXPORT_SYMBOL(xudma_reserve_rflow_range); + +int xudma_free_rflow_range(struct udma_dev *ud, int from, int cnt) +{ + return __udma_free_rflow_range(ud, from, cnt); +} +EXPORT_SYMBOL(xudma_free_rflow_range); + +#define XUDMA_GET_PUT_RESOURCE(res) \ +struct udma_##res *xudma_##res##_get(struct udma_dev *ud, int id) \ +{ \ + return __udma_reserve_##res(ud, false, id); \ +} \ +EXPORT_SYMBOL(xudma_##res##_get); \ + \ +void xudma_##res##_put(struct udma_dev *ud, struct udma_##res *p) \ +{ \ + clear_bit(p->id, ud->res##_map); \ +} \ +EXPORT_SYMBOL(xudma_##res##_put) +XUDMA_GET_PUT_RESOURCE(tchan); +XUDMA_GET_PUT_RESOURCE(rchan); +XUDMA_GET_PUT_RESOURCE(rflow); + +#define XUDMA_GET_RESOURCE_ID(res) \ +int xudma_##res##_get_id(struct udma_##res *p) \ +{ \ + return p->id; \ +} \ +EXPORT_SYMBOL(xudma_##res##_get_id) +XUDMA_GET_RESOURCE_ID(tchan); +XUDMA_GET_RESOURCE_ID(rchan); +XUDMA_GET_RESOURCE_ID(rflow); + +/* Exported register access functions */ +#define XUDMA_RT_IO_FUNCTIONS(res) \ +u32 xudma_##res##rt_read(struct udma_##res *p, int reg) \ +{ \ + return udma_##res##rt_read(p, reg); \ +} \ +EXPORT_SYMBOL(xudma_##res##rt_read); \ + \ +void xudma_##res##rt_write(struct udma_##res *p, int reg, u32 val) \ +{ \ + udma_##res##rt_write(p, reg, val); \ +} \ +EXPORT_SYMBOL(xudma_##res##rt_write) +XUDMA_RT_IO_FUNCTIONS(tchan); +XUDMA_RT_IO_FUNCTIONS(rchan); diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c index 2a2a2c8b1fc2..49d4c49b8139 100644 --- a/drivers/dma/ti/k3-udma.c +++ b/drivers/dma/ti/k3-udma.c @@ -3617,6 +3617,9 @@ static struct platform_driver udma_driver = { module_platform_driver(udma_driver); +/* Private interfaces to UDMA */ +#include "k3-udma-private.c" + MODULE_ALIAS("platform:ti-udma"); MODULE_DESCRIPTION("TI K3 DMA driver for CPPI 5.0 compliant devices"); MODULE_AUTHOR("Peter Ujfalusi "); diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h index 787b9feefe5d..2720d73ff3d3 100644 --- a/drivers/dma/ti/k3-udma.h +++ b/drivers/dma/ti/k3-udma.h @@ -129,4 +129,34 @@ struct udma_tisci_rm { struct ti_sci_resource *rm_ranges[RM_RANGE_LAST]; }; +/* Direct access to UDMA resources */ +int xudma_navss_psil_pair(struct udma_dev *ud, u32 src_thread,u32 dst_thread); +int xudma_navss_psil_unpair(struct udma_dev *ud, u32 src_thread, + u32 dst_thread); + +struct udma_dev *of_xudma_dev_get(struct device_node *np, const char *property); +void xudma_dev_put(struct udma_dev *ud); +u32 xudma_dev_get_psil_base(struct udma_dev *ud); +struct udma_tisci_rm *xudma_dev_get_tisci_rm(struct udma_dev *ud); + +int xudma_reserve_rflow_range(struct udma_dev *ud, int from, int cnt); +int xudma_free_rflow_range(struct udma_dev *ud, int from, int cnt); + +struct udma_tchan *xudma_tchan_get(struct udma_dev *ud, int id); +struct udma_rchan *xudma_rchan_get(struct udma_dev *ud, int id); +struct udma_rflow *xudma_rflow_get(struct udma_dev *ud, int id); + +void xudma_tchan_put(struct udma_dev *ud, struct udma_tchan *p); +void xudma_rchan_put(struct udma_dev *ud, struct udma_rchan *p); +void xudma_rflow_put(struct udma_dev *ud, struct udma_rflow *p); + +int xudma_tchan_get_id(struct udma_tchan *p); +int xudma_rchan_get_id(struct udma_rchan *p); +int xudma_rflow_get_id(struct udma_rflow *p); + +u32 xudma_tchanrt_read(struct udma_tchan *tchan, int reg); +void xudma_tchanrt_write(struct udma_tchan *tchan, int reg, u32 val); +u32 xudma_rchanrt_read(struct udma_rchan *rchan, int reg); +void xudma_rchanrt_write(struct udma_rchan *rchan, int reg, u32 val); + #endif /* K3_UDMA_H_ */ diff --git a/include/linux/dma/k3-navss-udma.h b/include/linux/dma/k3-navss-udma.h new file mode 100644 index 000000000000..64a8bcbb8703 --- /dev/null +++ b/include/linux/dma/k3-navss-udma.h @@ -0,0 +1,152 @@ +/* + * K3 NAVSS DMA glue interface + * + * Copyright (C) 2017 Texas Instruments Incorporated - http://www.ti.com + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * version 2 as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + */ + +#ifndef K3_NAVSS_UDMA_H_ +#define K3_NAVSS_UDMA_H_ + +#include + +struct k3_nav_udmax_tx_channel_cfg { + struct k3_nav_ring_cfg tx_cfg; + struct k3_nav_ring_cfg txcq_cfg; + + bool tx_pause_on_err; + bool tx_filt_einfo; + bool tx_filt_pswords; + bool tx_supr_tdpkt; + u32 swdata_size; +}; + +struct k3_nav_udmax_tx_channel; + +struct k3_nav_udmax_tx_channel *k3_nav_udmax_request_tx_chn( + struct device *dev, + const char *name, + struct k3_nav_udmax_tx_channel_cfg *cfg); + +void k3_nav_udmax_release_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn); +int k3_nav_udmax_push_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn, + struct knav_udmap_host_desc_t *desc_tx, + dma_addr_t desc_dma); +int k3_nav_udmax_pop_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn, + dma_addr_t *desc_dma); +int k3_nav_udmax_enable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn); +void k3_nav_udmax_disable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn); +void k3_nav_udmax_tdown_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn, + bool sync); +void k3_nav_udmax_reset_tx_chn( + struct k3_nav_udmax_tx_channel *tx_chn, void *data, + void (*cleanup)(void *data, dma_addr_t desc_dma)); +u32 k3_nav_udmax_tx_get_hdesc_size(struct k3_nav_udmax_tx_channel *tx_chn); +u32 k3_nav_udmax_tx_get_txcq_id(struct k3_nav_udmax_tx_channel *tx_chn); +int k3_nav_udmax_tx_get_irq(struct k3_nav_udmax_tx_channel *tx_chn, + unsigned int *irq, u32 flags, bool share, + struct k3_nav_udmax_tx_channel *tx_chn_share); +void k3_nav_udmax_tx_put_irq(struct k3_nav_udmax_tx_channel *tx_chn); + +enum { + K3_NAV_UDMAX_SRC_TAG_LO_KEEP = 0, + K3_NAV_UDMAX_SRC_TAG_LO_USE_FLOW_REG = 1, + K3_NAV_UDMAX_SRC_TAG_LO_USE_REMOTE_FLOW_ID = 2, + K3_NAV_UDMAX_SRC_TAG_LO_USE_REMOTE_SRC_TAG = 4, +}; + +/** + * k3_nav_udmax_rx_flow_cfg - UDMA RX flow cfg + * + * @rx_cfg: RX ring configuration + * @rxfdq_cfg: RX free Host PD ring configuration + * @ring_rxq_id: RX ring id (or -1 for any) + * @ring_rxfdq0_id: RX free Host PD ring (FDQ) if (or -1 for any) + * @rx_error_handling: Rx Error Handling Mode (0 - drop, 1 - re-try) + * @src_tag_lo_sel: Rx Source Tag Low Byte Selector in Host PD + */ +struct k3_nav_udmax_rx_flow_cfg { + struct k3_nav_ring_cfg rx_cfg; + struct k3_nav_ring_cfg rxfdq_cfg; + int ring_rxq_id; + int ring_rxfdq0_id; + bool rx_error_handling; + int src_tag_lo_sel; +}; + +/** + * k3_nav_udmax_rx_channel_cfg - UDMA RX channel cfg + * + * @psdata_size: SW Data is present in Host PD of @swdata_size bytes + * @flow_id_base: first flow_id used by channel. + * if @flow_id_base = -1 - flow ids range will be allocated + * dynamically. + * @flow_id_num: number of RX flows used by channel + * @flow_id_use_rxchan_id: use RX channel id as flow id, + * used only if @flow_id_num = 1 + * @def_flow_cfg default RX flow configuration, + * used only if @flow_id_num = 1 + */ +struct k3_nav_udmax_rx_channel_cfg { + u32 swdata_size; + int flow_id_base; + int flow_id_num; + bool flow_id_use_rxchan_id; + bool skip_psil; + + struct k3_nav_udmax_rx_flow_cfg *def_flow_cfg; +}; + +struct k3_nav_udmax_rx_channel; + +struct k3_nav_udmax_rx_channel *k3_nav_udmax_request_rx_chn( + struct device *dev, + const char *name, + struct k3_nav_udmax_rx_channel_cfg *cfg); + +void k3_nav_udmax_release_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn); +int k3_nav_udmax_enable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn); +void k3_nav_udmax_disable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn); +void k3_nav_udmax_tdown_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn, + bool sync); +int k3_nav_udmax_push_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, + struct knav_udmap_host_desc_t *desc_tx, + dma_addr_t desc_dma); +int k3_nav_udmax_pop_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, + dma_addr_t *desc_dma); +int k3_nav_udmax_rx_flow_init( + struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx, + struct k3_nav_udmax_rx_flow_cfg *flow_cfg); +u32 k3_nav_udmax_rx_flow_get_fdq_id( + struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx); +u32 k3_nav_udmax_rx_get_flow_id_base( + struct k3_nav_udmax_rx_channel *rx_chn); +int k3_nav_udmax_rx_get_irq(struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, + unsigned int *irq, u32 flags, bool share, + u32 flow_num_share); +void k3_nav_udmax_rx_put_irq(struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num); +void k3_nav_udmax_reset_rx_chn( + struct k3_nav_udmax_rx_channel *rx_chn, + u32 flow_num, void *data, + void (*cleanup)(void *data, dma_addr_t desc_dma), + bool skip_fdq); + +#endif /* K3_NAVSS_UDMA_H_ */