From patchwork Fri Apr 7 19:22:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiner Kallweit X-Patchwork-Id: 9670317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AFA93602B3 for ; Fri, 7 Apr 2017 19:32:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9902228643 for ; Fri, 7 Apr 2017 19:32:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8C5EC28645; Fri, 7 Apr 2017 19:32:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DF6D528643 for ; Fri, 7 Apr 2017 19:32:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Date:Message-ID:To:Subject :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=S+qZhSWyiYxiEP44VDMd0kvqXhz03HDV0IgeB6HRweQ=; b=fp6aqNnxirr6wM /IX6W0dLelR0dzE9Ol68lXkE2qJTUfawtCAeNJIn3+82C85osFmQvttdPBNHorChwBvrnpxP9lCn3 YWVcTQQvh4dtrvsdA/hFHev0A/3tJB5ZyjDVpaD62AHCPh3WuQ4utBjNmLQNdbtj5uAKnYMshI/Vv /xv8nylkIACBgd8me33teM3zpm4bK0nx52xRUgREMS74ch4BBqFK46apMbMaShVgwpR44DuCc8nxv 7Xnk42vSjhbvOIvsMEhszwm+ba7oDLxs9vWYOXQgSE/uUd2rXPonzp5eiaOdMUhRSyv+6xBYFs1+T NZk6bJ75BFhomHtPPmsQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1cwZch-00075E-NY; Fri, 07 Apr 2017 19:32:43 +0000 Received: from casper.infradead.org ([2001:770:15f::2]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cwZcg-000753-6b for linux-amlogic@bombadil.infradead.org; Fri, 07 Apr 2017 19:32:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:Content-Type: MIME-Version:Date:Message-ID:Cc:To:Subject:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=pSxH0dmy7rPh3yIp7o/6Cs9IqUb+e2XdLvy8c2q6GB0=; b=ndUSLLDeGkKPLZ4Vht/isM8+Dt kKKmYHCjWHFZAnNRSAdmO8qoD8MkFNeBnOS3eSBcpoYJG+bsbOuJbBQCnLhf2egPtsEs3z02U5rbp UENQ/rooGuMpAfATJOEE9ksykj4iUNdiRAL132AQ5TOWLwNhdG/EBa+zfNp99/n16DOCzDMY3Qq5J 41G++ARcrGeETrI2MMMuYD1JCfiP/vI+egEl/n0FXDad4lffgqvGQQJqEhfppdYz3gU/OL3bdDVuY yGB+N7jH46MEHOU8vDUrI9Jj3Ux5BzmMOk7rXLqPzWOwwxjxD/6p1q2GYJ+DL2ffWkTCW0tFAFgvd /7TrO7tQ==; Received: from mail-wr0-x242.google.com ([2a00:1450:400c:c0c::242]) by casper.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1cwZTd-0002QZ-PT for linux-amlogic@lists.infradead.org; Fri, 07 Apr 2017 19:23:23 +0000 Received: by mail-wr0-x242.google.com with SMTP id o21so16335659wrb.3 for ; Fri, 07 Apr 2017 12:23:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:to:cc:message-id:date:user-agent:mime-version :content-transfer-encoding; bh=pSxH0dmy7rPh3yIp7o/6Cs9IqUb+e2XdLvy8c2q6GB0=; b=avoCk47yH3T86+omoO8ZSdWiHWVoIW0+lacU56iFe1Y1xFX9CXt0HDfMvMVd5CqP/h 88XQ1mI9rbxbO4FZxMNuVBdDVRYfn869NN45RR7+xh7UIlHn5ekB/orbpPRF/obj+/qx hHwSLomtZ+ePcc34vUNvMykwdclC2Kz3TK0tWHuL7wgqZP8F0rSi8tK+Do7rSOlT42jM ZJoHIxWcr08JG0W/+S70zihm1Qm+zF/eLfXeh1Dhdkub2GVV4LYOzHvpgPt0JpjwANIF 440AE+qk2zfJSZT+DL2E7WAULhi2AHsYTQDHrBYOUWIPTAKO5+Wy3bOxOYJfwrE7obMK mj+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:message-id:date:user-agent :mime-version:content-transfer-encoding; bh=pSxH0dmy7rPh3yIp7o/6Cs9IqUb+e2XdLvy8c2q6GB0=; b=p1Vt3H3B53hOJvWoYjueSP+vrC6t4wUw4oT4RK6egBwPOG6mdDj1/f6G7BA4HRihA4 hATRlJqR7ob2PGxAB5VIIuVp15taKLwXSLncX3LwNHmt1HyEJWNC6nWHV8vLS2BjQJtl t0vyqGKyn/qCNjZrFxR64RJZ106eromKUF4xbUox2Rq6npsVpWKvglmIlJbC3H3isrfU I929fsZH4HxGZj+KnJoH922+EX3F3dTgmTG9YlMWTtWADVO8lUBdQSgeNXrECbYgiKyQ Fm3HzDaMx+CNzu90Yb/4wUKzpX5DrvZoN9KEDXCxYPD7NmCYW1MA8eCg1cppjWAxIaQY bYlw== X-Gm-Message-State: AFeK/H13gpBbwlEb16s65A9Er09RB5Ft8ltkEGQ2B1hfW+bRk7RqMbKTswRDsVYoItnn9w== X-Received: by 10.223.160.69 with SMTP id l5mr9826496wrl.49.1491592979752; Fri, 07 Apr 2017 12:22:59 -0700 (PDT) Received: from ?IPv6:2003:c6:ebdc:4000:48ee:a67f:48a3:153? (p200300C6EBDC400048EEA67F48A30153.dip0.t-ipconnect.de. [2003:c6:ebdc:4000:48ee:a67f:48a3:153]) by smtp.googlemail.com with ESMTPSA id d7sm7063983wrc.6.2017.04.07.12.22.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 07 Apr 2017 12:22:59 -0700 (PDT) From: Heiner Kallweit Subject: [PATCH] mmc: meson-gx: add support for descriptor chain mode To: Ulf Hansson , Kevin Hilman , Neil Armstrong Message-ID: <214d58f4-9368-8cea-4f30-841dfaae780e@gmail.com> Date: Fri, 7 Apr 2017 21:22:44 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.0 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170407_202321_902498_39587EE4 X-CRM114-Status: GOOD ( 22.83 ) X-BeenThere: linux-amlogic@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-amlogic@lists.infradead.org, Helmut Klein , "linux-mmc@vger.kernel.org" Sender: "linux-amlogic" Errors-To: linux-amlogic-bounces+patchwork-linux-amlogic=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP So far a bounce buffer is used to serialize the scatterlist(s). This overhead can be avoided by switching to descriptor chain mode. As result the performance is drastically improved. On a Odroid-C2 with a 128 GB eMMC module raw reads reach 140 MB/s. Prerequisite for descriptor chain mode is that all scatterlist buffers are 8 byte aligned for 64-bit DMA. That's not always the case, at least the brcmfmac SDIO WiFi driver is known to cause problems. Therefore, for each request, check upfront whether all scatterlist buffers are 8 byte aligned and fall back to bounce buffer mode if that's not the case. Signed-off-by: Heiner Kallweit --- drivers/mmc/host/meson-gx-mmc.c | 172 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 156 insertions(+), 16 deletions(-) diff --git a/drivers/mmc/host/meson-gx-mmc.c b/drivers/mmc/host/meson-gx-mmc.c index eb5ee68d..1842ed34 100644 --- a/drivers/mmc/host/meson-gx-mmc.c +++ b/drivers/mmc/host/meson-gx-mmc.c @@ -111,6 +111,11 @@ #define SD_EMMC_CMD_TIMEOUT 1024 /* in ms */ #define SD_EMMC_CMD_TIMEOUT_DATA 4096 /* in ms */ #define SD_EMMC_CFG_CMD_GAP 16 /* in clock cycles */ +#define SD_EMMC_DESC_BUF_LEN PAGE_SIZE + +#define SD_EMMC_PRE_REQ_DONE BIT(0) +#define SD_EMMC_DESC_CHAIN_MODE BIT(1) + #define MUX_CLK_NUM_PARENTS 2 struct meson_tuning_params { @@ -119,6 +124,13 @@ struct meson_tuning_params { u8 rx_phase; }; +struct sd_emmc_desc { + u32 cmd_cfg; + u32 cmd_arg; + u32 cmd_data; + u32 cmd_resp; +}; + struct meson_host { struct device *dev; struct mmc_host *mmc; @@ -137,18 +149,13 @@ struct meson_host { unsigned int bounce_buf_size; void *bounce_buf; dma_addr_t bounce_dma_addr; + struct sd_emmc_desc *descs; + dma_addr_t descs_dma_addr; struct meson_tuning_params tp; bool vqmmc_enabled; }; -struct sd_emmc_desc { - u32 cmd_cfg; - u32 cmd_arg; - u32 cmd_data; - u32 cmd_resp; -}; - #define CMD_CFG_LENGTH_MASK GENMASK(8, 0) #define CMD_CFG_BLOCK_MODE BIT(9) #define CMD_CFG_R1B BIT(10) @@ -195,6 +202,66 @@ static struct mmc_command *meson_mmc_get_next_command(struct mmc_command *cmd) return NULL; } +static void meson_mmc_get_transfer_mode(struct mmc_host *mmc, + struct mmc_request *mrq) +{ + struct mmc_data *data = mrq->data; + struct scatterlist *sg; + int i; + bool use_desc_chain_mode = true; + + for_each_sg(data->sg, sg, data->sg_len, i) + /* check for 8 byte alignment */ + if (sg->offset & 7) { + WARN_ONCE(1, "unaligned scatterlist buffer\n"); + use_desc_chain_mode = false; + break; + } + + if (use_desc_chain_mode) + data->host_cookie |= SD_EMMC_DESC_CHAIN_MODE; +} + +static inline bool meson_mmc_desc_chain_mode(const struct mmc_data *data) +{ + return data->host_cookie & SD_EMMC_DESC_CHAIN_MODE; +} + +static inline bool meson_mmc_bounce_buf_read(const struct mmc_data *data) +{ + return data && data->flags & MMC_DATA_READ && + !meson_mmc_desc_chain_mode(data); +} + +static void meson_mmc_pre_req(struct mmc_host *mmc, struct mmc_request *mrq) +{ + struct mmc_data *data = mrq->data; + + if (!data) + return; + + meson_mmc_get_transfer_mode(mmc, mrq); + data->host_cookie |= SD_EMMC_PRE_REQ_DONE; + + if (!meson_mmc_desc_chain_mode(data)) + return; + + data->sg_count = dma_map_sg(mmc_dev(mmc), data->sg, data->sg_len, + mmc_get_dma_dir(data)); + if (!data->sg_count) + dev_err(mmc_dev(mmc), "dma_map_sg failed"); +} + +static void meson_mmc_post_req(struct mmc_host *mmc, struct mmc_request *mrq, + int err) +{ + struct mmc_data *data = mrq->data; + + if (data && meson_mmc_desc_chain_mode(data) && data->sg_count) + dma_unmap_sg(mmc_dev(mmc), data->sg, data->sg_len, + mmc_get_dma_dir(data)); +} + static int meson_mmc_clk_set(struct meson_host *host, unsigned long clk_rate) { struct mmc_host *mmc = host->mmc; @@ -509,6 +576,44 @@ static void meson_mmc_set_response_bits(struct mmc_command *cmd, u32 *cmd_cfg) } } +static void meson_mmc_desc_chain_transfer(struct mmc_host *mmc, u32 cmd_cfg) +{ + struct meson_host *host = mmc_priv(mmc); + struct sd_emmc_desc *desc = host->descs; + struct mmc_data *data = host->cmd->data; + struct scatterlist *sg; + u32 start; + int i; + + if (data->flags & MMC_DATA_WRITE) + cmd_cfg |= CMD_CFG_DATA_WR; + + if (data->blocks > 1) { + cmd_cfg |= CMD_CFG_BLOCK_MODE; + meson_mmc_set_blksz(mmc, data->blksz); + } + + for_each_sg(data->sg, sg, data->sg_count, i) { + unsigned int len = sg_dma_len(sg); + + if (data->blocks > 1) + len /= data->blksz; + + desc[i].cmd_cfg = cmd_cfg; + desc[i].cmd_cfg |= FIELD_PREP(CMD_CFG_LENGTH_MASK, len); + if (i > 0) + desc[i].cmd_cfg |= CMD_CFG_NO_CMD; + desc[i].cmd_arg = host->cmd->arg; + desc[i].cmd_resp = 0; + desc[i].cmd_data = sg_dma_address(sg); + } + desc[data->sg_count - 1].cmd_cfg |= CMD_CFG_END_OF_CHAIN; + + dma_wmb(); /* ensure descriptor is written before kicked */ + start = host->descs_dma_addr | START_DESC_BUSY; + writel(start, host->regs + SD_EMMC_START); +} + static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd) { struct meson_host *host = mmc_priv(mmc); @@ -519,6 +624,8 @@ static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd) /* Setup descriptors */ dma_rmb(); + host->cmd = cmd; + cmd_cfg |= FIELD_PREP(CMD_CFG_CMD_INDEX_MASK, cmd->opcode); cmd_cfg |= CMD_CFG_OWNER; /* owned by CPU */ @@ -526,10 +633,16 @@ static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd) /* data? */ if (data) { + data->bytes_xfered = 0; cmd_cfg |= CMD_CFG_DATA_IO; cmd_cfg |= FIELD_PREP(CMD_CFG_TIMEOUT_MASK, ilog2(meson_mmc_get_timeout_msecs(data))); + if (meson_mmc_desc_chain_mode(data)) { + meson_mmc_desc_chain_transfer(mmc, cmd_cfg); + return; + } + if (data->blocks > 1) { cmd_cfg |= CMD_CFG_BLOCK_MODE; cmd_cfg |= FIELD_PREP(CMD_CFG_LENGTH_MASK, @@ -539,7 +652,6 @@ static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd) cmd_cfg |= FIELD_PREP(CMD_CFG_LENGTH_MASK, data->blksz); } - data->bytes_xfered = 0; xfer_bytes = data->blksz * data->blocks; if (data->flags & MMC_DATA_WRITE) { cmd_cfg |= CMD_CFG_DATA_WR; @@ -555,8 +667,6 @@ static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd) ilog2(SD_EMMC_CMD_TIMEOUT)); } - host->cmd = cmd; - /* Last descriptor */ cmd_cfg |= CMD_CFG_END_OF_CHAIN; writel(cmd_cfg, host->regs + SD_EMMC_CMD_CFG); @@ -569,14 +679,25 @@ static void meson_mmc_start_cmd(struct mmc_host *mmc, struct mmc_command *cmd) static void meson_mmc_request(struct mmc_host *mmc, struct mmc_request *mrq) { struct meson_host *host = mmc_priv(mmc); + bool needs_pre_post_req = mrq->data && + !(mrq->data->host_cookie & SD_EMMC_PRE_REQ_DONE); + + if (needs_pre_post_req) { + meson_mmc_get_transfer_mode(mmc, mrq); + if (!meson_mmc_desc_chain_mode(mrq->data)) + needs_pre_post_req = false; + } + + if (needs_pre_post_req) + meson_mmc_pre_req(mmc, mrq); /* Stop execution */ writel(0, host->regs + SD_EMMC_START); - if (mrq->sbc) - meson_mmc_start_cmd(mmc, mrq->sbc); - else - meson_mmc_start_cmd(mmc, mrq->cmd); + meson_mmc_start_cmd(mmc, mrq->sbc ?: mrq->cmd); + + if (needs_pre_post_req) + meson_mmc_post_req(mmc, mrq, 0); } static void meson_mmc_read_resp(struct mmc_host *mmc, struct mmc_command *cmd) @@ -654,7 +775,9 @@ static irqreturn_t meson_mmc_irq(int irq, void *dev_id) if (status & (IRQ_END_OF_CHAIN | IRQ_RESP_STATUS)) { if (data && !cmd->error) data->bytes_xfered = data->blksz * data->blocks; - ret = IRQ_WAKE_THREAD; + if (meson_mmc_bounce_buf_read(data) || + meson_mmc_get_next_command(cmd)) + ret = IRQ_WAKE_THREAD; } else { dev_warn(host->dev, "Unknown IRQ! status=0x%04x: MMC CMD%u arg=0x%08x flags=0x%08x stop=%d\n", status, cmd->opcode, cmd->arg, @@ -691,7 +814,7 @@ static irqreturn_t meson_mmc_irq_thread(int irq, void *dev_id) return IRQ_NONE; data = cmd->data; - if (data && data->flags & MMC_DATA_READ) { + if (meson_mmc_bounce_buf_read(data)) { xfer_bytes = data->blksz * data->blocks; WARN_ON(xfer_bytes > host->bounce_buf_size); sg_copy_from_buffer(data->sg, data->sg_len, @@ -760,6 +883,8 @@ static const struct mmc_host_ops meson_mmc_ops = { .request = meson_mmc_request, .set_ios = meson_mmc_set_ios, .get_cd = meson_mmc_get_cd, + .pre_req = meson_mmc_pre_req, + .post_req = meson_mmc_post_req, .execute_tuning = meson_mmc_execute_tuning, }; @@ -845,6 +970,8 @@ static int meson_mmc_probe(struct platform_device *pdev) mmc->caps |= MMC_CAP_CMD23; mmc->max_blk_count = CMD_CFG_LENGTH_MASK; mmc->max_req_size = mmc->max_blk_count * mmc->max_blk_size; + mmc->max_segs = SD_EMMC_DESC_BUF_LEN / sizeof(struct sd_emmc_desc); + mmc->max_seg_size = mmc->max_req_size; /* data bounce buffer */ host->bounce_buf_size = mmc->max_req_size; @@ -857,11 +984,22 @@ static int meson_mmc_probe(struct platform_device *pdev) goto err_div_clk; } + host->descs = dma_alloc_coherent(host->dev, SD_EMMC_DESC_BUF_LEN, + &host->descs_dma_addr, GFP_KERNEL); + if (!host->descs) { + dev_err(host->dev, "Allocating descriptor DMA buffer failed\n"); + ret = -ENOMEM; + goto err_bounce_buf; + } + mmc->ops = &meson_mmc_ops; mmc_add_host(mmc); return 0; +err_bounce_buf: + dma_free_coherent(host->dev, host->bounce_buf_size, + host->bounce_buf, host->bounce_dma_addr); err_div_clk: clk_disable_unprepare(host->cfg_div_clk); err_core_clk: @@ -880,6 +1018,8 @@ static int meson_mmc_remove(struct platform_device *pdev) /* disable interrupts */ writel(0, host->regs + SD_EMMC_IRQ_EN); + dma_free_coherent(host->dev, SD_EMMC_DESC_BUF_LEN, + host->descs, host->descs_dma_addr); dma_free_coherent(host->dev, host->bounce_buf_size, host->bounce_buf, host->bounce_dma_addr);