From patchwork Tue Oct 3 23:00:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Lamparter X-Patchwork-Id: 9983627 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 470106029B for ; Tue, 3 Oct 2017 23:00:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D92528A38 for ; Tue, 3 Oct 2017 23:00:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 324FB28A41; Tue, 3 Oct 2017 23:00:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B54F528A38 for ; Tue, 3 Oct 2017 23:00:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751250AbdJCXA0 (ORCPT ); Tue, 3 Oct 2017 19:00:26 -0400 Received: from mail-wr0-f195.google.com ([209.85.128.195]:38632 "EHLO mail-wr0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751263AbdJCXAY (ORCPT ); Tue, 3 Oct 2017 19:00:24 -0400 Received: by mail-wr0-f195.google.com with SMTP id f12so1018036wra.5 for ; Tue, 03 Oct 2017 16:00:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :in-reply-to:references; bh=/I5CcA7ZCtlSh5kTujBeU9RdiNQ6BtghlU9IJMJQJJo=; b=vBENYJCIawFwc76+WpbBgpdyJgEhxXCT5TJ+EC24Iin0X7elWeYcljsPlhxSVK0tE2 BXT1JPa1KJyEF1vdpgRDNlgBlSJtSaqlPEGL01eQGi0rgCLEQdy/kZ00eCCwAuCFMaB3 QcjtC5PaxTsQJgf7ttjxG2boGSFHnNKBD6FwGsnKv3kOJXUS3a5lt33hPXI1J6AQ9awD 8bgEBVmgQ1wUZLLGnD3Lb9BV8BwLNtMKvxdbfghZ6U/iwgbNJaOt1Z/LepFaQw2iMM7c mMHukHa4MjJoGLFuNy/2PpVaLnSaiqXaDnaPAX2OzHm67yB1mPCtTDgoLfaiEZrrjnpM /Gtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:in-reply-to:references; bh=/I5CcA7ZCtlSh5kTujBeU9RdiNQ6BtghlU9IJMJQJJo=; b=bulLY2nA4PhvWX5cQheEAseQ/x1UCUkzMdT2U3IIYeF006G4ZR51Yu5m/fMb1O9LW8 YyfIxqcmnBpM8wd8O2A2KDb0qpK/iLKOIMLeEHnefGkd7tbdXygxDD0Mnb7kR+WlO9GA 37QVXa4JFOKaaN6lO71aorQkCT7izuqoEVRzCB+SYK1pSywcZ+8YaMoqbVwDUK8As6QJ jVwTtJmRsqy/qsgvohpXGt8ik4Ex6JmN3bZ8FDH5pINC8ZQmj9CR6SqFnWh3OaBIZAsB v1GnLWS2HFPbve/vfPKyaCqmPcvYW+MkQf0wg8+6xzDdoUdhyCUgsQFp0et1MJ9XD/p2 PusQ== X-Gm-Message-State: AHPjjUhl/9MsXzOpK2oHT0OiWJCEl1fcUNko/72YMa6MKfXVeMz5zuXI JPF1u3A4QGixfLxWhOY8FVPNNUUX X-Google-Smtp-Source: AOwi7QCt1eHKS9JUVj4TLnSgeVcHcrf933EJ7gkaVOhzCXoAvL8ZW4PI4KAsuNTEJanq+c1Oa1SXoQ== X-Received: by 10.223.198.79 with SMTP id u15mr16177821wrg.10.1507071623226; Tue, 03 Oct 2017 16:00:23 -0700 (PDT) Received: from debian64.daheim (p200300D5FBCA48FCD63D7EFFFEBDE96E.dip0.t-ipconnect.de. [2003:d5:fbca:48fc:d63d:7eff:febd:e96e]) by smtp.gmail.com with ESMTPSA id o27sm7362447wro.72.2017.10.03.16.00.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 03 Oct 2017 16:00:19 -0700 (PDT) Received: from chuck by debian64.daheim with local (Exim 4.89) (envelope-from ) id 1dzWAj-00010g-LD; Wed, 04 Oct 2017 01:00:17 +0200 From: Christian Lamparter To: linux-crypto@vger.kernel.org Cc: Herbert Xu Subject: [PATCH 09/13] crypto: crypto4xx: fix stalls under heavy load Date: Wed, 4 Oct 2017 01:00:13 +0200 Message-Id: X-Mailer: git-send-email 2.14.2 In-Reply-To: <8c9b4bc7e3a88970fe0fc308034627b8ae972600.1507070985.git.chunkeey@gmail.com> References: <8c9b4bc7e3a88970fe0fc308034627b8ae972600.1507070985.git.chunkeey@gmail.com> In-Reply-To: <8c9b4bc7e3a88970fe0fc308034627b8ae972600.1507070985.git.chunkeey@gmail.com> References: <8c9b4bc7e3a88970fe0fc308034627b8ae972600.1507070985.git.chunkeey@gmail.com> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If the crypto4xx device is continuously loaded by dm-crypt and ipsec work, it will start to work intermittent after a few (between 20-30) seconds, hurting throughput and latency. This patch contains various stability improvements in order to fix this issue. So far, the hardware has survived more than a day without suffering any stalls under the continuous load. Signed-off-by: Christian Lamparter --- drivers/crypto/amcc/crypto4xx_core.c | 33 ++++++++++++++++++--------------- drivers/crypto/amcc/crypto4xx_reg_def.h | 3 +++ 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/drivers/crypto/amcc/crypto4xx_core.c b/drivers/crypto/amcc/crypto4xx_core.c index 1752ea2125db..de9044201a23 100644 --- a/drivers/crypto/amcc/crypto4xx_core.c +++ b/drivers/crypto/amcc/crypto4xx_core.c @@ -280,17 +280,20 @@ static u32 crypto4xx_get_pd_from_pdr_nolock(struct crypto4xx_device *dev) static u32 crypto4xx_put_pd_to_pdr(struct crypto4xx_device *dev, u32 idx) { struct pd_uinfo *pd_uinfo = &dev->pdr_uinfo[idx]; + u32 tail; unsigned long flags; spin_lock_irqsave(&dev->core_dev->lock, flags); + pd_uinfo->state = PD_ENTRY_FREE; + if (dev->pdr_tail != PPC4XX_LAST_PD) dev->pdr_tail++; else dev->pdr_tail = 0; - pd_uinfo->state = PD_ENTRY_FREE; + tail = dev->pdr_tail; spin_unlock_irqrestore(&dev->core_dev->lock, flags); - return 0; + return tail; } /** @@ -854,16 +857,16 @@ int crypto4xx_build_pd(struct crypto_async_request *req, } } - sa->sa_command_1.bf.hash_crypto_offset = 0; - pd->pd_ctl.w = 0; - pd->pd_ctl.bf.hash_final = - (crypto_tfm_alg_type(req->tfm) == CRYPTO_ALG_TYPE_AHASH); - pd->pd_ctl.bf.host_ready = 1; + pd->pd_ctl.w = PD_CTL_HOST_READY | + ((crypto_tfm_alg_type(req->tfm) == CRYPTO_ALG_TYPE_AHASH) | + (crypto_tfm_alg_type(req->tfm) == CRYPTO_ALG_TYPE_AEAD) ? + PD_CTL_HASH_FINAL : 0); pd->pd_ctl_len.w = 0x00400000 | datalen; pd_uinfo->state = PD_ENTRY_INUSE | (is_busy ? PD_ENTRY_BUSY : 0); wmb(); /* write any value to push engine to read a pd */ + writel(0, dev->ce_base + CRYPTO4XX_INT_DESCR_RD); writel(1, dev->ce_base + CRYPTO4XX_INT_DESCR_RD); return is_busy ? -EBUSY : -EINPROGRESS; } @@ -964,23 +967,23 @@ static void crypto4xx_bh_tasklet_cb(unsigned long data) struct crypto4xx_core_device *core_dev = dev_get_drvdata(dev); struct pd_uinfo *pd_uinfo; struct ce_pd *pd; - u32 tail; + u32 tail = core_dev->dev->pdr_tail; + u32 head = core_dev->dev->pdr_head; - while (core_dev->dev->pdr_head != core_dev->dev->pdr_tail) { - tail = core_dev->dev->pdr_tail; + do { pd_uinfo = &core_dev->dev->pdr_uinfo[tail]; pd = &core_dev->dev->pdr[tail]; if ((pd_uinfo->state & PD_ENTRY_INUSE) && - pd->pd_ctl.bf.pe_done && - !pd->pd_ctl.bf.host_ready) { - pd->pd_ctl.bf.pe_done = 0; + ((READ_ONCE(pd->pd_ctl.w) & + (PD_CTL_PE_DONE | PD_CTL_HOST_READY)) == + PD_CTL_PE_DONE)) { crypto4xx_pd_done(core_dev->dev, tail); - crypto4xx_put_pd_to_pdr(core_dev->dev, tail); + tail = crypto4xx_put_pd_to_pdr(core_dev->dev, tail); } else { /* if tail not done, break */ break; } - } + } while (head != tail); } /** diff --git a/drivers/crypto/amcc/crypto4xx_reg_def.h b/drivers/crypto/amcc/crypto4xx_reg_def.h index 279b8725559f..0a22ec5d1a96 100644 --- a/drivers/crypto/amcc/crypto4xx_reg_def.h +++ b/drivers/crypto/amcc/crypto4xx_reg_def.h @@ -261,6 +261,9 @@ union ce_pd_ctl { } bf; u32 w; } __attribute__((packed)); +#define PD_CTL_HASH_FINAL BIT(4) +#define PD_CTL_PE_DONE BIT(1) +#define PD_CTL_HOST_READY BIT(0) union ce_pd_ctl_len { struct {