From patchwork Mon Jul 8 06:32:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 11034617 X-Patchwork-Delegate: kvalo@adurom.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D15CD138D for ; Mon, 8 Jul 2019 06:36:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3C4927F10 for ; Mon, 8 Jul 2019 06:36:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A45B428173; Mon, 8 Jul 2019 06:36:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B8D627F10 for ; Mon, 8 Jul 2019 06:36:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728913AbfGHGgp (ORCPT ); Mon, 8 Jul 2019 02:36:45 -0400 Received: from mail-pl1-f196.google.com ([209.85.214.196]:43426 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727420AbfGHGgo (ORCPT ); Mon, 8 Jul 2019 02:36:44 -0400 Received: by mail-pl1-f196.google.com with SMTP id cl9so7689330plb.10 for ; Sun, 07 Jul 2019 23:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessm-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=U5faaB2QS25gWcbZp0574GO7urSkcBDHpx7JYrayj6s=; b=k8AtqPPBAIU9BFgDc1LoJIroW15SoAkdAibpdma80r6iYC7+o9XxniV5aE+9a8Kcy7 JsfzCV4LoqoMGDyWAXbL8IqAQQ6ANOsk74ESWQ972qbNDrsf0Irzbm8rGHqwIZwaewZ7 xkEHNYC2wm5qTxlC8E8dWNp8znXKneU6XMJxcjqwsyaNSHV6/76+2bcasa5kmADB+UPl hrxgyIIvLY8QT2pUiwg+zKSWMI72eaq9go97g8Mey4V6Z/F0DBGAgHPc80HwTl4ARX9N GUUEo6RT5DvII4ycF6gwUa1uuC0n630vdKZU5IWN1FBLlL+7AtHDxs+y2iUQNTzI8yo1 02kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=U5faaB2QS25gWcbZp0574GO7urSkcBDHpx7JYrayj6s=; b=GPD9MIhzMFFbKi66d5mBY57WDOuQZsX3oL12zAh+wEllIKNsWgYTgnfzIyXKLw/LYC SIz4qvnyehQW9/p2Kx/itB/uwaKj4RN7xNtnQ9JE0H5LwIjJ6WXCnvvA/stfVO7pXxJd evc97tzTdUlrpQmu7KCLz9+1NchIlkLntOc9lHvIVJNEpu+Ftld/r4urp8of0olWrASi UsNTUrO7FtLzzbha1Ie7ieko+nweIPf9vUNCJo1BQ0UmtXD83VFCEiOVtSLyWtaEeEeZ 4JEOG9qx8skaoNjqjuCV90nbIxcNcrKxLOC7DAFGeQrDiN0Waf5rvcimwx7LkruXHwTs fU8w== X-Gm-Message-State: APjAAAVIlTtMGdspizXUSMO6jb48erTo2jxfIoIzPBUkKZcULnCKRGkr qHXPbQuNx37blaqc14y7ConNIg== X-Google-Smtp-Source: APXvYqxA2c9ltYeJuPS/96QyMzXMHZoAV+lGrhI7hijTtcZlfz7f4W53t2DqphIMtl/XnK0J1m4Lvw== X-Received: by 2002:a17:902:724:: with SMTP id 33mr21908419pli.49.1562567804038; Sun, 07 Jul 2019 23:36:44 -0700 (PDT) Received: from localhost.localdomain (123-204-46-122.static.seed.net.tw. [123.204.46.122]) by smtp.gmail.com with ESMTPSA id s66sm21388130pgs.39.2019.07.07.23.36.41 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sun, 07 Jul 2019 23:36:43 -0700 (PDT) From: Jian-Hong Pan To: Yan-Hsuan Chuang , Kalle Valo , "David S . Miller" Cc: linux-wireless@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessm.com, Jian-Hong Pan , Daniel Drake , stable@vger.kernel.org Subject: [PATCH] rtw88/pci: Rearrange the memory usage for skb in RX ISR Date: Mon, 8 Jul 2019 14:32:53 +0800 Message-Id: <20190708063252.4756-1-jian-hong@endlessm.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 Sender: linux-wireless-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-wireless@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Testing with RTL8822BE hardware, when available memory is low, we frequently see a kernel panic and system freeze. First, rtw_pci_rx_isr encounters a memory allocation failure (trimmed): rx routine starvation WARNING: CPU: 7 PID: 9871 at drivers/net/wireless/realtek/rtw88/pci.c:822 rtw_pci_rx_isr.constprop.25+0x35a/0x370 [rtwpci] [ 2356.580313] RIP: 0010:rtw_pci_rx_isr.constprop.25+0x35a/0x370 [rtwpci] Then we see a variety of different error conditions and kernel panics, such as this one (trimmed): rtw_pci 0000:02:00.0: pci bus timeout, check dma status skbuff: skb_over_panic: text:00000000091b6e66 len:415 put:415 head:00000000d2880c6f data:000000007a02b1ea tail:0x1df end:0xc0 dev: ------------[ cut here ]------------ kernel BUG at net/core/skbuff.c:105! invalid opcode: 0000 [#1] SMP NOPTI RIP: 0010:skb_panic+0x43/0x45 When skb allocation fails and the "rx routine starvation" is hit, the function returns immediately without updating the RX ring. At this point, the RX ring may continue referencing an old skb which was already handed off to ieee80211_rx_irqsafe(). When it comes to be used again, bad things happen. This patch allocates a new skb first in RX ISR. If we don't have memory available, we discard the current frame, allowing the existing skb to be reused in the ring. Otherwise, we simplify the code flow and just hand over the RX-populated skb over to mac80211. In addition, to fixing the kernel crash, the RX routine should now generally behave better under low memory conditions. Buglink: https://bugzilla.kernel.org/show_bug.cgi?id=204053 Signed-off-by: Jian-Hong Pan Reviewed-by: Daniel Drake Cc: --- drivers/net/wireless/realtek/rtw88/pci.c | 28 +++++++++++------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/drivers/net/wireless/realtek/rtw88/pci.c b/drivers/net/wireless/realtek/rtw88/pci.c index cfe05ba7280d..1bfc99ae6b84 100644 --- a/drivers/net/wireless/realtek/rtw88/pci.c +++ b/drivers/net/wireless/realtek/rtw88/pci.c @@ -786,6 +786,15 @@ static void rtw_pci_rx_isr(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci, rx_desc = skb->data; chip->ops->query_rx_desc(rtwdev, rx_desc, &pkt_stat, &rx_status); + /* discard current skb if the new skb cannot be allocated as a + * new one in rx ring later + * */ + new = dev_alloc_skb(RTK_PCI_RX_BUF_SIZE); + if (WARN(!new, "rx routine starvation\n")) { + new = skb; + goto next_rp; + } + /* offset from rx_desc to payload */ pkt_offset = pkt_desc_sz + pkt_stat.drv_info_sz + pkt_stat.shift; @@ -803,25 +812,14 @@ static void rtw_pci_rx_isr(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci, skb_put(skb, pkt_stat.pkt_len); skb_reserve(skb, pkt_offset); - /* alloc a smaller skb to mac80211 */ - new = dev_alloc_skb(pkt_stat.pkt_len); - if (!new) { - new = skb; - } else { - skb_put_data(new, skb->data, skb->len); - dev_kfree_skb_any(skb); - } /* TODO: merge into rx.c */ rtw_rx_stats(rtwdev, pkt_stat.vif, skb); - memcpy(new->cb, &rx_status, sizeof(rx_status)); - ieee80211_rx_irqsafe(rtwdev->hw, new); + memcpy(skb->cb, &rx_status, sizeof(rx_status)); + ieee80211_rx_irqsafe(rtwdev->hw, skb); } - /* skb delivered to mac80211, alloc a new one in rx ring */ - new = dev_alloc_skb(RTK_PCI_RX_BUF_SIZE); - if (WARN(!new, "rx routine starvation\n")) - return; - +next_rp: + /* skb delivered to mac80211, attach the new one into rx ring */ ring->buf[cur_rp] = new; rtw_pci_reset_rx_desc(rtwdev, new, ring, cur_rp, buf_desc_sz); From patchwork Tue Jul 9 10:21:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jian-Hong Pan X-Patchwork-Id: 11036811 X-Patchwork-Delegate: kvalo@adurom.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1D551395 for ; Tue, 9 Jul 2019 10:22:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B1C8928714 for ; Tue, 9 Jul 2019 10:22:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A598728717; Tue, 9 Jul 2019 10:22:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9052728714 for ; Tue, 9 Jul 2019 10:22:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726565AbfGIKW1 (ORCPT ); Tue, 9 Jul 2019 06:22:27 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:41652 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726126AbfGIKW1 (ORCPT ); Tue, 9 Jul 2019 06:22:27 -0400 Received: by mail-pf1-f194.google.com with SMTP id m30so9066217pff.8 for ; Tue, 09 Jul 2019 03:22:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=endlessm-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ys5VCG//C+7d/Mv4iPcZy1mrBP7z7sVwq8ShwHAlzGY=; b=EYqSBY5RYT3l0wkuw1OiniPSthEwR1m6QScRVqkNRuncNcPiHi18uDfiPzlQ64tyOT tFnG4+QleLyJ5BINvC4LqWUj6LzpVUIwfmTehOtXgG+QCpSO2kmuYWp4LBvDqw4TAycS rUPJnPjVy196wMzOuZn7AILaZrq5Otwr4AfORpaIWUOi8XFXWgYvtdPXoso5l+txhymh 3kr4MOeTNuPTCctGwVMZnXh4EeeJGe2tiGybWZlLLrOrRpCTaKMwQpUdPkM8bettx8Wr fS9nzTh9CWZuuNv6NWzmVBkaHo/g7OgOJFLMdKB7JtjNZvusIkI0fUWh6GaLzdxhiZM+ WRLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ys5VCG//C+7d/Mv4iPcZy1mrBP7z7sVwq8ShwHAlzGY=; b=nU967YWBVp8aGz5TyZEpbtmEZyBSmRD8ExLpdaKmDhXe/69jTtatabuQ/6tRiq5TDP z/GfiJEK9erY46mFhmOhsbg9a2o8uJSEs5ANdl2+JkzngyzwLokavlbJdXEg8NeFT8cV tTk3XdOfRMr3DbDKV5TNFZsyDfPUa06iAzGay51LHZ4jjDkrSEUZoz+6DMcqFUggrgel VafMElDY1VeB1x7ItEep4hcs5azvG7SHUxmfzfiBRJmiVo2Xp6tPdMmOToBBeL59Pmce cPbxA4wvwg27j1o7ch0QCxf/9kXvfoFsJ3wUPAkorr3lwkId+pN46wDMSRAjGtrSfMVT dlpA== X-Gm-Message-State: APjAAAVWFUH1AXxrT3wG9YPNIgRjoVPc0eJE9JqaQqpTTCVyXvu4pBOQ B0blc8a801+MXPHFXf6UL8204A== X-Google-Smtp-Source: APXvYqy4rOzsSukM8QjFevK1EGTmIXDq4LXaDasAKMlIyOuWJyyfPkpJ2F67P17lv+jigztmLW01bw== X-Received: by 2002:a65:56c1:: with SMTP id w1mr25026367pgs.395.1562667746195; Tue, 09 Jul 2019 03:22:26 -0700 (PDT) Received: from localhost.localdomain (123-204-46-122.static.seed.net.tw. [123.204.46.122]) by smtp.gmail.com with ESMTPSA id j1sm40161849pgl.12.2019.07.09.03.22.23 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 09 Jul 2019 03:22:25 -0700 (PDT) From: Jian-Hong Pan To: Yan-Hsuan Chuang , Kalle Valo , "David S . Miller" , Larry Finger , David Laight Cc: linux-wireless@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux@endlessm.com, Daniel Drake , Jian-Hong Pan Subject: [PATCH v2 2/2] rtw88: pci: Use DMA sync instead of remapping in RX ISR Date: Tue, 9 Jul 2019 18:21:01 +0800 Message-Id: <20190709102059.7036-2-jian-hong@endlessm.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190708063252.4756-1-jian-hong@endlessm.com> References: <20190708063252.4756-1-jian-hong@endlessm.com> MIME-Version: 1.0 Sender: linux-wireless-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-wireless@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since each skb in RX ring is reused instead of new allocation, we can treat the DMA in a more efficient way by DMA synchronization. Signed-off-by: Jian-Hong Pan --- drivers/net/wireless/realtek/rtw88/pci.c | 35 ++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/drivers/net/wireless/realtek/rtw88/pci.c b/drivers/net/wireless/realtek/rtw88/pci.c index e9fe3ad896c8..28ca76f71dfe 100644 --- a/drivers/net/wireless/realtek/rtw88/pci.c +++ b/drivers/net/wireless/realtek/rtw88/pci.c @@ -206,6 +206,35 @@ static int rtw_pci_reset_rx_desc(struct rtw_dev *rtwdev, struct sk_buff *skb, return 0; } +static int rtw_pci_sync_rx_desc_cpu(struct rtw_dev *rtwdev, dma_addr_t dma) +{ + struct device *dev = rtwdev->dev; + int buf_sz = RTK_PCI_RX_BUF_SIZE; + + dma_sync_single_for_cpu(dev, dma, buf_sz, PCI_DMA_FROMDEVICE); + + return 0; +} + +static int rtw_pci_sync_rx_desc_device(struct rtw_dev *rtwdev, dma_addr_t dma, + struct rtw_pci_rx_ring *rx_ring, + u32 idx, u32 desc_sz) +{ + struct device *dev = rtwdev->dev; + struct rtw_pci_rx_buffer_desc *buf_desc; + int buf_sz = RTK_PCI_RX_BUF_SIZE; + + dma_sync_single_for_device(dev, dma, buf_sz, PCI_DMA_FROMDEVICE); + + buf_desc = (struct rtw_pci_rx_buffer_desc *)(rx_ring->r.head + + idx * desc_sz); + memset(buf_desc, 0, sizeof(*buf_desc)); + buf_desc->buf_size = cpu_to_le16(RTK_PCI_RX_BUF_SIZE); + buf_desc->dma = cpu_to_le32(dma); + + return 0; +} + static int rtw_pci_init_rx_ring(struct rtw_dev *rtwdev, struct rtw_pci_rx_ring *rx_ring, u8 desc_size, u32 len) @@ -782,8 +811,7 @@ static void rtw_pci_rx_isr(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci, rtw_pci_dma_check(rtwdev, ring, cur_rp); skb = ring->buf[cur_rp]; dma = *((dma_addr_t *)skb->cb); - pci_unmap_single(rtwpci->pdev, dma, RTK_PCI_RX_BUF_SIZE, - PCI_DMA_FROMDEVICE); + rtw_pci_sync_rx_desc_cpu(rtwdev, dma); rx_desc = skb->data; chip->ops->query_rx_desc(rtwdev, rx_desc, &pkt_stat, &rx_status); @@ -818,7 +846,8 @@ static void rtw_pci_rx_isr(struct rtw_dev *rtwdev, struct rtw_pci *rtwpci, next_rp: /* new skb delivered to mac80211, re-enable original skb DMA */ - rtw_pci_reset_rx_desc(rtwdev, skb, ring, cur_rp, buf_desc_sz); + rtw_pci_sync_rx_desc_device(rtwdev, dma, ring, cur_rp, + buf_desc_sz); /* host read next element in ring */ if (++cur_rp >= ring->r.len)