From patchwork Tue Nov 28 00:26:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Fraker X-Patchwork-Id: 13470431 X-Patchwork-Delegate: kuba@kernel.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rIrpXgXI" Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AD75C1 for ; Mon, 27 Nov 2023 16:27:22 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-daee86e2d70so5942009276.0 for ; Mon, 27 Nov 2023 16:27:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701131241; x=1701736041; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kekvBZIjgx8B5SHHHMl/JjBecTTdofRLTOShhgmahqU=; b=rIrpXgXIw8aAAxedCzO8X2MnVTbXt0YZ85pVLOi4ahoTTlqA1s3DQLbwi3/wqvDQnG h2ypjzhXaAKFL7x4DE4zp7wSghb+06wNP92j7krcinlTir+3ABmo8TjI8N33WH98Lg1Q X4tgWzZ9npkv8fJ3/L6pMLjOo3YFr98YuNqsPXtDZOXn0uTrY5u9rC+XY/a4v/FrvZJ9 3YYdheufOOvJ2ix5XSu4/qAimR0kMQXC0OeYYgxQNHrcgXrIspg27mnRB7upBz9g7tGh TB32uFtGwWyzgcP/f2Dqaki/Ch4NhUnAqH7niB/kDHM/IxXysmqWyvdj3H3bETvw8gEx yH/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701131241; x=1701736041; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kekvBZIjgx8B5SHHHMl/JjBecTTdofRLTOShhgmahqU=; b=oTvPHyx5nKZNBYAMVSEnrjCHxVS7P5Fp4hWX0Shve2rZ6IjW6DetJk7nwOd0kzMRKx jj9u2R0nQGHp4fxyZm+s75mpD3KPCgf0IAfO2jnAaXvokOeVOxWHrHHNQi9DmoAHzUeA McBrom83Y9YH+rJ7IVdaD/zKzWu01YnDp46UtriwsUhPFL6BI2dbro1S2diN3h2Kemy/ Vzz285cwj/GHhAMl2mx8U6kq1SZKUx5RPxEvtZJ0jlM4wZAT9BUpKlGSkGYP/B7YSkDw HYabVPyNtntl9XA09nAkL4a6j1WYTlqkVaNPwJoqjDVi7IMOneHo1twUAuoihcrm4aIL WBRA== X-Gm-Message-State: AOJu0YzZ/eleVeF3WDcmQekh3OUGMjKRl7IDkfWyhFxcCC64PcVrksvs ATzs9qVfeuT3MMwqBkWU3jY75Prz12itdkgE6shb3RdI8OA1Cjxeh5J+mtZQVU41/q0IGqGcCHm DgYxQtXRVBlrjgE1jLHgbcZdK2CrBKCTsNRj63v+tfLna7tnqWkTLUstEFvMp1voI X-Google-Smtp-Source: AGHT+IF+VYh18iez2e0dTFhqxypcKofIu1kgfuCV72Cv8cTSUAgjbCIz5Av8rr5ZC9uMheAKFogk5VSvT+uj X-Received: from jfraker202.plv.corp.google.com ([2620:15c:11c:202:19d5:f826:3460:9345]) (user=jfraker job=sendgmr) by 2002:a25:3d44:0:b0:db3:fa34:50b0 with SMTP id k65-20020a253d44000000b00db3fa3450b0mr423367yba.4.1701131241575; Mon, 27 Nov 2023 16:27:21 -0800 (PST) Date: Mon, 27 Nov 2023 16:26:44 -0800 In-Reply-To: <20231128002648.320892-1-jfraker@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231128002648.320892-1-jfraker@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231128002648.320892-2-jfraker@google.com> Subject: [PATCH net-next 1/5] gve: Perform adminq allocations through a dma_pool. From: John Fraker To: netdev@vger.kernel.org Cc: John Fraker , Jordan Kimbrough , Willem de Bruijn X-Patchwork-Delegate: kuba@kernel.org This allows the adminq to be smaller than a page, paving the way for non 4k page support. This is to support platforms where PAGE_SIZE is not 4k, such as some ARM platforms. Signed-off-by: Jordan Kimbrough Signed-off-by: John Fraker Reviewed-by: Willem de Bruijn --- drivers/net/ethernet/google/gve/gve.h | 4 ++++ drivers/net/ethernet/google/gve/gve_adminq.c | 28 ++++++++++++++++++---------- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index 0d1e681be..abc0c708b 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -8,6 +8,7 @@ #define _GVE_H_ #include +#include #include #include #include @@ -41,6 +42,8 @@ #define NIC_TX_STATS_REPORT_NUM 0 #define NIC_RX_STATS_REPORT_NUM 4 +#define GVE_ADMINQ_BUFFER_SIZE 4096 + #define GVE_DATA_SLOT_ADDR_PAGE_MASK (~(PAGE_SIZE - 1)) /* PTYPEs are always 10 bits. */ @@ -672,6 +675,7 @@ struct gve_priv { /* Admin queue - see gve_adminq.h*/ union gve_adminq_command *adminq; dma_addr_t adminq_bus_addr; + struct dma_pool *adminq_pool; u32 adminq_mask; /* masks prod_cnt to adminq size */ u32 adminq_prod_cnt; /* free-running count of AQ cmds executed */ u32 adminq_cmd_fail; /* free-running count of AQ cmds failed */ diff --git a/drivers/net/ethernet/google/gve/gve_adminq.c b/drivers/net/ethernet/google/gve/gve_adminq.c index 79db7a6d4..d3f3a0152 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.c +++ b/drivers/net/ethernet/google/gve/gve_adminq.c @@ -194,12 +194,19 @@ gve_process_device_options(struct gve_priv *priv, int gve_adminq_alloc(struct device *dev, struct gve_priv *priv) { - priv->adminq = dma_alloc_coherent(dev, PAGE_SIZE, - &priv->adminq_bus_addr, GFP_KERNEL); - if (unlikely(!priv->adminq)) + priv->adminq_pool = dma_pool_create("adminq_pool", dev, + GVE_ADMINQ_BUFFER_SIZE, 0, 0); + if (unlikely(!priv->adminq_pool)) return -ENOMEM; + priv->adminq = dma_pool_alloc(priv->adminq_pool, GFP_KERNEL, + &priv->adminq_bus_addr); + if (unlikely(!priv->adminq)) { + dma_pool_destroy(priv->adminq_pool); + return -ENOMEM; + } - priv->adminq_mask = (PAGE_SIZE / sizeof(union gve_adminq_command)) - 1; + priv->adminq_mask = + (GVE_ADMINQ_BUFFER_SIZE / sizeof(union gve_adminq_command)) - 1; priv->adminq_prod_cnt = 0; priv->adminq_cmd_fail = 0; priv->adminq_timeouts = 0; @@ -251,7 +258,8 @@ void gve_adminq_free(struct device *dev, struct gve_priv *priv) if (!gve_get_admin_queue_ok(priv)) return; gve_adminq_release(priv); - dma_free_coherent(dev, PAGE_SIZE, priv->adminq, priv->adminq_bus_addr); + dma_pool_free(priv->adminq_pool, priv->adminq, priv->adminq_bus_addr); + dma_pool_destroy(priv->adminq_pool); gve_clear_admin_queue_ok(priv); } @@ -778,8 +786,8 @@ int gve_adminq_describe_device(struct gve_priv *priv) u16 mtu; memset(&cmd, 0, sizeof(cmd)); - descriptor = dma_alloc_coherent(&priv->pdev->dev, PAGE_SIZE, - &descriptor_bus, GFP_KERNEL); + descriptor = dma_pool_alloc(priv->adminq_pool, GFP_KERNEL, + &descriptor_bus); if (!descriptor) return -ENOMEM; cmd.opcode = cpu_to_be32(GVE_ADMINQ_DESCRIBE_DEVICE); @@ -787,7 +795,8 @@ int gve_adminq_describe_device(struct gve_priv *priv) cpu_to_be64(descriptor_bus); cmd.describe_device.device_descriptor_version = cpu_to_be32(GVE_ADMINQ_DEVICE_DESCRIPTOR_VERSION); - cmd.describe_device.available_length = cpu_to_be32(PAGE_SIZE); + cmd.describe_device.available_length = + cpu_to_be32(GVE_ADMINQ_BUFFER_SIZE); err = gve_adminq_execute_cmd(priv, &cmd); if (err) @@ -868,8 +877,7 @@ int gve_adminq_describe_device(struct gve_priv *priv) dev_op_jumbo_frames, dev_op_dqo_qpl); free_device_descriptor: - dma_free_coherent(&priv->pdev->dev, PAGE_SIZE, descriptor, - descriptor_bus); + dma_pool_free(priv->adminq_pool, descriptor, descriptor_bus); return err; } From patchwork Tue Nov 28 00:26:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Fraker X-Patchwork-Id: 13470430 X-Patchwork-Delegate: kuba@kernel.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qOBU2jKi" Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B36CC1B1 for ; Mon, 27 Nov 2023 16:27:24 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5cb92becbf6so72920547b3.2 for ; Mon, 27 Nov 2023 16:27:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701131244; x=1701736044; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Re5mrkQDhsmsBqEX/isMp/hVwh4xSdN/Lox5QrdMG60=; b=qOBU2jKinFzgaPEHxaJfe41F624KCMs++r0koVCoSIttPz8VqTZplqFAIk6ZEvxrKT QMngF+T0rI92WZEso1EqD/vBKLzCLpQjW5AnqoWTouPKNBlVwjrVjBd/N+9srXyb4xUa D6/bC7QNKBaRhG4ihnQmXS7MwmvW5KS8Wb14ZO29brny7mzrgrcTUS3hyiLVnpZVyHE4 WUP8V8QkkFXReU5SGaadiC/WibRAntKfmfDaOODly87bb5hRCsrFFvLgP4VJ66EDOood fVtPXdUubOwEuAZgplZoB5eJZLUAHo06wnNGZpPIXnlD3DnuG3MkUM7Sn4ohRpSf/kYr i9vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701131244; x=1701736044; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Re5mrkQDhsmsBqEX/isMp/hVwh4xSdN/Lox5QrdMG60=; b=rOtKLO7fRWEe+o+YkHOnDwBwzjUqoGuQcgmKcTAv+Ff4/j+g16ZpBzuSZWahsQlzm/ JImyU8CyuAyX6L9sX1Ks4HSXeVhft7Gi+fApcY2/PWIxgTb3m3p8nhgskIuPAjgBaJF9 N8y85Av9KLbovDNd31tBB0ZT2+z140VoMUWhZ9hTMiHoaOZ7+7zr4eH78y2OtEFT8YZN FJsjtiQFZQIa4v8tkfWefp6iYHENuaeN1z/IQx0ixAdFDmBo+rGbrQbaK/JMeEKkzwMX I3cvzTIKfEz9tBaQl49w4xF+AZ6lWSg0NWtPQ98DkSkuMbBA0Vb+QpW1eOztz4AHikhT Ia2A== X-Gm-Message-State: AOJu0YxI7usMIKK/6BHPgwuIfwBAtiVJ3tV9JrHZ0Ll5hP2a6D50I6Hk NvL58S4SQbGQyMjzttZg0LOtPPq3VvT9llfzJFMkg7eYj5/VZhyVPc7PQZ9Nhecy2n9e9Uvnv2J GEvpk7jGjGLuM7fF7wUJ1X1YIO7laEoZD5lfV0OeqUAc8E7Qp5VBLrE4CNT0aFwP7 X-Google-Smtp-Source: AGHT+IEzKG48RUQbySYjUN8Jw+eoWQEXFZD6O5//zOPMY9y71ZkaF4Ew5IgLGo/glvrfUYP/x+6DZz3rIPjl X-Received: from jfraker202.plv.corp.google.com ([2620:15c:11c:202:19d5:f826:3460:9345]) (user=jfraker job=sendgmr) by 2002:a05:690c:3105:b0:5cc:4276:1c6b with SMTP id fb5-20020a05690c310500b005cc42761c6bmr372945ywb.4.1701131243881; Mon, 27 Nov 2023 16:27:23 -0800 (PST) Date: Mon, 27 Nov 2023 16:26:45 -0800 In-Reply-To: <20231128002648.320892-1-jfraker@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231128002648.320892-1-jfraker@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231128002648.320892-3-jfraker@google.com> Subject: [PATCH net-next 2/5] gve: Deprecate adminq_pfn for pci revision 0x1. From: John Fraker To: netdev@vger.kernel.org Cc: John Fraker , Jordan Kimbrough , Willem de Bruijn X-Patchwork-Delegate: kuba@kernel.org adminq_pfn assumes a page size of 4k, causing this mechanism to break in kernels compiled with different page sizes. A new PCI device revision was needed for the device to be able to communicate with the driver how to set up the admin queue prior to having access to the admin queue. Signed-off-by: Jordan Kimbrough Signed-off-by: John Fraker Reviewed-by: Willem de Bruijn --- drivers/net/ethernet/google/gve/gve_adminq.c | 48 ++++++++++++++++++++++++++++----------- drivers/net/ethernet/google/gve/gve_register.h | 9 ++++++++ 2 files changed, 44 insertions(+), 13 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_adminq.c b/drivers/net/ethernet/google/gve/gve_adminq.c index d3f3a0152..f81ed6f62 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.c +++ b/drivers/net/ethernet/google/gve/gve_adminq.c @@ -225,9 +225,20 @@ int gve_adminq_alloc(struct device *dev, struct gve_priv *priv) priv->adminq_get_ptype_map_cnt = 0; /* Setup Admin queue with the device */ - iowrite32be(priv->adminq_bus_addr / PAGE_SIZE, - &priv->reg_bar0->adminq_pfn); - + if (priv->pdev->revision < 0x1) { + iowrite32be(priv->adminq_bus_addr / PAGE_SIZE, + &priv->reg_bar0->adminq_pfn); + } else { + iowrite16be(GVE_ADMINQ_BUFFER_SIZE, + &priv->reg_bar0->adminq_length); +#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT + iowrite32be(priv->adminq_bus_addr >> 32, + &priv->reg_bar0->adminq_base_address_hi); +#endif + iowrite32be(priv->adminq_bus_addr, + &priv->reg_bar0->adminq_base_address_lo); + iowrite32be(GVE_DRIVER_STATUS_RUN_MASK, &priv->reg_bar0->driver_status); + } gve_set_admin_queue_ok(priv); return 0; } @@ -237,16 +248,27 @@ void gve_adminq_release(struct gve_priv *priv) int i = 0; /* Tell the device the adminq is leaving */ - iowrite32be(0x0, &priv->reg_bar0->adminq_pfn); - while (ioread32be(&priv->reg_bar0->adminq_pfn)) { - /* If this is reached the device is unrecoverable and still - * holding memory. Continue looping to avoid memory corruption, - * but WARN so it is visible what is going on. - */ - if (i == GVE_MAX_ADMINQ_RELEASE_CHECK) - WARN(1, "Unrecoverable platform error!"); - i++; - msleep(GVE_ADMINQ_SLEEP_LEN); + if (priv->pdev->revision < 0x1) { + iowrite32be(0x0, &priv->reg_bar0->adminq_pfn); + while (ioread32be(&priv->reg_bar0->adminq_pfn)) { + /* If this is reached the device is unrecoverable and still + * holding memory. Continue looping to avoid memory corruption, + * but WARN so it is visible what is going on. + */ + if (i == GVE_MAX_ADMINQ_RELEASE_CHECK) + WARN(1, "Unrecoverable platform error!"); + i++; + msleep(GVE_ADMINQ_SLEEP_LEN); + } + } else { + iowrite32be(GVE_DRIVER_STATUS_RESET_MASK, &priv->reg_bar0->driver_status); + while (!(ioread32be(&priv->reg_bar0->device_status) + & GVE_DEVICE_STATUS_DEVICE_IS_RESET)) { + if (i == GVE_MAX_ADMINQ_RELEASE_CHECK) + WARN(1, "Unrecoverable platform error!"); + i++; + msleep(GVE_ADMINQ_SLEEP_LEN); + } } gve_clear_device_rings_ok(priv); gve_clear_device_resources_ok(priv); diff --git a/drivers/net/ethernet/google/gve/gve_register.h b/drivers/net/ethernet/google/gve/gve_register.h index fb655463c..8e72b9700 100644 --- a/drivers/net/ethernet/google/gve/gve_register.h +++ b/drivers/net/ethernet/google/gve/gve_register.h @@ -18,11 +18,20 @@ struct gve_registers { __be32 adminq_event_counter; u8 reserved[3]; u8 driver_version; + __be32 adminq_base_address_hi; + __be32 adminq_base_address_lo; + __be16 adminq_length; }; enum gve_device_status_flags { GVE_DEVICE_STATUS_RESET_MASK = BIT(1), GVE_DEVICE_STATUS_LINK_STATUS_MASK = BIT(2), GVE_DEVICE_STATUS_REPORT_STATS_MASK = BIT(3), + GVE_DEVICE_STATUS_DEVICE_IS_RESET = BIT(4), +}; + +enum gve_driver_status_flags { + GVE_DRIVER_STATUS_RUN_MASK = BIT(0), + GVE_DRIVER_STATUS_RESET_MASK = BIT(1), }; #endif /* _GVE_REGISTER_H_ */ From patchwork Tue Nov 28 00:26:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Fraker X-Patchwork-Id: 13470432 X-Patchwork-Delegate: kuba@kernel.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qBSInZYN" Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E67119D for ; Mon, 27 Nov 2023 16:27:27 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5cb6271b225so72898987b3.1 for ; Mon, 27 Nov 2023 16:27:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701131246; x=1701736046; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WivxbZ9dw1KXjCGb/iIEv0+mZOVEPFRyY+vy9pp8ozo=; b=qBSInZYNVFOvVGPwpSTO1V9PRe2SGj20FiseKo2jRaGZEBceYkG4aR5U/HANdUeOf8 bA0vL6QFHrJcTiKZcDtaz2zsln+MW3cRa8bbdGpya6pPJI8g+j2Rq/vWTJNqV0iFRGJb 26DSXIrAZiJ8bjdJF6l3nhyKoy40vpF1p+sHPa1nioFGbMKQkH901bvQgk82j7Oqw4fQ 5PCuR+AMFOjk9P4U5hK0MrFBdfgM0h2YNX37xF28Cv67Y8ndjMU8QYQxfEwME9RsL81T EHJ1DUkQlQ+SBlYxp+gA7FPjGpW+vmIZaO01N/R09VCLraztkRbvI1rnARe6OzjjXmSA KtKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701131246; x=1701736046; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WivxbZ9dw1KXjCGb/iIEv0+mZOVEPFRyY+vy9pp8ozo=; b=J77lSaJWD/2fMC0+/MfPSG1QI7+f7XtWg1qJhJ5QChrDckZ3YK1aT+cEP9HQG0G8gK r/CiZmq1/ZEdoXS0Q2EEYZfeF/rxdlLWhymp/njATnMO2jWyeiuGzxxwPbW5gSSbJy8i CufV/oZ/A2P1sEJ2wJgVGQiErxCWrJarNRrMxFzbzRayRICaI5SCvVrqu2scfgcC6g6I BxlWgw/7OJ2DqXIUznu1mFWsod75YVSTfKShrzfoi0ufCJFHhh4bBmEi0+c7texDFK8D vzu4jJaWX0/SiLZBDId0+M2ShrK7bU8P4aaerB3+MScbZthjtuACpcorqoKC6nnS4Pw4 BKbQ== X-Gm-Message-State: AOJu0YyiDcVe61e/+n8j0lkze1hxHzhP9JVhTfZKUUWMqJV8GegR3ENO F8y/O9SGPCBmLj05MtGoSz9+fLvFakLScinaBIrxPRrnyzEeLRPOS00wnvpOojt2Rpihm5aJw9u wKtuiUIUbElecYEq7GeArKoWBH7y1Vnpk/ymu7NrYkFZp/xwEoYUxy5LpdXQa6YaF X-Google-Smtp-Source: AGHT+IEoAJxrDCMkPF1DVUUGDvKQbPlNETXVO9knOBDKuwOyWrBd1fk+95/5LJ/j3L0BnUJ7t2YE7z7t6B3B X-Received: from jfraker202.plv.corp.google.com ([2620:15c:11c:202:19d5:f826:3460:9345]) (user=jfraker job=sendgmr) by 2002:a05:690c:3209:b0:5ca:f1d:6131 with SMTP id ff9-20020a05690c320900b005ca0f1d6131mr379437ywb.7.1701131246314; Mon, 27 Nov 2023 16:27:26 -0800 (PST) Date: Mon, 27 Nov 2023 16:26:46 -0800 In-Reply-To: <20231128002648.320892-1-jfraker@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231128002648.320892-1-jfraker@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231128002648.320892-4-jfraker@google.com> Subject: [PATCH net-next 3/5] gve: Remove obsolete checks that rely on page size. From: John Fraker To: netdev@vger.kernel.org Cc: John Fraker , Jordan Kimbrough , Willem de Bruijn X-Patchwork-Delegate: kuba@kernel.org These checks are safe to remove as they are no longer enforced by the backend. Retaining them would require updating them to work differently with page sizes larger than 4k. Signed-off-by: Jordan Kimbrough Signed-off-by: John Fraker Reviewed-by: Willem de Bruijn --- drivers/net/ethernet/google/gve/gve_adminq.c | 11 ----------- drivers/net/ethernet/google/gve/gve_rx.c | 8 +------- 2 files changed, 1 insertion(+), 18 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve_adminq.c b/drivers/net/ethernet/google/gve/gve_adminq.c index f81ed6f62..bebb7ed11 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.c +++ b/drivers/net/ethernet/google/gve/gve_adminq.c @@ -727,18 +727,7 @@ static int gve_set_desc_cnt(struct gve_priv *priv, struct gve_device_descriptor *descriptor) { priv->tx_desc_cnt = be16_to_cpu(descriptor->tx_queue_entries); - if (priv->tx_desc_cnt * sizeof(priv->tx->desc[0]) < PAGE_SIZE) { - dev_err(&priv->pdev->dev, "Tx desc count %d too low\n", - priv->tx_desc_cnt); - return -EINVAL; - } priv->rx_desc_cnt = be16_to_cpu(descriptor->rx_queue_entries); - if (priv->rx_desc_cnt * sizeof(priv->rx->desc.desc_ring[0]) - < PAGE_SIZE) { - dev_err(&priv->pdev->dev, "Rx desc count %d too low\n", - priv->rx_desc_cnt); - return -EINVAL; - } return 0; } diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c index 3c8a80d18..3d6b26ac6 100644 --- a/drivers/net/ethernet/google/gve/gve_rx.c +++ b/drivers/net/ethernet/google/gve/gve_rx.c @@ -210,9 +210,9 @@ static int gve_rx_alloc_ring(struct gve_priv *priv, int idx) { struct gve_rx_ring *rx = &priv->rx[idx]; struct device *hdev = &priv->pdev->dev; - u32 slots, npages; int filled_pages; size_t bytes; + u32 slots; int err; netif_dbg(priv, drv, priv->dev, "allocating rx ring\n"); @@ -269,12 +269,6 @@ static int gve_rx_alloc_ring(struct gve_priv *priv, int idx) /* alloc rx desc ring */ bytes = sizeof(struct gve_rx_desc) * priv->rx_desc_cnt; - npages = bytes / PAGE_SIZE; - if (npages * PAGE_SIZE != bytes) { - err = -EIO; - goto abort_with_q_resources; - } - rx->desc.desc_ring = dma_alloc_coherent(hdev, bytes, &rx->desc.bus, GFP_KERNEL); if (!rx->desc.desc_ring) { From patchwork Tue Nov 28 00:26:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Fraker X-Patchwork-Id: 13470433 X-Patchwork-Delegate: kuba@kernel.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MYTMihHE" Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E940C1 for ; Mon, 27 Nov 2023 16:27:29 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-5c19a2f606dso4900889a12.3 for ; Mon, 27 Nov 2023 16:27:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701131248; x=1701736048; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BYowvgqz4YCAb9fWpxc3YDJE9fwRyx0aEGPB1qlGo5k=; b=MYTMihHEjlwf/Qjm8esxvpu5tQ3HZt9O3UYURJsSqbesEEQeWDShlHZYJ6FIcgaQ4n HBQD2PRWR7tlGu2LL5JpOtbZ69lJQCn0ZLwrQfP0v4vElSziCpImo0fX2KGVciKYcS8e cBTH9msLGj99Wv8Ly+PQyrc4zU5OoFO23Icw9mqGbwuNOIQ/m63q+6yKKst9TthFGy7b 54O9ig5S1qeHuomN/lazZEBAqSoswk4VvkNkhg5GabANcgbJy0BYw6nGvUYzFIBIeP3k w0QSjcvQcFp809ZFtP1Sy83E0dryzgCSfdLF25w/LBCX+K7u6/kimqoAXE65Zy1cCM12 EZ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701131248; x=1701736048; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BYowvgqz4YCAb9fWpxc3YDJE9fwRyx0aEGPB1qlGo5k=; b=CUv8Sb6Ze1OmQNNxUta+q+GQdtlryXBeBjKwWIfHhO3hFdEECqZ8HgXkWzF1f27INT FdghXW1CRGwLoXgrMKyUISPlfBlKWvpMaoaWMJ4bX5GBLNKkErxJJzBmIUCX2bbajQ+p eJka114XrxIaGYbL7OEcz6OSJ40Z4kjzEcSsdnwWksH6USEb9I1RbCK80qAIjh6oURC0 ntJ46NVRHCP6tFf3YXDN5e2n9PBDnoDjB6vXvyp4+ijRGY2sn+xZn05w3PDiiouSuIdg X0E1eY1Qek7nOaS6aM5OW/EmTZzpTePiI/DP7KjWF8x+pBCNKd3T2ZMdeuagLDJBXABZ J42Q== X-Gm-Message-State: AOJu0YxcCI/Wa4C9kv3djqSWXJuOEeY+MBkZumHG1CcC4Tdl8s5Of0yU P/o1ihUMEWihE+PqGQhMAfKWynMO56KMakcXdnuPHiJE5FYPEIliDLUHHg44WlsJNNo1Wl/cicW YUIBANxLi+59TOrg/pVhUAzjkYEnQqWW/jo55I4eQ1KijCdlrYcgHqU2i1OZFCZV0 X-Google-Smtp-Source: AGHT+IESuj4pI5rCCot9Y2YIdnfc3JPd7ehNjMvHeqJYc74hWhgmPnGPQ/CrIj7vZVrA8Seb/3wpolt4EOHk X-Received: from jfraker202.plv.corp.google.com ([2620:15c:11c:202:19d5:f826:3460:9345]) (user=jfraker job=sendgmr) by 2002:a63:f608:0:b0:5be:4aa:616c with SMTP id m8-20020a63f608000000b005be04aa616cmr2220213pgh.4.1701131248518; Mon, 27 Nov 2023 16:27:28 -0800 (PST) Date: Mon, 27 Nov 2023 16:26:47 -0800 In-Reply-To: <20231128002648.320892-1-jfraker@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231128002648.320892-1-jfraker@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231128002648.320892-5-jfraker@google.com> Subject: [PATCH net-next 4/5] gve: Add page size register to the register_page_list command. From: John Fraker To: netdev@vger.kernel.org Cc: John Fraker , Jordan Kimbrough , Willem de Bruijn X-Patchwork-Delegate: kuba@kernel.org This register is required on platforms with page sizes greater than 4k. This is because the tx side of the driver vmaps the entire queue page list of pages into a single flat address space, then uses the entire space. Without communicating the guest page size to the backend, the backend will only access the first 4k of each page in the queue page list. Signed-off-by: Jordan Kimbrough Signed-off-by: John Fraker Reviewed-by: Willem de Bruijn --- drivers/net/ethernet/google/gve/gve_adminq.c | 1 + drivers/net/ethernet/google/gve/gve_adminq.h | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/google/gve/gve_adminq.c b/drivers/net/ethernet/google/gve/gve_adminq.c index bebb7ed11..12fbd723e 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.c +++ b/drivers/net/ethernet/google/gve/gve_adminq.c @@ -917,6 +917,7 @@ int gve_adminq_register_page_list(struct gve_priv *priv, .page_list_id = cpu_to_be32(qpl->id), .num_pages = cpu_to_be32(num_entries), .page_address_list_addr = cpu_to_be64(page_list_bus), + .page_size = cpu_to_be64(PAGE_SIZE), }; err = gve_adminq_execute_cmd(priv, &cmd); diff --git a/drivers/net/ethernet/google/gve/gve_adminq.h b/drivers/net/ethernet/google/gve/gve_adminq.h index 38a22279e..5865ccdcc 100644 --- a/drivers/net/ethernet/google/gve/gve_adminq.h +++ b/drivers/net/ethernet/google/gve/gve_adminq.h @@ -219,9 +219,10 @@ struct gve_adminq_register_page_list { __be32 page_list_id; __be32 num_pages; __be64 page_address_list_addr; + __be64 page_size; }; -static_assert(sizeof(struct gve_adminq_register_page_list) == 16); +static_assert(sizeof(struct gve_adminq_register_page_list) == 24); struct gve_adminq_unregister_page_list { __be32 page_list_id; From patchwork Tue Nov 28 00:26:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Fraker X-Patchwork-Id: 13470434 X-Patchwork-Delegate: kuba@kernel.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1ZlfvYMQ" Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 681D91A2 for ; Mon, 27 Nov 2023 16:27:31 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-6cbcd3e9758so5599574b3a.2 for ; Mon, 27 Nov 2023 16:27:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701131251; x=1701736051; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=grvOf3tFsqgV1RjCdzyibD/4PsRcEMJwLE5B/qmXM0I=; b=1ZlfvYMQcKTx/lANStip+nAaa4xCbpKP73sqoUnCl9QCKbmkAljQZt/4N7bgE0kpkV HhHYvCHkDxefFU8C0BeyKCb35BR8ez7B/ciGYWygNWQY5D54UANi+k04YfPA8obJ+PNz 8ozpKdXeG5HtGRZ0olFz9C31r2khmatFIEDQyCyDOFAQ8+eb2bRRazbCjuCrLMDCBIlX K4wGdLKseLPkl4ZJ9xjOu3lTT1e6uXqLn2Pf1CTJhKXobdbzaFMnayxFg3J5spNy4CHH mHWNUK5EcA/a8noafjkqcOhobhc2UbjJ3wibNS+VTdz+qTN5RgNc4ntexkR0XZoS01ef SQWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701131251; x=1701736051; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=grvOf3tFsqgV1RjCdzyibD/4PsRcEMJwLE5B/qmXM0I=; b=XHhjV4xMQjdFTJOzWelHdH6csNyY4qN/vtkuFnbIF5BMlfmgnHnm1Fljo/2ZefbGLv KAOv7YygBN8LBdnGE5ETGxhRbQ/AROiRb7NggJk/0j+xXmMxz7/W53R3tmWpW67SwQCK g7sYwB0Fb8SdVJ8C1RbG1wAWLeIjv7m5NSyqyVdOthqIiBz+q3UbjRwaJabkHDcITGoq Nh7s5JuQL1Nmr8XPy5soluDuhrUBetAFMIeKXSOPW+OgLB9h0VbgkuxHA0+rpKWN+t7L 5spTuwdWwZu64/p9YYmrwc30ccM6TKTGoV95/VolaJoUVYxpcdVUq6SESfmHsF/Jtn5Y 8RXA== X-Gm-Message-State: AOJu0Ywo6hIrqdHKbskiPle/ppA4MQN0lA0zNHepGE0rLryg01Hd7iBD gk3EqjQzxkxpcj5LA2IyvU6A7PWlDa3f/64kQzn2OGl/Nvjef+1RoWM/hu74WGZNWE1erVEWimQ jh9ZOZJZaaKk2M/OFwI98Ngu2pPVGyHKgeVm8kHYXNPnxzx3O/Bwxe/80aSRj90kG X-Google-Smtp-Source: AGHT+IH8/nM7rG5fsjNfIopxU2v+zO8QT3lDhTdWMR5GcPUHoTFlJ9lR8zawFnHi8ze8BIUo5guw3VSLay/v X-Received: from jfraker202.plv.corp.google.com ([2620:15c:11c:202:19d5:f826:3460:9345]) (user=jfraker job=sendgmr) by 2002:a05:6a00:18a4:b0:6cd:852a:8a79 with SMTP id x36-20020a056a0018a400b006cd852a8a79mr1946455pfh.6.1701131250846; Mon, 27 Nov 2023 16:27:30 -0800 (PST) Date: Mon, 27 Nov 2023 16:26:48 -0800 In-Reply-To: <20231128002648.320892-1-jfraker@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231128002648.320892-1-jfraker@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231128002648.320892-6-jfraker@google.com> Subject: [PATCH net-next 5/5] gve: Remove dependency on 4k page size. From: John Fraker To: netdev@vger.kernel.org Cc: John Fraker , Jordan Kimbrough , Willem de Bruijn X-Patchwork-Delegate: kuba@kernel.org Prior to this change, gve crashes when attempting to run in kernels with page sizes other than 4k. This change removes unnecessary references to PAGE_SIZE and replaces them with more meaningful constants. Signed-off-by: Jordan Kimbrough Signed-off-by: John Fraker Reviewed-by: Willem de Bruijn --- drivers/net/ethernet/google/gve/gve.h | 4 +++- drivers/net/ethernet/google/gve/gve_ethtool.c | 2 +- drivers/net/ethernet/google/gve/gve_main.c | 4 ++-- drivers/net/ethernet/google/gve/gve_rx.c | 9 ++++----- drivers/net/ethernet/google/gve/gve_tx.c | 2 +- 5 files changed, 11 insertions(+), 10 deletions(-) diff --git a/drivers/net/ethernet/google/gve/gve.h b/drivers/net/ethernet/google/gve/gve.h index abc0c708b..b80349154 100644 --- a/drivers/net/ethernet/google/gve/gve.h +++ b/drivers/net/ethernet/google/gve/gve.h @@ -49,7 +49,9 @@ /* PTYPEs are always 10 bits. */ #define GVE_NUM_PTYPES 1024 -#define GVE_RX_BUFFER_SIZE_DQO 2048 +#define GVE_DEFAULT_RX_BUFFER_SIZE 2048 + +#define GVE_DEFAULT_RX_BUFFER_OFFSET 2048 #define GVE_XDP_ACTIONS 5 diff --git a/drivers/net/ethernet/google/gve/gve_ethtool.c b/drivers/net/ethernet/google/gve/gve_ethtool.c index 233e59469..e5397aa1e 100644 --- a/drivers/net/ethernet/google/gve/gve_ethtool.c +++ b/drivers/net/ethernet/google/gve/gve_ethtool.c @@ -519,7 +519,7 @@ static int gve_set_tunable(struct net_device *netdev, case ETHTOOL_RX_COPYBREAK: { u32 max_copybreak = gve_is_gqi(priv) ? - (PAGE_SIZE / 2) : priv->data_buffer_size_dqo; + GVE_DEFAULT_RX_BUFFER_SIZE : priv->data_buffer_size_dqo; len = *(u32 *)value; if (len > max_copybreak) diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index cc169748f..619bf63ec 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1328,7 +1328,7 @@ static int gve_open(struct net_device *dev) /* Hard code this for now. This may be tuned in the future for * performance. */ - priv->data_buffer_size_dqo = GVE_RX_BUFFER_SIZE_DQO; + priv->data_buffer_size_dqo = GVE_DEFAULT_RX_BUFFER_SIZE; } err = gve_create_rings(priv); if (err) @@ -1664,7 +1664,7 @@ static int verify_xdp_configuration(struct net_device *dev) return -EOPNOTSUPP; } - if (dev->mtu > (PAGE_SIZE / 2) - sizeof(struct ethhdr) - GVE_RX_PAD) { + if (dev->mtu > GVE_DEFAULT_RX_BUFFER_SIZE - sizeof(struct ethhdr) - GVE_RX_PAD) { netdev_warn(dev, "XDP is not supported for mtu %d.\n", dev->mtu); return -EOPNOTSUPP; diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c index 3d6b26ac6..3cb3a9ac6 100644 --- a/drivers/net/ethernet/google/gve/gve_rx.c +++ b/drivers/net/ethernet/google/gve/gve_rx.c @@ -282,7 +282,7 @@ static int gve_rx_alloc_ring(struct gve_priv *priv, int idx) /* Allocating half-page buffers allows page-flipping which is faster * than copying or allocating new pages. */ - rx->packet_buffer_size = PAGE_SIZE / 2; + rx->packet_buffer_size = GVE_DEFAULT_RX_BUFFER_SIZE; gve_rx_ctx_clear(&rx->ctx); gve_rx_add_to_block(priv, idx); @@ -398,10 +398,10 @@ static struct sk_buff *gve_rx_add_frags(struct napi_struct *napi, static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info, __be64 *slot_addr) { - const __be64 offset = cpu_to_be64(PAGE_SIZE / 2); + const __be64 offset = cpu_to_be64(GVE_DEFAULT_RX_BUFFER_OFFSET); /* "flip" to other packet buffer on this page */ - page_info->page_offset ^= PAGE_SIZE / 2; + page_info->page_offset ^= GVE_DEFAULT_RX_BUFFER_OFFSET; *(slot_addr) ^= offset; } @@ -506,8 +506,7 @@ static struct sk_buff *gve_rx_copy_to_pool(struct gve_rx_ring *rx, return NULL; gve_dec_pagecnt_bias(copy_page_info); - copy_page_info->page_offset += rx->packet_buffer_size; - copy_page_info->page_offset &= (PAGE_SIZE - 1); + copy_page_info->page_offset ^= GVE_DEFAULT_RX_BUFFER_OFFSET; if (copy_page_info->can_flip) { /* We have used both halves of this copy page, it diff --git a/drivers/net/ethernet/google/gve/gve_tx.c b/drivers/net/ethernet/google/gve/gve_tx.c index 9f6ffc4a5..07ba12478 100644 --- a/drivers/net/ethernet/google/gve/gve_tx.c +++ b/drivers/net/ethernet/google/gve/gve_tx.c @@ -819,7 +819,7 @@ int gve_xdp_xmit_one(struct gve_priv *priv, struct gve_tx_ring *tx, return 0; } -#define GVE_TX_START_THRESH PAGE_SIZE +#define GVE_TX_START_THRESH 4096 static int gve_clean_tx_done(struct gve_priv *priv, struct gve_tx_ring *tx, u32 to_do, bool try_to_wake)