| Message ID | 20221207085752.82458-3-michaelgur@nvidia.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | RDMA/mlx5: Switch MR cache to use RB-tree | expand |
On Wed, Dec 07, 2022 at 10:57:48AM +0200, Michael Guralnik wrote: > From: Aharon Landau <aharonl@nvidia.com> > > Explicit ODP mkey doesn't have unique properties. There is no need to > devote to it a special entry. Removing it. This isn't quite true, the purpose of this entry to is to create a cache block of order 30, which is higher than the usual max of 22 Though, I'm not really sure we should even be caching such huge MRs, I can understand why ODP would want this cache, at least for fairly short term. At the end of this do we still have caching for order 30 MRs? Also, is the MTT vs KSM access_mode UMRable? Jason
On Wed, Dec 07, 2022 at 10:57:48AM +0200, Michael Guralnik wrote: > From: Aharon Landau <aharonl@nvidia.com> > > Explicit ODP mkey doesn't have unique properties. There is no need to > devote to it a special entry. Removing it. > > Signed-off-by: Aharon Landau <aharonl@nvidia.com> > --- > drivers/infiniband/hw/mlx5/odp.c | 19 ++++--------------- > include/linux/mlx5/driver.h | 1 - > 2 files changed, 4 insertions(+), 16 deletions(-) > > diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c > index e9a29adef7dc..71b733fcac37 100644 > --- a/drivers/infiniband/hw/mlx5/odp.c > +++ b/drivers/infiniband/hw/mlx5/odp.c > @@ -406,6 +406,7 @@ static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev, > static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, > unsigned long idx) > { > + int order = order_base_2(MLX5_IMR_MTT_ENTRIES); unsigned int > struct mlx5_ib_dev *dev = mr_to_mdev(imr); > struct ib_umem_odp *odp; > struct mlx5_ib_mr *mr; > @@ -418,7 +419,7 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, > if (IS_ERR(odp)) > return ERR_CAST(odp); > > - mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[MLX5_IMR_MTT_CACHE_ENTRY], > + mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[order], Okay, I get it now, this is actually just order 18, so it is covered by the existing order. You should clarify the commit message that it is the same as the order 18 entry. Also add a compile time assertion here: static_assert(order < MKEY_CACHE_LAST_STD_ENTRY); Jason
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index e9a29adef7dc..71b733fcac37 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -406,6 +406,7 @@ static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev, static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, unsigned long idx) { + int order = order_base_2(MLX5_IMR_MTT_ENTRIES); struct mlx5_ib_dev *dev = mr_to_mdev(imr); struct ib_umem_odp *odp; struct mlx5_ib_mr *mr; @@ -418,7 +419,7 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr, if (IS_ERR(odp)) return ERR_CAST(odp); - mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[MLX5_IMR_MTT_CACHE_ENTRY], + mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[order], imr->access_flags); if (IS_ERR(mr)) { ib_umem_odp_release(odp); @@ -1592,20 +1593,8 @@ void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) { if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT)) return; - - switch (ent->order - 2) { - case MLX5_IMR_MTT_CACHE_ENTRY: - ent->ndescs = MLX5_IMR_MTT_ENTRIES; - ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT; - ent->limit = 0; - break; - - case MLX5_IMR_KSM_CACHE_ENTRY: - ent->ndescs = mlx5_imr_ksm_entries; - ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; - ent->limit = 0; - break; - } + ent->ndescs = mlx5_imr_ksm_entries; + ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM; } static const struct ib_device_ops mlx5_ib_dev_odp_ops = { diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index af2ceb4160bc..cdf6ed25778e 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -736,7 +736,6 @@ enum { enum { MKEY_CACHE_LAST_STD_ENTRY = 20, - MLX5_IMR_MTT_CACHE_ENTRY, MLX5_IMR_KSM_CACHE_ENTRY, MAX_MKEY_CACHE_ENTRIES };