From patchwork Fri Aug 10 02:14:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10562187 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7001F1057 for ; Fri, 10 Aug 2018 02:15:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 623532BBBA for ; Fri, 10 Aug 2018 02:15:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 568A42BBD7; Fri, 10 Aug 2018 02:15:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B79F12BBAF for ; Fri, 10 Aug 2018 02:14:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725198AbeHJEmg (ORCPT ); Fri, 10 Aug 2018 00:42:36 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:44482 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727143AbeHJEmg (ORCPT ); Fri, 10 Aug 2018 00:42:36 -0400 Received: by mail-pf1-f194.google.com with SMTP id k21-v6so3742846pff.11 for ; Thu, 09 Aug 2018 19:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1cW+QbGnpJp4x6th2sj+CnM5URXVJOPmFY0wziUhEY4=; b=a/NMTnX9caKwzbkJJCXl8FXJ30vyHazUG1i/X5qlrdgFYQ3xQ4HeZHOJBLBed36tAl lDbc+6jadsQYQRdoNw6d9H4gjeWB5UF9nm0YLQ4neC823ImetIdqyYx5GspX/TJXo0/p hQM0LPUXfeUiELU9dPqonXhbzWOIUpYNP4T0X9bEOsiAYa+CKHYVofXuEhuc0CtVwPs1 PWVGxu1OoAG00B3ORxzXS16ktkWnMnRxv9Rk7a2UIdd7od5k9UTw2FsTP25QgwHGwCWr A3QY3GNvG86mVYlMVtDV9dTuy8nh9XAg2HCtA/D/UtWiyin7+QpPWUD9Q3EpxaVhDMeq xRHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1cW+QbGnpJp4x6th2sj+CnM5URXVJOPmFY0wziUhEY4=; b=uEm1xKzNJT6wYvF7FjTRTJxRPHQteNzCRLfekU55KVV6Of5PWQqv4GMCV+DEc9hk5g WUqdICvWyFl/4CP63K2xalWUSnmeTq9GJVulLP3SUvExwzHsHtWx1xPFc84imkOtXHGu GjMwO89kx3/+dy+7OKHn5ohuD9mYGYWU1sW9L+E+nbVkMgvpI64/Ta9sNsZVjscTjeGb jUou55sKDCbv2M9hLdiLbzBsq/hWhpmSS+0QcY697MOlZmQMDioqYAXx0pwR/gxg/iNo 1lQr7O5YonBB7l9zlzi2qk72JYKmWGSX0l1RPK3dRvhLTfIOpa2YIjPvbeuzih3taron g22w== X-Gm-Message-State: AOUpUlEw39LZb72f2RynIs0vcH7N2klyf11obHuIgi2By7HuFNnwQn8J 9mAEg2WXIYFIck98UnuRHGxdYIRG8q0= X-Google-Smtp-Source: AA+uWPzhT8BVgCG0fsKsGL7VRdKe20vWmj+kf9eyEQ6r50+EoypiuJIIWdU77HR27o+POBSQOoxq+w== X-Received: by 2002:a62:cd82:: with SMTP id o124-v6mr4864972pfg.206.1533867295728; Thu, 09 Aug 2018 19:14:55 -0700 (PDT) Received: from ziepe.ca (S010614cc2056d97f.ed.shawcable.net. [174.3.196.123]) by smtp.gmail.com with ESMTPSA id j1-v6sm15287943pfk.125.2018.08.09.19.14.51 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 09 Aug 2018 19:14:52 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.90_1) (envelope-from ) id 1fnwx1-0003gf-FJ; Thu, 09 Aug 2018 20:14:51 -0600 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org, Leon Romanovsky , "Guy Levi(SW)" , Yishai Hadas , "Ruhl, Michael J" Cc: Jason Gunthorpe Subject: [PATCH v1 06/10] IB/uverbs: Add a simple allocator to uverbs_attr_bundle Date: Thu, 9 Aug 2018 20:14:40 -0600 Message-Id: <20180810021444.14014-7-jgg@ziepe.ca> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180810021444.14014-1-jgg@ziepe.ca> References: <20180810021444.14014-1-jgg@ziepe.ca> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jason Gunthorpe This is similar in spirit to devm, it keeps track of any allocations linked to this method call and ensures they are all freed when the method exits. Further, if there is space in the internal/onstack buffer then the allocator will hand out that memory and avoid an expensive call to kalloc/kfree in the syscall path. Signed-off-by: Jason Gunthorpe Reviewed-by: Leon Romanovsky --- drivers/infiniband/core/uverbs_ioctl.c | 106 ++++++++++++++++++++----- include/rdma/uverbs_ioctl.h | 24 ++++++ 2 files changed, 110 insertions(+), 20 deletions(-) diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index c5aa64ee099efe..820e7ff3066751 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -35,7 +35,18 @@ #include "rdma_core.h" #include "uverbs.h" +struct bundle_alloc_head { + struct bundle_alloc_head *next; + u8 data[]; +}; + struct bundle_priv { + /* Must be first */ + struct bundle_alloc_head alloc_head; + struct bundle_alloc_head *allocated_mem; + size_t internal_avail; + size_t internal_used; + struct ib_uverbs_attr __user *user_attrs; struct ib_uverbs_attr *uattrs; struct uverbs_obj_attr *destroy_attr; @@ -45,8 +56,53 @@ struct bundle_priv { * internal_buffer. */ struct uverbs_attr_bundle bundle; + u64 internal_buffer[32]; }; +/** + * uverbs_alloc() - Quickly allocate memory for use with a bundle + * @bundle: The bundle + * @size: Number of bytes to allocate + * @flags: Allocator flags + * + * The bundle allocator is intended for allocations that are connected with + * processing the system call related to the bundle. The allocated memory is + * always freed once the system call completes, and cannot be freed any other + * way. + * + * This tries to use a small pool of pre-allocated memory for performance. + */ +__malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, + gfp_t flags) +{ + struct bundle_priv *pbundle = + container_of(bundle, struct bundle_priv, bundle); + size_t new_used; + void *res; + + if (check_add_overflow(size, pbundle->internal_used, &new_used)) + return ERR_PTR(-EINVAL); + + if (new_used > pbundle->internal_avail) { + struct bundle_alloc_head *buf; + + buf = kvmalloc(struct_size(buf, data, size), flags); + if (!buf) + return ERR_PTR(-ENOMEM); + buf->next = pbundle->allocated_mem; + pbundle->allocated_mem = buf; + return buf->data; + } + + res = (void *)pbundle->internal_buffer + pbundle->internal_used; + pbundle->internal_used = + ALIGN(new_used, sizeof(*pbundle->internal_buffer)); + if (flags & __GFP_ZERO) + memset(res, 0, size); + return res; +} +EXPORT_SYMBOL(_uverbs_alloc); + static bool uverbs_is_attr_cleared(const struct ib_uverbs_attr *uattr, u16 len) { @@ -129,17 +185,15 @@ static int uverbs_process_attr(struct bundle_priv *pbundle, if (val_spec->alloc_and_copy && !uverbs_attr_ptr_is_inline(e)) { void *p; - p = kvmalloc(uattr->len, GFP_KERNEL); - if (!p) - return -ENOMEM; + p = uverbs_alloc(&pbundle->bundle, uattr->len); + if (IS_ERR(p)) + return PTR_ERR(p); e->ptr_attr.ptr = p; if (copy_from_user(p, u64_to_user_ptr(uattr->data), - uattr->len)) { - kvfree(p); + uattr->len)) return -EFAULT; - } } else { e->ptr_attr.data = uattr->data; } @@ -234,10 +288,6 @@ static int uverbs_finalize_attrs(struct bundle_priv *pbundle, spec->u.obj.access, commit); if (!ret) ret = current_ret; - } else if (spec->type == UVERBS_ATTR_TYPE_PTR_IN && - spec->alloc_and_copy && - !uverbs_attr_ptr_is_inline(attr)) { - kvfree(attr->ptr_attr.ptr); } } } @@ -372,7 +422,15 @@ static int uverbs_handle_method(size_t num_uattrs, return ret ? ret : finalize_ret; } -#define UVERBS_OPTIMIZE_USING_STACK_SZ 256 +static void bundle_destroy(struct bundle_priv *pbundle) +{ + struct bundle_alloc_head *memblock; + + for (memblock = pbundle->allocated_mem; memblock; + memblock = memblock->next) + kvfree(memblock); +} + static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, struct ib_uverbs_file *file, struct ib_uverbs_ioctl_hdr *hdr, @@ -382,11 +440,11 @@ static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, const struct uverbs_method_spec *method_spec; long err = 0; unsigned int i; + struct bundle_priv onstack_pbundle; struct bundle_priv *ctx; struct uverbs_attr *curr_attr; unsigned long *curr_bitmap; size_t ctx_size; - uintptr_t data[UVERBS_OPTIMIZE_USING_STACK_SZ / sizeof(uintptr_t)]; if (hdr->driver_id != ib_dev->driver_id) return -EINVAL; @@ -399,7 +457,7 @@ static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, if (!method_spec) return -EPROTONOSUPPORT; - ctx_size = sizeof(*ctx) + + ctx_size = sizeof(*ctx) - sizeof(ctx->internal_buffer) + sizeof(struct uverbs_attr_bundle_hash) * method_spec->num_buckets + sizeof(*ctx->uattrs) * hdr->num_attrs + sizeof(*ctx->bundle.hash[0].attrs) * @@ -408,17 +466,26 @@ static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, (method_spec->num_child_attrs / BITS_PER_LONG + method_spec->num_buckets); - if (ctx_size <= UVERBS_OPTIMIZE_USING_STACK_SZ) - ctx = (void *)data; - if (!ctx) + if (ctx_size <= sizeof(onstack_pbundle)) { + ctx = &onstack_pbundle; + ctx->internal_avail = + sizeof(onstack_pbundle) - + offsetof(struct bundle_priv, internal_buffer); + ctx->allocated_mem = NULL; + } else { ctx = kmalloc(ctx_size, GFP_KERNEL); - if (!ctx) - return -ENOMEM; + if (!ctx) + return -ENOMEM; + ctx->internal_avail = 0; + ctx->alloc_head.next = NULL; + ctx->allocated_mem = &ctx->alloc_head; + } ctx->uattrs = (void *)(ctx + 1) + (sizeof(ctx->bundle.hash[0]) * method_spec->num_buckets); curr_attr = (void *)(ctx->uattrs + hdr->num_attrs); curr_bitmap = (void *)(curr_attr + method_spec->num_child_attrs); + ctx->internal_used = ALIGN(ctx_size, sizeof(*ctx->internal_buffer)); /* * We just fill the pointers and num_attrs here. The data itself will be @@ -462,8 +529,7 @@ static long ib_uverbs_cmd_verbs(struct ib_device *ib_dev, err = -EINVAL; } out: - if (ctx != (void *)data) - kfree(ctx); + bundle_destroy(ctx); return err; } diff --git a/include/rdma/uverbs_ioctl.h b/include/rdma/uverbs_ioctl.h index ecf028446cdf53..1dbf663f7f436a 100644 --- a/include/rdma/uverbs_ioctl.h +++ b/include/rdma/uverbs_ioctl.h @@ -651,6 +651,20 @@ int uverbs_get_flags32(u32 *to, const struct uverbs_attr_bundle *attrs_bundle, size_t idx, u64 allowed_bits); int uverbs_copy_to(const struct uverbs_attr_bundle *attrs_bundle, size_t idx, const void *from, size_t size); +__malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, + gfp_t flags); + +static inline __malloc void *uverbs_alloc(struct uverbs_attr_bundle *bundle, + size_t size) +{ + return _uverbs_alloc(bundle, size, GFP_KERNEL); +} + +static inline __malloc void *uverbs_zalloc(struct uverbs_attr_bundle *bundle, + size_t size) +{ + return _uverbs_alloc(bundle, size, GFP_KERNEL | __GFP_ZERO); +} #else static inline int uverbs_get_flags64(u64 *to, const struct uverbs_attr_bundle *attrs_bundle, @@ -669,6 +683,16 @@ static inline int uverbs_copy_to(const struct uverbs_attr_bundle *attrs_bundle, { return -EINVAL; } +static inline __malloc void *uverbs_alloc(struct uverbs_attr_bundle *bundle, + size_t size) +{ + return ERR_PTR(-EINVAL); +} +static inline __malloc void *uverbs_zalloc(struct uverbs_attr_bundle *bundle, + size_t size) +{ + return ERR_PTR(-EINVAL); +} #endif /* =================================================