From patchwork Tue Jan 24 05:42:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13113519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F57AC54EED for ; Tue, 24 Jan 2023 05:43:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232692AbjAXFns (ORCPT ); Tue, 24 Jan 2023 00:43:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232036AbjAXFnq (ORCPT ); Tue, 24 Jan 2023 00:43:46 -0500 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2065.outbound.protection.outlook.com [40.107.94.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B16CD3B0FD; Mon, 23 Jan 2023 21:43:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kFiXQfbvBkmWql4u3YiSkcsVXxx4fdxDzFh+cfG0F7YG8RKF2myht+SFIZWPiJ90I+y1/CVQqAgI0vhDNpGIJz0XjEoBz37W/5V8S7cisFLFCvVaqLctGASNfBvYQvgM6t2zZmoUQNV9mmMg8MkP5VXod+lriaP92oQcMXTyqj6OsV8yyxcnC3Phl4VT+c+APnMGU4Uq+SPs/v/KQ+O550bl1MhAmXjlUFPbDMFi0GJW0ZRoc2d7VpmIa3N08a8bztR4bmhdkD3kh9siSvj+Sm5EKd0HARzz46HJoEUBQ5G8J2JwaILF0SURY0EpY42U1VgVoQmSYxIvTIF3DjFHJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9MtXqt5AI6xt4ICbluiJvN2UYWO1rvf+ghlewsHqu9o=; b=Lqn6Uk7KNBKGCHzoBnigDNo1RiNUI8Hb/hygwqIRZayF1fZm0PeeQoBCWA9nipmIFQ9O6sDUYWZCryEJ8xlpLY6HMZYu+Fmi1cMAqM3Jlg6vvGhq30PYRoqrEtycC3aRavYUAMnlMhFnDUBpc4tyvtadjwPTU0KA5nUGvEj1RjMV0KryOFZFiVcQKdGtcUibr5k9n1TRN1pn0f8V6kJOcFGWueZMv3skF2iCKaPbkGb0MMRXa4hMMHja14TyZGSzd70Bp2QtBQRqIEI7cNFX859fy9XvJkeSMPzklr8FUU+nm8AkIfFemdZj3DX8ag4ZAujP5wCUtmEMJ/8PURug0Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9MtXqt5AI6xt4ICbluiJvN2UYWO1rvf+ghlewsHqu9o=; b=bQbX4+VEqdrv0T+n3h6V6HKNVTnrJLoLgKshMGWPDjRq+Be3ItoR2iOEfJU8l91nO1ej+Qx7LNgn6gES/pIh9BOzUDP6NL0vmAHEyYxWI9g6rU+TN3EXswdvJL+ZIGEElHreUTXi4GDqPmJXkp/rNTr051kR+aUGHxWFhc1uIw3Oqc81W7WWfX30n5aTOYTvtM2Biu2z7mj1uEVUIn4TnlUlldMEU2ywLWoUl2asaa0LzZg4b2zfIbStdb35n4e9D/vlclBBG8r9NfLa5kYRgze837+mJQO3/Qptnk6Yt8oBp6DBreUwxtxoJS4KcAXc26StmTMiZT3dBMoU8H+/2A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by MW3PR12MB4540.namprd12.prod.outlook.com (2603:10b6:303:52::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 05:43:38 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 05:43:38 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, Alistair Popple , linuxppc-dev@lists.ozlabs.org, linux-fpga@vger.kernel.org, linux-rdma@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, netdev@vger.kernel.org, io-uring@vger.kernel.org, bpf@vger.kernel.org, rds-devel@oss.oracle.com, linux-kselftest@vger.kernel.org Subject: [RFC PATCH 01/19] mm: Introduce vm_account Date: Tue, 24 Jan 2023 16:42:30 +1100 Message-Id: <748338ffe4c42d86669923159fe0426808ecb04d.1674538665.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYCP282CA0018.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:80::30) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|MW3PR12MB4540:EE_ X-MS-Office365-Filtering-Correlation-Id: 949f3254-124b-4c17-6368-08dafdcdf107 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iMmh0azBbDX7+rL+BE5BEHNjFZdAqJn47L6YKbnxs3D7TsqxrUTQx2RSCUk3jZThDL42O5RGyuwH8zpvTaaLcOekSn4NVxKfSwSxRDATckJ/zWVIPqmuFSlgCVOonXKwtxpTwN1omoNXUf9FGHoaM9MxMnZ2w15IS/57C4DkJPJAlPOxG4ND1JVCtSVxP5K5MV6xFqVsVvrgy56/sKfAKBQdvPbVKI/omxH9zkik6NzSSXsY0fzHd1QTbpYr8sVHLVVCDgb7VA3AdOZUSIEaesfXlSVYkw3EchIllny8q8jD3x0bOdk7FzQVwHxfcXEVeeWJxJLdHoLJJRZs2hHdCT+sImwmAY9nSEE2OyMPhMChDjSCXO7tjAfKmy3I6TxDv6oa/TbXajFwGAbtXr7yTAav86G8TJ7anCZX5Rg8wLUQZM5NTm+tL7CsdosPro8EUMDw3Y37M+N5P5/Tva6u2OvPFeIcqtJwjyYAN1HbpxEVsaK0AkiZI0y5DXwfJow2IaIkLv0djeJUZRxfWusx8xSjt9EGndEsQuEoVc5uGYhZbIs9OgHm9KHFlSm6z8iQf+O+kNdXH+N7nzD+JBy/zgYzeNgm8ao//zRWixl67VmgGTUH/03nhbyX+5GhLzyrA6+f/T6I+iHfINFXU8Hpmw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(366004)(376002)(136003)(396003)(39860400002)(451199015)(36756003)(86362001)(5660300002)(38100700002)(2906002)(4326008)(8936002)(7416002)(83380400001)(41300700001)(66476007)(478600001)(6486002)(66556008)(6506007)(6512007)(8676002)(26005)(186003)(316002)(66946007)(2616005)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: j5U5QqlRpm/Gik3sp+z+oKUI1HHnjDZEA4OFtF3hAjU3mL9XzunDP3p9FVmzdc7AYMqirmc+EwX01nM+UL6Va9Qt8dzQr/HKEZPZdyPwz1AZF8tS8qY9AV1pI442Z0Df88RUPMt+VWzZkND6qan6r06X/eAu39uUHw/eVW6jQRbgjSw0SV5MCYWUklPKU2UaJ/5RHBoNnUTtZFrtigudjmyEdIek0jtqqIOp5QsDafeV1vUkPoPcElpHS8v8SOnMklNkkoBYkshtlC9CaGS2qb/DjLMz7hvvxz7KNCtJ4ySVqxEiaV+GM5UK8w3UZRftuefpBb/dek3LIF8RgY3vjiTTnjNNE4LuLLKT8jVg8VsQ/QV3NWyRV6lHrhuVPOjjjVckgokD1HVffBtuVxFGG9Gq7UBCWTOF8O7MM+ayDF6MVDmKddQmxdIWpgy5tKhwPV+y0IZT+LnY8w/DkkvKUK+5I0q6gsqKi14EDz7V7tpIto4QUwYasZ7e1ZCVcbMW+dVTPL8d7h/mZ23Ue3UenIO3VzjS5L88Yq4eNLIdcOWE8B4znTMJu570YAx4Qmxz80zQIUYp23iBgByjNfITPJ3Pd/hVafLEcYXQ17b/SyDimpsDTD8ZC4zPjO5yxiy23pKXSgoYkPRBstRPUDsZK6cho2XfYlDq7338t5Wi6nSBx+D8/0ETEvAFHmxit2H3wqktgA4YYzD43azOK4EMt7f5cuthU5krTGsKnTYQ3+/Lk5vaN6+gIJTXHvauTOdYpHQNBMU6k+B2TGyIIMR86LrtFu2pdyjwmIV945IvAA7HmwvN7jsKjaewpK/tmzBgabRSPg0Q3WZLYbyPSYwqIqUOx8yxB21fNXe6eA3hzFrorABOkoZS1jOGmgo1MCV61ulFEMkwSMrtE1zoPOPcnbB1LFDJWk+TXPBMqewFDcQCvi/Ntiazm8iR6Rg2E5tzomKQ42Dj6L5TGfG5QZjGbf91kV+oI8lWdBxzve+vS2uPQ+KEsLB0DtP7yPfOvIGcyT/nlDXCfDYQF8obLAgsRb36HHqlttTASui5EzGFokjNVp0oj29zBYD4T/LBce33RBhN39V7wgATxPS4Fc4N7vh98uz1EGDLvMRyJe9cu6gdGQdHp4riSS5mqB9MittsgVLeB9TP1R1NFA0RnvOIQdYzxS2zP2PzJ6O9teQYZzFbuxROsaeqg9JxT8fcmCIpmzYJc1NQK0Czh2LALiWJ3S7feot3MQOBwrfJo3FbIGJSYGGWoM3juiDkZ0F9RLR0+T7CwRuada/tWH8cRMO8hGypN7GELE75lheA9jRkafV85GnnMy7uRqRodSp3LKxtlzbYY9wzpV6LrBubYD5ZV0rNorAuBK4v/76yB8mlW3jc+dbAsfVkmcmM00ED40e0zr35seRfbe0iFtT+4qUOfnhWhY6JQGblGbtl8TCYEQuXfNTjisSHSXLJcpkRQa0fkSAFa8Zo8TyMq96w9gNyOivNcOGpdAmHEnmSVeGrFbvR32gtRkftbUeq7R3YF3F73v7vWIB1n+GnUTQuRCz49pDfaKVxzGq7t2I113W3R3C0c+rT1iX7AMaDutlmuub9 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 949f3254-124b-4c17-6368-08dafdcdf107 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 05:43:38.3387 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uIP4ouzbfWVCa0XUJTot9rM9L+PhLKwqPJg7yXDOHet2Jme+y1DQ/VxC+ZGllQMyEfpyeIVZtQ1TVWuxyBQR6g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4540 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Kernel drivers that pin pages should account these pages against either user->locked_vm or mm->pinned_vm and fail the pinning if RLIMIT_MEMLOCK is exceeded and CAP_IPC_LOCK isn't held. Currently drivers open-code this accounting and use various methods to update the atomic variables and check against the limits leading to various bugs and inconsistencies. To fix this introduce a standard interface for charging pinned and locked memory. As this involves taking references on kernel objects such as mm_struct or user_struct we introduce a new vm_account struct to hold these references. Several helper functions are then introduced to grab references and check limits. As the way these limits are charged and enforced is visible to userspace we need to be careful not to break existing applications by charging to different counters. As a result the vm_account functions support accounting to different counters as required. A future change will extend this to also account against a cgroup for pinned pages. Signed-off-by: Alistair Popple Cc: linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-fpga@vger.kernel.org Cc: linux-rdma@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: kvm@vger.kernel.org Cc: netdev@vger.kernel.org Cc: cgroups@vger.kernel.org Cc: io-uring@vger.kernel.org Cc: linux-mm@kvack.org Cc: bpf@vger.kernel.org Cc: rds-devel@oss.oracle.com Cc: linux-kselftest@vger.kernel.org --- include/linux/mm_types.h | 87 ++++++++++++++++++++++++++++++++++++++++- mm/util.c | 89 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 176 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 9757067..7de2168 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1085,4 +1085,91 @@ enum fault_flag { typedef unsigned int __bitwise zap_flags_t; +/** + * enum vm_account_flags - Determine how pinned/locked memory is accounted. + * @VM_ACCOUNT_TASK: Account pinned memory to mm->pinned_vm. + * @VM_ACCOUNT_BYPASS: Don't enforce rlimit on any charges. + * @VM_ACCOUNT_USER: Accounnt locked memory to user->locked_vm. + * + * Determines which statistic pinned/locked memory is accounted + * against. All limits will be enforced against RLIMIT_MEMLOCK and the + * pins cgroup if CONFIG_CGROUP_PINS is enabled. + * + * New drivers should use VM_ACCOUNT_TASK. VM_ACCOUNT_USER is used by + * pre-existing drivers to maintain existing accounting against + * user->locked_mm rather than mm->pinned_mm. + * + * VM_ACCOUNT_BYPASS may also be specified to bypass rlimit + * checks. Typically this is used to cache CAP_IPC_LOCK from when a + * driver is first initialised. Note that this does not bypass cgroup + * limit checks. + */ +enum vm_account_flags { + VM_ACCOUNT_TASK = 0, + VM_ACCOUNT_BYPASS = 1, + VM_ACCOUNT_USER = 2, +}; + +struct vm_account { + struct task_struct *task; + union { + struct mm_struct *mm; + struct user_struct *user; + } a; + enum vm_account_flags flags; +}; + +/** + * vm_account_init - Initialise a new struct vm_account. + * @vm_account: pointer to uninitialised vm_account. + * @task: task to charge against. + * @user: user to charge against. Must be non-NULL for VM_ACCOUNT_USER. + * @flags: flags to use when charging to vm_account. + * + * Initialise a new uninitialiused struct vm_account. Takes references + * on the task/mm/user/cgroup as required although callers must ensure + * any references passed in remain valid for the duration of this + * call. + */ +void vm_account_init(struct vm_account *vm_account, struct task_struct *task, + struct user_struct *user, enum vm_account_flags flags); +/** + * vm_account_init_current - Initialise a new struct vm_account. + * @vm_account: pointer to uninitialised vm_account. + * + * Helper to initialise a vm_account for the common case of charging + * with VM_ACCOUNT_TASK against current. + */ +void vm_account_init_current(struct vm_account *vm_account); + +/** + * vm_account_release - Initialise a new struct vm_account. + * @vm_account: pointer to initialised vm_account. + * + * Drop any object references obtained by vm_account_init(). The + * vm_account must not be used after calling this unless reinitialised + * with vm_account_init(). + */ +void vm_account_release(struct vm_account *vm_account); + +/** + * vm_account_pinned - Charge pinned or locked memory to the vm_account. + * @vm_account: pointer to an initialised vm_account. + * @npages: number of pages to charge. + * + * Return: 0 on success, -ENOMEM if a limit would be exceeded. + * + * Note: All pages must be explicitly uncharged with + * vm_unaccount_pinned() prior to releasing the vm_account with + * vm_account_release(). + */ +int vm_account_pinned(struct vm_account *vm_account, unsigned long npages); + +/** + * vm_unaccount_pinned - Uncharge pinned or locked memory to the vm_account. + * @vm_account: pointer to an initialised vm_account. + * @npages: number of pages to uncharge. + */ +void vm_unaccount_pinned(struct vm_account *vm_account, unsigned long npages); + #endif /* _LINUX_MM_TYPES_H */ diff --git a/mm/util.c b/mm/util.c index b56c92f..af40b1e 100644 --- a/mm/util.c +++ b/mm/util.c @@ -430,6 +430,95 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) } #endif +void vm_account_init(struct vm_account *vm_account, struct task_struct *task, + struct user_struct *user, enum vm_account_flags flags) +{ + vm_account->task = get_task_struct(task); + + if (flags & VM_ACCOUNT_USER) { + vm_account->a.user = get_uid(user); + } else { + mmgrab(task->mm); + vm_account->a.mm = task->mm; + } + + vm_account->flags = flags; +} +EXPORT_SYMBOL_GPL(vm_account_init); + +void vm_account_init_current(struct vm_account *vm_account) +{ + vm_account_init(vm_account, current, NULL, VM_ACCOUNT_TASK); +} +EXPORT_SYMBOL_GPL(vm_account_init_current); + +void vm_account_release(struct vm_account *vm_account) +{ + put_task_struct(vm_account->task); + if (vm_account->flags & VM_ACCOUNT_USER) + free_uid(vm_account->a.user); + else + mmdrop(vm_account->a.mm); +} +EXPORT_SYMBOL_GPL(vm_account_release); + +/* + * Charge pages with an atomic compare and swap. Returns -ENOMEM on + * failure, 1 on success and 0 for retry. + */ +static int vm_account_cmpxchg(struct vm_account *vm_account, + unsigned long npages, unsigned long lock_limit) +{ + u64 cur_pages, new_pages; + + if (vm_account->flags & VM_ACCOUNT_USER) + cur_pages = atomic_long_read(&vm_account->a.user->locked_vm); + else + cur_pages = atomic64_read(&vm_account->a.mm->pinned_vm); + + new_pages = cur_pages + npages; + if (lock_limit != RLIM_INFINITY && new_pages > lock_limit) + return -ENOMEM; + + if (vm_account->flags & VM_ACCOUNT_USER) { + return atomic_long_cmpxchg(&vm_account->a.user->locked_vm, + cur_pages, new_pages) == cur_pages; + } else { + return atomic64_cmpxchg(&vm_account->a.mm->pinned_vm, + cur_pages, new_pages) == cur_pages; + } +} + +int vm_account_pinned(struct vm_account *vm_account, unsigned long npages) +{ + unsigned long lock_limit = RLIM_INFINITY; + int ret; + + if (!(vm_account->flags & VM_ACCOUNT_BYPASS) && !capable(CAP_IPC_LOCK)) + lock_limit = task_rlimit(vm_account->task, + RLIMIT_MEMLOCK) >> PAGE_SHIFT; + + while (true) { + ret = vm_account_cmpxchg(vm_account, npages, lock_limit); + if (ret > 0) + break; + else if (ret < 0) + return ret; + } + + return 0; +} +EXPORT_SYMBOL_GPL(vm_account_pinned); + +void vm_unaccount_pinned(struct vm_account *vm_account, unsigned long npages) +{ + if (vm_account->flags & VM_ACCOUNT_USER) + atomic_long_sub(npages, &vm_account->a.user->locked_vm); + else + atomic64_sub(npages, &vm_account->a.mm->pinned_vm); +} +EXPORT_SYMBOL_GPL(vm_unaccount_pinned); + /** * __account_locked_vm - account locked pages to an mm's locked_vm * @mm: mm to account against From patchwork Tue Jan 24 05:42:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13113520 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B51FFC54E94 for ; Tue, 24 Jan 2023 05:44:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229507AbjAXFo5 (ORCPT ); Tue, 24 Jan 2023 00:44:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230191AbjAXFo4 (ORCPT ); Tue, 24 Jan 2023 00:44:56 -0500 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2076.outbound.protection.outlook.com [40.107.95.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D63C3B3EE; Mon, 23 Jan 2023 21:44:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c3m1K3OEaJE6HDt3yWKerlp6uEtIx3TIbG8xVEw2t6zYQMu8HEK0S2sPJ4a7tnI/LAKuGRik4sgj9euKhW5cxkSurXQugCv5HEugWMGys3d8BU92744bl+quPJnMSWVHLscM1EBAHKduhiCYe4/7vxqewVRZAXKMvr/HvlfdfIujLsyrphScDeFvFiKykUS+mKBpn5cRefLjs0bUiBGCAXGVKGZ7lg5jl/wc8wb5QEhNQv+BZfTrLDkM0uheytr9LyC+11yEGupylNkU0N86ifvvXceEdcUAaFt1YREeYWZVDZ/bpXWNoJMKGVHVU/sBxr+wmCsHqAyN7x7btL/aKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QMvEWLA5MngzK2+7cVOKPt4IrdKvnbRxz60Bk/vscac=; b=ajCNE+NXxcJtcQOvPZ+lVkpEGV9Tc2kVMJCFWKTMGBdJErLvyDI3KRYjqEVzvxN47xeJB7LqHUwC7TJ9yCseWMJsZULMcLEvAtclSPZKelDGhwWkYP6SNvBudN92kuKiIgRCEYTZMzO4cW7oC2EuZ9iByTCbGeFFwvPTu+nIFInwbQMYfUeB68miGS0hZIRn5b5IYvDp13wNfRrVM6N/x8419241MqOjSTzU7Quq8jUfCI7H0k1qEvVl9xwfADqVJ84Iw8ks8Yf2EojPUBGkAl03yyBU89ZsupgANTT0z946mDrTK4gM4z05nKotwn1w4rtKRBA4S/kYjUf6Bq7xOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QMvEWLA5MngzK2+7cVOKPt4IrdKvnbRxz60Bk/vscac=; b=PUtCVRp5axRVSwNeRP8p3B8IWqlmfZAq9ZQtbWmMDDnZ2q7GEAZU5xSokM6rdIQBZNqti/uqdPQCeQRvoH1G4/G5laKu7HD4KjhFGdFJ9DVHfZLEkcwenX84h7fKafAqd1ZP8wmmfzPFG0FiFsBueaXOnfDZEhN4ESlCr7GfGBtbgs4aToGR6CeAVgVw1w0w+kaDYrog5WWYkDdOaWjAG+58cytY0WUL4/ZEoypTXSLMNKle0MKVlhk4JhpfZmnlTGZJxzSlTBf1HMmgsV1HwbQH9cx/MyvDa8BTdD+BBCdng2Ze6wEpoz0mvfPrvEYYZKdZ4yAkDyvaRDqRkWC9/Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by MW3PR12MB4540.namprd12.prod.outlook.com (2603:10b6:303:52::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 05:44:14 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 05:44:14 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, Alistair Popple , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [RFC PATCH 04/19] infiniband/umem: Convert to use vm_account Date: Tue, 24 Jan 2023 16:42:33 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYAPR01CA0047.ausprd01.prod.outlook.com (2603:10c6:1:1::35) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|MW3PR12MB4540:EE_ X-MS-Office365-Filtering-Correlation-Id: 65e42a4d-43cf-432c-e806-08dafdce06be X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wBYv9oGtcnNGEOsgD3DowtILQGny8oO3+jIWJmhR6fN488+hTxa4W9le8XuLCmP4Z0mVzgB+BvDS3L06m0WRaD8Ly/9z4XALjk9qKDZAb4ZWdqSaQzwHgYikyNWZ9b1WOrXGbDKQpR33NNeBFsIr0PepoQK4V4lw+3hnyKCD5nIC50+dKR5ttWvunPe8cT73RuUgyM7UnLaOPHGv7LWiL0DI753n/eUTi74aNMYAmCdiQf+/XZZJcdISmamrvFU7zYBXSEjU1z82XQ5ldGZS3yDLLfdB7ZmZv/peP8LxUEUJCraue3na5/ARUKkoH/gk14fXhritwuv/0wpoFvk2zPGQfsdMkT1pXCquW7dST6HHnWvWqMkIPQqEAvXFVg4Ky7c8rI8nyzuUtE8/zTxqluUfc4zA24FXidAb0rJAJ1pStZw0lIwZX3qGm9pX62Whe0vcBP1+C1fbnBtMEmssMAs9XcVSFi9TfsegvIMgx+5mtHa3g/2DhUcNRJkKog6T9eu7PlLwL7UlHA9ODTax913cp26bMXE3dpK9nwV5mzR+ynOxMpwnKEXP9c5BHf4JqsHwBxYRpcJaujLsXtgcrwMPdZhy+9DMTirjQ+9pw8pnSZ5ngOfnEcAjvAfHcltG X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(366004)(376002)(136003)(396003)(39860400002)(451199015)(36756003)(86362001)(5660300002)(38100700002)(2906002)(4326008)(8936002)(7416002)(83380400001)(41300700001)(66476007)(478600001)(6486002)(66556008)(6506007)(6512007)(8676002)(26005)(186003)(316002)(66946007)(2616005)(54906003)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: MT+N3d/4mSlKuRr2m9nmAjRq63EiccwwQQ7lZmvY7j5qqyUP9+N1VbbJCOHbqR8u0/RQPg1so28uvrxGS46AeMFG8saRrrNleX3+BZTlWmMUGP8DQObElSKZpwD2xsKLuMiFtPZK/Ehg6ZQroSgDBF0p5yqufvdTaqKSLAdncnHCSofONqvhIxSmMmtLYGVhc3O3UwxXDHlOvhX3Zd+ngHD1l7DHcyQDklRAE3pV/Hc2CFoBEA57r7A4ZddYzm6BflhsiLx7a0fE4C4jllEOqfJNInIg+Ucm7KF8mloDS9eX3SIyM7wSFa8UX6WpUHNAZ3RNMft+LxRBZQsJuEcSV7DdrdpEKyTfj6oSu1hbGsBOjuh43m5fAgy3TY/UF0v8DJFHfKUh3XW4s3+6ZsnHEuvsxf3r4AUqQa3vyrDkNOnLId1JxX0At5SVkAv3pBZd40+ldQoaFb2AKIp080ElC7cKAQVEMETq89w0s2rF0ZMPl2BviJ6MWXlHCpzXBMR5jqC9f1zcva8R3mUqfJ+cP4UnG6wIHp/ZwNRPXkuc+3M65tstBRn3s/p/JI8k0QLCTKdOIMJm6jTtwotQ9981CxvPFHosOYFr4BBkcJvyVbmCNDzZP6sBHOZN9So1rJG2z1pOUkhst+efbY1l9RSzTWq+4GuOY3rjOljR5emi+8aWVwqw3KTSLpoMTayrZIoGb2NRhqvF2uLWYcSpGPFcYdvI/5fXuVVuGQpZq3K5CBfQnuGdS2rAPHqsP79R/P/lWgLTyXK3bwsaGiutCU/sq9Gn0xkuWfs/k6Fzh7hVi3QkN3m7bk8Z5HXp5qRl42a+3vY5ZYGe1KAMyjBSv21Dyf4P9jd9dX1Xy7av5W2zDdQtbU3rJpjpO/8Fu1yKoSXMDwYYuoeXrdr1u8+XiZhXv6fLc6L6rZ9FDHHFHZqLs9cNDqcA8QcySHQhWanlwPhz0Sl+9EMR/aYGG9Cf5lmlTJR0jrDMIz+bLeOybZFR14vEBj7PqT9VPNqqze7czmM1Br6Zr0PloI79Sq4w7xzTjVqZXwgQA/akL5oUbZfL0gjlsIkGSvs90LUbGU7jQ3IcSzknRl+kn6QzD0BJMtRfgghdxMmF5B4JrSv52nFw2Do5Nrj22gJ0L3ptbHdG3ELzt6c7PK4hcOsw2fGvJ/AyuGtFTVP6TOxZh4OnKKbElCOdw5AvdWezAUYcRVbmeKoDraCFW8vC8y88A8uiHYdFVK3e3jNtXXhWgeZCzuzQtpkW+3PJgPhFYc46VjisdvmrkGcIc7YpaZsQxfECDRoml0rMI2E4PKrMzPMYyrJ7hTJYLwT9nYrCP4e2sBXbMZmmWgdV37sclyQihcxU9MGJ8Xs9OWoE+5KiV2sUEyXA/CFYEKi/Thv7ssN51lqU7EKQuU7Js3wvPgpmJDtEm9fmvgRMsij6XorcCrhSAHTEyPun97fxMsSjIwEXLVXyyJnz6cdyTzwEbo/YeP5091UXu78KVUlKf/Bo1IlKpyNK3lITN08h9vGYpY8OawjlNU4wmAjyiPDC77CcqjULpBUuqmbtWlBsh6wRwGMMlrexQSlgdFiekDAPM4WQWdEKXQRZ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 65e42a4d-43cf-432c-e806-08dafdce06be X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 05:44:14.7111 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6WdbGYEZ+nXsvkLIvlU/MYyEp4s32vlCTCrm0ONclheAVhWnpmFK5z/ud6LIQAcYJJQegeK+EQZ/gnCveB0deA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4540 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Converts the infiniband core umem code to use the vm_account structure so that pinned pages can be charged to the correct cgroup with account_pinned_vm(). Signed-off-by: Alistair Popple Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/infiniband/core/umem.c | 16 ++++++---------- drivers/infiniband/core/umem_odp.c | 6 ++++++ include/rdma/ib_umem.h | 1 + 3 files changed, 13 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 755a9c5..479b7f0 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -149,8 +149,6 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, { struct ib_umem *umem; struct page **page_list; - unsigned long lock_limit; - unsigned long new_pinned; unsigned long cur_base; unsigned long dma_attr = 0; struct mm_struct *mm; @@ -186,6 +184,7 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, umem->writable = ib_access_writable(access); umem->owning_mm = mm = current->mm; mmgrab(mm); + vm_account_init_current(&umem->vm_account); page_list = (struct page **) __get_free_page(GFP_KERNEL); if (!page_list) { @@ -199,11 +198,7 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, goto out; } - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - new_pinned = atomic64_add_return(npages, &mm->pinned_vm); - if (new_pinned > lock_limit && !capable(CAP_IPC_LOCK)) { - atomic64_sub(npages, &mm->pinned_vm); + if (vm_account_pinned(&umem->vm_account, npages)) { ret = -ENOMEM; goto out; } @@ -248,12 +243,13 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, umem_release: __ib_umem_release(device, umem, 0); - atomic64_sub(ib_umem_num_pages(umem), &mm->pinned_vm); + vm_unaccount_pinned(&umem->vm_account, ib_umem_num_pages(umem)); out: free_page((unsigned long) page_list); umem_kfree: if (ret) { mmdrop(umem->owning_mm); + vm_account_release(&umem->vm_account); kfree(umem); } return ret ? ERR_PTR(ret) : umem; @@ -275,8 +271,8 @@ void ib_umem_release(struct ib_umem *umem) __ib_umem_release(umem->ibdev, umem, 1); - atomic64_sub(ib_umem_num_pages(umem), &umem->owning_mm->pinned_vm); - mmdrop(umem->owning_mm); + vm_unaccount_pinned(&umem->vm_account, ib_umem_num_pages(umem)); + vm_account_release(&umem->vm_account); kfree(umem); } EXPORT_SYMBOL(ib_umem_release); diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index e9fa22d..4fbca3e 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -130,6 +130,7 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, umem->ibdev = device; umem->writable = ib_access_writable(access); umem->owning_mm = current->mm; + vm_account_init_current(&umem->vm_account); umem_odp->is_implicit_odp = 1; umem_odp->page_shift = PAGE_SHIFT; @@ -137,6 +138,7 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, ret = ib_init_umem_odp(umem_odp, NULL); if (ret) { put_pid(umem_odp->tgid); + vm_account_release(&umem->vm_account); kfree(umem_odp); return ERR_PTR(ret); } @@ -179,6 +181,7 @@ ib_umem_odp_alloc_child(struct ib_umem_odp *root, unsigned long addr, umem->address = addr; umem->writable = root->umem.writable; umem->owning_mm = root->umem.owning_mm; + umem->vm_account = root->umem.vm_account; odp_data->page_shift = PAGE_SHIFT; odp_data->notifier.ops = ops; @@ -239,6 +242,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *device, umem_odp->umem.address = addr; umem_odp->umem.writable = ib_access_writable(access); umem_odp->umem.owning_mm = current->mm; + vm_account_init_current(&umem_odp->umem.vm_account); umem_odp->notifier.ops = ops; umem_odp->page_shift = PAGE_SHIFT; @@ -255,6 +259,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *device, err_put_pid: put_pid(umem_odp->tgid); + vm_account_release(&umem_odp->umem.vm_account); kfree(umem_odp); return ERR_PTR(ret); } @@ -278,6 +283,7 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) kvfree(umem_odp->pfn_list); } put_pid(umem_odp->tgid); + vm_account_release(&umem_odp->umem.vm_account); kfree(umem_odp); } EXPORT_SYMBOL(ib_umem_odp_release); diff --git a/include/rdma/ib_umem.h b/include/rdma/ib_umem.h index 92a673c..de51406 100644 --- a/include/rdma/ib_umem.h +++ b/include/rdma/ib_umem.h @@ -19,6 +19,7 @@ struct dma_buf_attach_ops; struct ib_umem { struct ib_device *ibdev; struct mm_struct *owning_mm; + struct vm_account vm_account; u64 iova; size_t length; unsigned long address; From patchwork Tue Jan 24 05:42:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13113521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22E46C54E94 for ; Tue, 24 Jan 2023 05:45:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232758AbjAXFp2 (ORCPT ); Tue, 24 Jan 2023 00:45:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232512AbjAXFp0 (ORCPT ); Tue, 24 Jan 2023 00:45:26 -0500 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2076.outbound.protection.outlook.com [40.107.95.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C3B33B3D4; Mon, 23 Jan 2023 21:44:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Bb3O+BkGV052ySnoKOa7LGiPh2Q84Rl3VxYSPCZyfCMfut4iT4sIDGkl0m0xkmmGCUUutkX30pHcOkY0zi5PeG9T9BxhydURS+/JEjrZy7ZbAS6PfXEwbKW/Ve8xD0hCXjTjuJJkSULQRd0U0exHZyIzl4n4TyEog8fju9JGQkUMxAyeO72CJmdAdQQafabNayo2i/0WHzsg47+q7/nz/pg8wp+Cvn5eWe+HueUP2k3g0d/tPn+Yp3Kc2bXjVCMWyFX8NGwJMPL3TUzWGD+glJjs6yk0n84hoTN1ii8hpSctLyz+a07gPOC9tPHR+jg90szeA7px7KBlhVkx0eSr5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ys/BSuVjvYg3vlQckN9qotw9N9puGnTTdwB3lsffmG0=; b=XcYRMNMgrw1kZusBBbteylsMkZYnSX4eKvlNjEzTBw3gs/8ojFq7VioI8U+nGpn+aD1qFyHZxW66vVb7jdzzV+ZElUZ7ap6+nWw6A9xlZ0+C8mIf0PJ5yxrJ2loS7SaJ1zIK0f31KT8+oyg3X7GAFTX1zlWhx8bCrftTlxHzWuqkLlq3wH2Pl3CakN/JaG2v3x4tZUjtoopH5xl3AoK+4k70TOMxEJomFgvBYl+s4VhqHUwypzi7HO9lucXbCuDvT0wWnF09x6Rzm6mZ1jLGg0FSaPpGAWPb1ZtJKg7WrDaKaW8/bKRQPbzxj+ezXWClmdHVkCPZ/XZRRdcqvVHEXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ys/BSuVjvYg3vlQckN9qotw9N9puGnTTdwB3lsffmG0=; b=VMqRAsovhuiP1xlHzwhCtlAjrD4l00fCvvvfVy44+q4zWKTW6AUznfapM3qZYyLL7Zo0TAow/blShV90PvoRb/xlIdIEcSWt1wjlefZcGYfPH0PNi0Nim0zEBTM+KQtfCZljuIo/dZej0/O7Lzij2jWwT/I+zJefY+jRSTh6d4LY/bwZvypcYNH2x98rb1c4MLPKTRRvBBYBeeXFeVkE8CAfo9r49jlyUqbQiDI8wo6tgtDIECAIdDXMM+gvgr5drIdK9BDxSZKUgWyhK6xcrjZyT5pIQHWglSQlxWrX4UxkJoj5AeQXqe3CsNNxgwNKwvj+DHbEuJ99Zl0nVNKhwg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by MW3PR12MB4540.namprd12.prod.outlook.com (2603:10b6:303:52::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 05:44:27 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 05:44:27 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, Alistair Popple , Bernard Metzler , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [RFC PATCH 05/19] RMDA/siw: Convert to use vm_account Date: Tue, 24 Jan 2023 16:42:34 +1100 Message-Id: <4f8b1d54ab5d6909c46ba5470065f8326f9fcc62.1674538665.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0005.ausprd01.prod.outlook.com (2603:10c6:10::17) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|MW3PR12MB4540:EE_ X-MS-Office365-Filtering-Correlation-Id: 90992122-bf85-4cfb-bbac-08dafdce0e2f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xPoVJFoi2Q5dygKTO+tA+wp3wYXP9Odmw4uNetAYU4PsftNIGUxGUSIGMDEW66kbv0L2pSv8IZjHerNBaTqA5BYNpiMVI5B0Hq1KDbrtgEYfIOmjrCbE9o1Gy66CvMgAQWSSnDjRI4vKUhN6AH1ANkK2Gcq6AxxGPJhBjNZ4PWb74q+wDutA68XJ3+qsGGufYgOukDkgeO809WDn5iWQdOMgF6bAi0c7p2Tc18KbyVMSI0xfykAiXcSOM/LhUll1taz4rr5qY6CPR6ZU1Ax6bOiaOyg5lysmjGowykz0T9yVTX884HzXeCOzlLeJG2IgDJgDft/LMh1zMxh8wYhUqmEjIfSjW7Ts+8NYvTuyPwM9aO3BUym3YQVYX/FtBuC7YdHUG9LGX0CAGY0Ak9PH79YJDLqglVlzK/sQaUcEZdAkPBH2i2ML2OcN4TQGnvMzEIt42NY3seep/YvCG8Tz6av5ZRBs5qF75zuUCFW9qMPWaisSprb57iJaalEWYqGDbk98sOkxIJIlgHv6Y3OBlk7LUI1zYj/nUwKHogF5HbJiLDSkAf6sSRnffjlwYOzZrE3CJ4LCsA4mciMOSwHysuMd6cNPB6K9Nje3vwCF3YvK3AuVoCmB2neSiiIYdpvz++SDX2Rl0IN+Cm9SHV8yhg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(366004)(376002)(136003)(396003)(39860400002)(451199015)(36756003)(86362001)(5660300002)(38100700002)(2906002)(4326008)(8936002)(7416002)(83380400001)(41300700001)(66476007)(478600001)(6486002)(66556008)(6506007)(6512007)(8676002)(26005)(186003)(316002)(66946007)(2616005)(54906003)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7INmr83+joiH8mGAyXhiTEKLgagnKOtAuPSs35DKgMe51e2kS33fbUWuaCDtjViRoxUpqVo1pOgkJxQY0rGOIOG5X861wNtgV7tX9idG2Z3nQ/mQV6g8gnuVrJfy0fS36QCEUBVwpvuvVQPIS1suhS1pTgZmoCPFuFlUMnGYd7+ZqpEcCj6JnZTtILSzltnzZySskWf6iLV66xj+NAxsUgLlrrq9fFRXrEgM5OBfGYoWp7GqGtEMVGw9BnwkAA+nChegXAC9jbvpal3V7vEUo/5BRjUt3eaWbXE72P/MFtWMPAR5VnG0B6H/VwTdMSJA1T1uVDSQYUyH7crXfv47Q09NAaWK3n7IE4CsQ09Ek8pw9FG4Pz3LMeeeoSjFtpwWHz301Ya7dafRTqfiLJgIDeJ4A7Js9CAsXWySqOSYSC5jf03idiW/PoZABpMOchr+S41KTQN102Pb5ge8sUZ1eTuoRpfQXhJ5/gT17z+Z+aSxVdlv+4VuURtVdEnINALsORK64gTgRrU5lsGrQ1WNvVw/ajyMAzAqvvkkIoqLhHhwqVAhO+uHsnHCaRreRtD0JvQ3CBsuj1sf7QvEIop9aCaACT5Q3x4H3W0YeRUfrkK7yhSYnk5oyonPDzsPxuCGGSYouLNhRbvm09eQGbfxPUL7HHcQQCP9CfRatdBrxe0XqspdxXGou8Sfo+/AecJ2u0pmnDx0xETm43qT1LJLyb3dsxnmtwJ0YFTe/qFIXSqCoAX1ojr7StL7AGRbiVZu+DRtOGjhBZ8tCA/DVbHoneq1J8FGedSTHTQzzjuhVzZKkDZXtJL86tJyHWZ4klqw0XjGKFjP/+ON514Zd6vpNYwDuqjU+XAUWAeBKbDcMJmQXbVmK79W7S9krG/eXvDGYJGtbCefdiiP4jx+OO9eVzgtTiTgpUBQw8AHPczYgd7Wxcyfx3HfcH73kcgWZtTFF4eVraYb5cioQ40FT/oV+Bbo/fT4aCftSI81r9hUwTLlgmtMlgaOgcfY+qB176mjqhgIv8bfWriwOFawZjJxIY+QSO6Ld883nMuD8f4uFmOTdOPjXRYWSV6Tp031k5ShYEPrbs5kivclLPnES47Dfdxi1JOFIVvN3Qd0jqEIWUb4Q510CywcoWZf3B1IhWN4TjLWv+4ZXKM+4m8xUu7wVkqbVo/oOLX35vQyZojGDBmCXkplz3IRzsvVje7dwCuXFx/7Ani3FB97FW36RzPkm/t4f69CP0djQOcgCr4lBRLbVAe4m0vdCzu85nuaD4gewwowQzpykrJWXIjnmYOP0YvEGie3xyu/SqYVeKxbYLW8bbD9052o1egjKcLyStEhIauRNHSjU5DSianf1ueeNODGOs+Fc7OMLKanJG9d1n8UAlpBgzMZlhqc0caF5A3nOItY2wP65A+X+mfpF8oEF4GcTCxIj5GhntDQrMKiGKyq67ghzWGMjY0cXyPLOm0RuxDnQ6r1DhOXixTPO6iPcXH+sIu5XZKhzpiUPwiFnFeGtqWG91VKBxyG/NLL3yzdMa4Lo6bujdGeta40XrCOQMhXY86m0rhApeqgbDFgjs+taJ9YlGhwGtiRMS7UZGxi X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 90992122-bf85-4cfb-bbac-08dafdce0e2f X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 05:44:27.2570 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GfimHfBEj5y93A4hAuyjqPMkRtt0veLtE4+3AW9Jcl3CCT/FaZIc5uyx7oJJGClA2Xa69XYBm4Y+udtVvo/ePg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4540 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Convert to using a vm_account structure to account pinned memory to both the mm and the pins cgroup. Signed-off-by: Alistair Popple Cc: Bernard Metzler Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/infiniband/sw/siw/siw.h | 2 +- drivers/infiniband/sw/siw/siw_mem.c | 20 ++++++-------------- drivers/infiniband/sw/siw/siw_verbs.c | 15 --------------- 3 files changed, 7 insertions(+), 30 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index 2f3a9cd..0c4a3ec 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -124,7 +124,7 @@ struct siw_umem { int num_pages; bool writable; u64 fp_addr; /* First page base address */ - struct mm_struct *owning_mm; + struct vm_account vm_account; }; struct siw_pble { diff --git a/drivers/infiniband/sw/siw/siw_mem.c b/drivers/infiniband/sw/siw/siw_mem.c index b2b33dd..9c53fc3 100644 --- a/drivers/infiniband/sw/siw/siw_mem.c +++ b/drivers/infiniband/sw/siw/siw_mem.c @@ -68,7 +68,6 @@ static void siw_free_plist(struct siw_page_chunk *chunk, int num_pages, void siw_umem_release(struct siw_umem *umem, bool dirty) { - struct mm_struct *mm_s = umem->owning_mm; int i, num_pages = umem->num_pages; for (i = 0; num_pages; i++) { @@ -79,9 +78,9 @@ void siw_umem_release(struct siw_umem *umem, bool dirty) kfree(umem->page_chunk[i].plist); num_pages -= to_free; } - atomic64_sub(umem->num_pages, &mm_s->pinned_vm); + vm_unaccount_pinned(&umem->vm_account, umem->num_pages); + vm_account_release(&umem->vm_account); - mmdrop(mm_s); kfree(umem->page_chunk); kfree(umem); } @@ -365,9 +364,7 @@ struct siw_pbl *siw_pbl_alloc(u32 num_buf) struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) { struct siw_umem *umem; - struct mm_struct *mm_s; u64 first_page_va; - unsigned long mlock_limit; unsigned int foll_flags = FOLL_LONGTERM; int num_pages, num_chunks, i, rv = 0; @@ -385,20 +382,16 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) if (!umem) return ERR_PTR(-ENOMEM); - mm_s = current->mm; - umem->owning_mm = mm_s; umem->writable = writable; - mmgrab(mm_s); + vm_account_init_current(&umem->vm_account); if (writable) foll_flags |= FOLL_WRITE; - mmap_read_lock(mm_s); + mmap_read_lock(current->mm); - mlock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - if (num_pages + atomic64_read(&mm_s->pinned_vm) > mlock_limit) { + if (vm_account_pinned(&umem->vm_account, num_pages)) { rv = -ENOMEM; goto out_sem_up; } @@ -429,7 +422,6 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) goto out_sem_up; umem->num_pages += rv; - atomic64_add(rv, &mm_s->pinned_vm); first_page_va += rv * PAGE_SIZE; nents -= rv; got += rv; @@ -437,7 +429,7 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) num_pages -= got; } out_sem_up: - mmap_read_unlock(mm_s); + mmap_read_unlock(current->mm); if (rv > 0) return umem; diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c index 906fde1..8fab009 100644 --- a/drivers/infiniband/sw/siw/siw_verbs.c +++ b/drivers/infiniband/sw/siw/siw_verbs.c @@ -1321,8 +1321,6 @@ struct ib_mr *siw_reg_user_mr(struct ib_pd *pd, u64 start, u64 len, struct siw_umem *umem = NULL; struct siw_ureq_reg_mr ureq; struct siw_device *sdev = to_siw_dev(pd->device); - - unsigned long mem_limit = rlimit(RLIMIT_MEMLOCK); int rv; siw_dbg_pd(pd, "start: 0x%pK, va: 0x%pK, len: %llu\n", @@ -1338,19 +1336,6 @@ struct ib_mr *siw_reg_user_mr(struct ib_pd *pd, u64 start, u64 len, rv = -EINVAL; goto err_out; } - if (mem_limit != RLIM_INFINITY) { - unsigned long num_pages = - (PAGE_ALIGN(len + (start & ~PAGE_MASK))) >> PAGE_SHIFT; - mem_limit >>= PAGE_SHIFT; - - if (num_pages > mem_limit - current->mm->locked_vm) { - siw_dbg_pd(pd, "pages req %lu, max %lu, lock %lu\n", - num_pages, mem_limit, - current->mm->locked_vm); - rv = -ENOMEM; - goto err_out; - } - } umem = siw_umem_get(start, len, ib_access_writable(rights)); if (IS_ERR(umem)) { rv = PTR_ERR(umem); From patchwork Tue Jan 24 05:42:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13113522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD395C38142 for ; Tue, 24 Jan 2023 05:45:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232851AbjAXFpc (ORCPT ); Tue, 24 Jan 2023 00:45:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232841AbjAXFpa (ORCPT ); Tue, 24 Jan 2023 00:45:30 -0500 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2083.outbound.protection.outlook.com [40.107.243.83]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C058B3B671; Mon, 23 Jan 2023 21:44:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lqmD3Rpm28y2RWwRFZGqVCsa7JVofEmG9+7vMC8/3em1LeutbRYHlIiAYitZQVNNXOe7h1fCU229mkbFExwhIuQctBwxIJOQSWAa76VwY9VVm9VSnqxczChzCumh1K7MTK+VIhJeL0vpwjwLPb96t8fTCt7ucFYFgm+fDl50DaFggcY1rUwVevrIKB5PBkf14cOTvYb6vc4II/O8Io7H4ipWGHMt8DJbCyxirkRnq7I2VLJfcixCZPv8tvW6mHo+cz65KNzb0K1A9ST3iLXUKotChGrOXlp0tAso6+o9bFFg6xFv2mSejOIN8OERe3NwV3PXTtVowNQ80jiS88RStA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xVSTTv9ijFATl+Jf+OCS62NKLjuBKLo0e271Utsbdkw=; b=GMhqHZ30CKDgpv63T0QUgee2Wk0qR5+vJh3tsMAoYRyH2M9iidJ/HXwRSje1HHQ+7DBg9g7lS2KujpmMtnTVoz1kkqYQmVl8Tzf/kyLaxiJZEikNJ1da7U+WL820WsESMWxnb2KFfW7P09M1KJNoSQPliylQlzQiQtzOV54+SYEsCMiqWGFQqs3IHzXCiXSSk3o2mQdLokGa2NpNV94GetF9/1YpBk2vSLZN25xnP8bv580SDKirmDd9uy5/z4OTtZ4YWsN2oVqAiDqMiPCkpq5/6RRT3Fhx6sSK3H6zDTy3AX2P25+mzduNqFkjuiPEmnLo9UUpswPqnfdM/POaQA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xVSTTv9ijFATl+Jf+OCS62NKLjuBKLo0e271Utsbdkw=; b=g6blWCgOw4iUL8iy6QPPIj9lyTmJs/dnfznW6R32Pmeru+gzBRXTerouGEqb7Z+3HZTfJnkY5dnd3tbrbOpFNGOlbpnijc87LlwZ5WL5u180Dalp1pgYj652yQ+DAJM5b5npClD9Xx9LD3Kw63lMR/ejDDl+FzkumRXJpgOjgG1klrGzgjNHW5QWAnZSvTqmIsokLExwf8MpAb16PIUQeip8FWzu5VXlJy5wQ8MLo28oBcCt7HsiOvb5phBbNs6/fSWYSmqAkWqj4awByCyxSGE4I01+waUt8CbOZ4ickPz61GsaKXaZqoQjuIV2l4Pl5VC/NDVJ63HGPGfW81C2zg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by IA0PR12MB7507.namprd12.prod.outlook.com (2603:10b6:208:441::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 05:44:40 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 05:44:39 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, Alistair Popple , Christian Benvenuti , Nelson Escobar , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [RFC PATCH 06/19] RDMA/usnic: convert to use vm_account Date: Tue, 24 Jan 2023 16:42:35 +1100 Message-Id: <03ed2d166826cd7055810c66a175e20fa2153c52.1674538665.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYCPR01CA0019.ausprd01.prod.outlook.com (2603:10c6:10:31::31) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|IA0PR12MB7507:EE_ X-MS-Office365-Filtering-Correlation-Id: 58502a66-ecb1-4092-f3a6-08dafdce1583 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MY1gXIJLTFkqkhadHxZi5X+pDcRr+i4Yj0KNEWMYFe/Lz7wyEznPlryhZ7i9earuwU3RgTCbBJbxJZh3Cy9eRkuzcEvZzNRgK07/Q4vbbjnesKtWPEwqQ4gAMhUoOX6JjPrC3s/3MtSd4kMug6WveXXt3z0mZ8+ggxB0e6nejHMWHE+nkJdNhZbtvFInYFnSF5jV/LnJFd92VcYWhhYI86QeKM9A/QaJo33Hs9OyeLHuuch/i0uxmSWhC4vzve4MFjbXXoN+eqah002aiPwTF9XLDsiTL7xKeLQWguHM6rX4o2Kba+yikE4EK4zEQ2PJsFZacW/+2esVbvd24othnTPRm+CHQFAfgTv5Ga7Mz0p8NL1md5hvhgboNSV8bAS6JdHMOZLB90gwjgjr/bgI6PC/YUlYp8AcjIpza1NE/XCWnpDmSHBc5WBFv6EwA90CvGzeUYjF2TF/j3LEIshOE84X/2fMnnSLbdNqvDMZu2pvdUX/RX9TUYQC6c8wSsPyEgHxwkFjK28Y1RUjYInRJ4ygrrHwwHLpmurXMLZTGtlllWkHPec/0TKcTWNE2C7SJLCUUthJHPmKy52Q07lGSXXWf1I92R5LvKA8ThxVMgSba7AomTiyDHxjJUaYcLGJUZhva7MwTus5Yi8y6ET6Pw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(366004)(396003)(39860400002)(136003)(376002)(451199015)(2906002)(41300700001)(8936002)(66946007)(66476007)(66556008)(7416002)(5660300002)(54906003)(316002)(6512007)(478600001)(6486002)(6666004)(6506007)(26005)(186003)(2616005)(83380400001)(38100700002)(8676002)(4326008)(36756003)(86362001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IFnCrbaLEJ+YkC6TFtIjvM/ty9nMjoS5C0rEiJzmuS5XONjK6OowLYw45AksHspKnJ1ZOqA0lFuyf3gG8YdVuNPhAKALvuIX/i0lzFlCuptefjn1RBYbeOPusXhqOKclQ0usF85lne57dCXJy/8aRJfkLUWRGkyjF5HdwrzE3Ss1kviZiABnB5KbmoLWuH1nOZbXfqvm4VmkdaEnuIQrNAOK0/a4FtQcmqkm7aJ9tgLsnYgSCobc0PrZM2fO8NMH/5q5C55BMlq0RgXY83D+MPsGrN9YMRVfnwm7pVX7pFpN6EyhJkXlEUNZiU3DztRqfAw8jaH7+y+DPebLE2g/7wjdzWOgvNNPMyyq5QyjQPKuZhh0O0aK4aQKlMVpYHsW/vUoTD8TfAqM1OsRtPF3+YG/ohnsvlPcnnF+wx8KC7Py6MNxscj6s/jSvTmocSP1Rfg3b4Pk1l4R5HMMlIuLwZIRLdBJq+p34fhEKMjEYXvCWYKUS6UMVOL07AtB/IWSGpFKqL8+eWRsmG9Rq/6ZtQlQXcp8ht2xGxJpT2p1NE0qe6OZmCFk1Tb77R3+5hlW02f2pc8GFYAOg9dWzIARFKI9SPXEh3dCsxeRb5sxgA7A/V4uTwXmgGzpJ7b4L/zoCe2/mEqzLeK1Bi9kb5VEo2y310oZ6c8MT6lLEgSnUN+8Crxs6lbBvUyBTeC1r3Nbyow2s2gxlhDl+bc/lMvpvIN3X8h+ZkH18S3ZN0LK0pOt+VIbP2RKOUH4lccTKtjM6z2o1f5Lp16Psf1ZUHWWv+ZfxGHFUJdYerA3M0AQsZWuO3E3Gqz28P00qRvNN4hwIHMaMq/nFsK5B+iBhBqEFj9ootbLXuF0RkDsSMOM4twR9TscwPPQHvnzbakfI9zNAKYj/twplzCDvKVsoRSCgiEyXQHtugfpzlZCIsGOUj64t8cYxhFM9wGsZ6NmEuELsXaR73JEPDOT20ezwon88uXr71jMpnkmubW1/sF5gc6ux/o2Bl8NzR+tIHVtwNE7ZHg/BK1gWU3mGNWS3r2Rd7TqjbcrfA20s6SixSERQkHAqnkcsuXKudbmplCsSQeiPhd3aPWbx41WiPSQ+mV4cctKrmeEChM1hkPYDUYasn4g4yT6+8ahJCYfyE6qJHEtVf5Gpa9iNfUeDy+6jsj/108x2K7d7RnF8oHL4jlj3fYWm5YPGk9MJkM4g/yWx1Eou6r95dCu5hIG5DILuIWyjGSC7eEak64Jfc70rVR4WoN3P8LZiUlUNCGQOAWR/xIdzRuFKGcxtrBaboEF5basRsGwVb3jT9bHEITXFBvt43q3Ive7KJSUZ+Auadt8LxK5wxnA39MVNncdn6VjHHkADO7SdlN3GVfySNQWOf+oLYIfRKYf4ls697sZRVCNStuI5C4Rpe1UKPiaKq3sg8taw07O0ynzjIA5vz+cRdP0rLFqDTI5+9o9oZFIilQ/WKcmnsYk0ejhB29+QrE3bIBkZHs1xN5QeqAu/munHwV8PSHn7bxrx2u1aCSiXYBs8abipnXRdEbLp5UHXpVDtLxn/P/RK0aZx19/vc8xEzKVnXPJx+4q8WyGFkg1rNfl1Fdj X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58502a66-ecb1-4092-f3a6-08dafdce1583 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 05:44:39.4280 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: imSbUZwV9HUvtiRCcJ3HUSILYWMUenFLV2egkLFBu52lqOJzgY65CbSSEnx1zofFfPtwPTEMLpq73ZrQcRmxzg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7507 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Convert to using a vm_account structure to account pinned memory to both the mm and the pins cgroup. Signed-off-by: Alistair Popple Cc: Christian Benvenuti Cc: Nelson Escobar Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/infiniband/hw/usnic/usnic_uiom.c | 13 +++++-------- drivers/infiniband/hw/usnic/usnic_uiom.h | 1 + 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.c b/drivers/infiniband/hw/usnic/usnic_uiom.c index c301b3b..250276e 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.c +++ b/drivers/infiniband/hw/usnic/usnic_uiom.c @@ -89,8 +89,6 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, struct page **page_list; struct scatterlist *sg; struct usnic_uiom_chunk *chunk; - unsigned long locked; - unsigned long lock_limit; unsigned long cur_base; unsigned long npages; int ret; @@ -123,10 +121,8 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, uiomr->owning_mm = mm = current->mm; mmap_read_lock(mm); - locked = atomic64_add_return(npages, ¤t->mm->pinned_vm); - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) { + vm_account_init_current(&uiomr->vm_account); + if (vm_account_pinned(&uiomr->vm_account, npages)) { ret = -ENOMEM; goto out; } @@ -178,7 +174,8 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, out: if (ret < 0) { usnic_uiom_put_pages(chunk_list, 0); - atomic64_sub(npages, ¤t->mm->pinned_vm); + vm_unaccount_pinned(&uiomr->vm_account, npages); + vm_account_release(&uiomr->vm_account); } else mmgrab(uiomr->owning_mm); @@ -430,7 +427,7 @@ void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr) { __usnic_uiom_reg_release(uiomr->pd, uiomr, 1); - atomic64_sub(usnic_uiom_num_pages(uiomr), &uiomr->owning_mm->pinned_vm); + vm_unaccount_pinned(&uiomr->vm_account, usnic_uiom_num_pages(uiomr)); __usnic_uiom_release_tail(uiomr); } diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.h b/drivers/infiniband/hw/usnic/usnic_uiom.h index 5a9acf9..5c296a7 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.h +++ b/drivers/infiniband/hw/usnic/usnic_uiom.h @@ -72,6 +72,7 @@ struct usnic_uiom_reg { struct list_head chunk_list; struct work_struct work; struct mm_struct *owning_mm; + struct vm_account vm_account; }; struct usnic_uiom_chunk { From patchwork Tue Jan 24 05:42:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13113547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F4013C38142 for ; Tue, 24 Jan 2023 05:56:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232535AbjAXF4E (ORCPT ); Tue, 24 Jan 2023 00:56:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229666AbjAXF4D (ORCPT ); Tue, 24 Jan 2023 00:56:03 -0500 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on20600.outbound.protection.outlook.com [IPv6:2a01:111:f400:7e88::600]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D38323DB2; Mon, 23 Jan 2023 21:55:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SF076JzFXJT5tBCcFPiNclt6XkNHHaTjgJv2GOGCKrQb9KvUdRwi842dnBINijk3BqqjJZtXlyZjsxwkfhkqC/82FP5TE0tdS5sFmJl1EmUTxSHTJJBeUptZ9w+MciD3sDsg45oEP6QPYrhjrVD0ATo4QQ0g/BKU/2JqcvCn9Dgw6L5rm0cfDgzrxiDivD4KrWDvi4+kce9lU8w7TVg/I40abIpceVVFb9mziRC28P9Am7etrYwvc3Sg0Cr7uyYS/9zOeQyIpEUMUbRtQdZdEVhEjxntUcminDwgYsi9oBoxvXDzQyh+HVEOIa1wfU0IMTyeJgXN01xUFmEEUSJWqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nXKmUiAwwip0H/gynwy5RsgsJUC5caFZfjG6m6383VY=; b=K4vUOpPclHN6c2a3PB7LmoTw+TpMR6xlL14QwH0wiP/fs5WXcHX6ZhS5etqkazcBoeAn8cL0fNIm8c7roKQv5A2dKS53t0TeBFWelk14KohoSjGpurLj+zJ2u/RFQvBV66bmTVBQ1GuVxRX/DMIiW7efPPc7iwbijaiRmSsHMskMKyrqL3vmlm54zANwOoGYgFkrO4hV8FfK7C0b3Ks6vdOgMG6yP4qxxLVJBTdb3SmgbzT/CfsSW35M4+6sIPTHLsnzl8Zt8RmDEQOrVzUnCaGrt/Z0l1EwMakjODkb+6qJCEeLPxLh8z0EhRKFe5X1KL8g7UnR0PC2Rs91+LJ89w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nXKmUiAwwip0H/gynwy5RsgsJUC5caFZfjG6m6383VY=; b=EyOJEsMdC9Pm2dVyZu8t7V04q7ar/Fng5FR0rXTfQdglnp59DaTP9cs40qJld9Gm1OjXSNecHDL3L2mWGVnKtEz0Y6zx+nMEhgvJFDI5j2J0R5piA2LNtMyLmR7dMAE6/qFxCq1JPagJCvMLOtc+t3iBnnJmKH3jZXZ/Sr5YQ8vuY28QyoHFsJK8y6xxzrl49r1MQMX2wPjeqpbKp7Y2rwX4lCd2/mi2e9KlnlUgvhErN8QPJEaPo2En3y9Hj/vTHUmH1bYEeCKOOZ5lovaVKTB4uHvGrwdKvA96iqPfaJHPSs1KhCRM9T55ZTIpA4k2u26EOn0aM2FRVcR0rUq73g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by PH7PR12MB7793.namprd12.prod.outlook.com (2603:10b6:510:270::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 05:45:27 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::465a:6564:6198:2f4e%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 05:45:27 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, Alistair Popple , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [RFC PATCH 10/19] net: skb: Switch to using vm_account Date: Tue, 24 Jan 2023 16:42:39 +1100 Message-Id: <9b54eef0b41b678cc5f318bd5ae0917bba5b8e21.1674538665.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY5PR01CA0077.ausprd01.prod.outlook.com (2603:10c6:10:1f5::7) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|PH7PR12MB7793:EE_ X-MS-Office365-Filtering-Correlation-Id: ee42ee53-a49c-420e-9228-08dafdce3232 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: X4FOg3pTqEzrbtNhZV/TzqlzK6iYksRf7X01zRRZJgtB61ceWw5M02H6c2/8AX4X6jpt0Oy4Nkrfv8Um7NxQb5kbBGOymB0H/VIdTWNmS1B1AA3IcGWIBtpzWj54z9cXpY3AgUwarAcf4T6MKlzB6Q18tLs2WDvuYkZFZZONvxIr9UBxVbjx6I77E7SkxdnppQHiP/FTp4CPFvaotoGWKgChqfJ0Dn/WLZSgn5MDDi5BHkardrWyPnobV+mVrr4wrduVNTb1FsvYtCJ8Cw/6zkRW9ofeLBEIDNdbbVjjc42xof3RJVZ9uhKhYcTLevhp4SyXUHJnl8L3ZMw2r3et2SJ0ULyureKIzUJcrj9jojPx16mlxWxsQ9TYeTxJhVbhJ6X+g5lTmg7VoE2ztYpYP02wWatmpBbTK5jBurtPebz6QLs1EcqC5CIkgM7CkA7bPKZWDaEW13RfvgWYeBz/kps5E2Vo/qcz845BwJlT1vsxp3zPXx9O+WNVi9q9/tQaiqSy2QTFQjhBjg7G9XjdRz3nLe1JCO2Q/gP2cQVJ7Z4cukytvsCbRmbjdgITqF4J1knrcs+7EX5YIDgMo12Lyc45QmU4RBPvcpJzNR2kJK/jbzLiKwEEQjzwfjcCGE0WRrhAz3NNjCsRmZOhyuVKhg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(136003)(396003)(366004)(376002)(39860400002)(451199015)(36756003)(316002)(4326008)(66556008)(66476007)(8676002)(86362001)(66946007)(186003)(26005)(6512007)(6506007)(6666004)(83380400001)(6486002)(478600001)(2616005)(7416002)(5660300002)(8936002)(41300700001)(2906002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: K8g3q04AKDRU9iOQM5sWeWNFrDSqaTYGIY8JLNyY5Ch7MRB3P6TSMF5aHbPV/vSwlDtOwyTARzEtSTcZDwUR9obeuANIwT7O0j9TlDKLNN9InYK3HRZOSefK7LMNZve4qAO5wcJ5U+lJWrzM85R1idW4nVOXAHX4LQqlivlJ/02I12HUO76+iVzWDqRtCnbAWWJerEXnhW43w6xst40krfxmG6sWmheGd7aWQDMWMdhUalzoydlgCZOQBcRV36w8So79YPQZdz6dakoeXAmots5Wh0JqU6jzOaffVr3aZLIs44j4u6J0+lwgM7rw2mn3IUdjWFgbaRMO+PO2WsK2digPnlz02DxAVBKOSaOARKkc1//jRS2Et+pTRhdZfOeDQaE8lAobOauTI8I18jvr3DOnSw1aaumXxWEoBjtUD0mjiAFtqveLnrdBvdaWZIfAwJKrrl8lFa5h9OD/5H30pkI3Of4aAnHrz+2HLMqFEmmNiz5Vn9raPJ6vc+rgq/+rx7FAiWa/1UFBIT0EEoApLyg2Ex0uQA0SpW4voskDpzGtFQ8vHKV/uVKxdoqempU/Vi2QvOY0oiOW1FiomFUL4nMdIgHe3msfE5zF+DXJNZgNjhUOMIZeVhi4/TIiQK+Ospv4tKTjTs9F/I0+N7GLEuekHBDHi+iOg6iCCNiGq1taZSl+ZXqDE3T2mmElH4U2WY+eCOAy/gKQ1pTVFMcbQeL+NmhdAS00eaVq3NZZL6NXHS3Hh2kLz4wlsf0uBGs4iuMXCvMkVH8MTycwnZNokHNArTZauYowPZpg4G0LMY0eHWXc+p6QtZ5EOdqEBefBpAShRNuUNJJl2tRKb16jeVGYP4M1EMfkzBFxm+HqI50tPJwuU+fJwX5n+9b9d4B0EPQsucTReSpqkSIo7xm7asiTwGHSNFy4MHasfcvO3jyGfe4QpS9XIOiISNObfOl7VUvC+qp7R1HX8/gpw1SEI/vhosSKID8FJPtwtQLmtv/3CUOzXnRY8EC85cLo0WRV+MLFzYjPV9ch0DEcJkVE2F1gsXOnH1FM+yufAb6555Vpc9evgkkjntjNOhEPuGPXKVwTuPojqiqDZipUJE4yKmG5MvDaCLZOjryUGd2cqRFpifTz1+1vS9a9TrQV9jpx7kbaTiHh0gMxaBrtGPHJT4L9KUh4grOlhwpiWeKZEx50y6s2Hz1yRdpcKO+Ih3BN3wezb06I/mUmW8kK00N5FpdjUHG5V/FRjA/BzI3ZXHE5tEQntDGaRead5oYmHdpNAGPP3b/MeeHfFC+kfHyFB/WiMcDf3TyadxR8/zK0uZCyNmdZVS8HCWqqSoDp0ikce/ZkNj9J/NFrQ38NY59c/vFFlxR4y28CShErvgGDEc33uatXhOHsPciTw86RfBs5PYJj1Lx8ddSY7kjlszjuVGgeOJTF4Qm7yF3Ufr1ntCW9G1P9NK7xrMVvauoJOjDaeG+9QPKcftj63XLkjJn0+zoOVdWrmn/1vGSsGT9dEsjqmBLpjwOfexQ2nvIPh/AlbAGtD205gmhM95MMMTjbIDqy4KksJOnCIolLjCTyEzpcC7sG2jzLzeqvPDZIJ+hs X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: ee42ee53-a49c-420e-9228-08dafdce3232 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 05:45:27.5330 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OtuzUzDD66upC3A63HQzCEWHuh0uxSOe/AI/iKdZI4NmKeNoSjN4u6UaYGvK84y+yktDJCOxf3qaU2IhcgmvOQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7793 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Switch to using vm_account to charge pinned pages. This will allow a future change to charge the pinned pages to a cgroup to limit the overall number of pinned pages in the system. Signed-off-by: Alistair Popple Cc: linux-kernel@vger.kernel.org Cc: netdev@vger.kernel.org Cc: linux-rdma@vger.kernel.org Cc: rds-devel@oss.oracle.com --- include/linux/skbuff.h | 6 ++--- include/net/sock.h | 2 ++- net/core/skbuff.c | 47 +++++++++++++++---------------------------- net/rds/message.c | 9 +++++--- 4 files changed, 28 insertions(+), 36 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 4c84924..c956405 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -554,7 +554,6 @@ struct ubuf_info_msgzc { }; struct mmpin { - struct user_struct *user; unsigned int num_pg; } mmp; }; @@ -563,8 +562,9 @@ struct ubuf_info_msgzc { #define uarg_to_msgzc(ubuf_ptr) container_of((ubuf_ptr), struct ubuf_info_msgzc, \ ubuf) -int mm_account_pinned_pages(struct mmpin *mmp, size_t size); -void mm_unaccount_pinned_pages(struct mmpin *mmp); +int mm_account_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp, + size_t size); +void mm_unaccount_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp); /* This data is invariant across clones and lives at * the end of the header data, ie. at skb->end. diff --git a/include/net/sock.h b/include/net/sock.h index dcd72e6..bc3a868 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -334,6 +334,7 @@ struct sk_filter; * @sk_security: used by security modules * @sk_mark: generic packet mark * @sk_cgrp_data: cgroup data for this cgroup + * @sk_vm_account: data for pinned memory accounting * @sk_memcg: this socket's memory cgroup association * @sk_write_pending: a write to stream socket waits to start * @sk_state_change: callback to indicate change in the state of the sock @@ -523,6 +524,7 @@ struct sock { void *sk_security; #endif struct sock_cgroup_data sk_cgrp_data; + struct vm_account sk_vm_account; struct mem_cgroup *sk_memcg; void (*sk_state_change)(struct sock *sk); void (*sk_data_ready)(struct sock *sk); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 4a0eb55..bed3fc9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1309,42 +1309,25 @@ struct sk_buff *skb_morph(struct sk_buff *dst, struct sk_buff *src) } EXPORT_SYMBOL_GPL(skb_morph); -int mm_account_pinned_pages(struct mmpin *mmp, size_t size) +int mm_account_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp, + size_t size) { - unsigned long max_pg, num_pg, new_pg, old_pg; - struct user_struct *user; - - if (capable(CAP_IPC_LOCK) || !size) - return 0; + unsigned int num_pg; num_pg = (size >> PAGE_SHIFT) + 2; /* worst case */ - max_pg = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - user = mmp->user ? : current_user(); + if (vm_account_pinned(vm_account, num_pg)) + return -ENOBUFS; - old_pg = atomic_long_read(&user->locked_vm); - do { - new_pg = old_pg + num_pg; - if (new_pg > max_pg) - return -ENOBUFS; - } while (!atomic_long_try_cmpxchg(&user->locked_vm, &old_pg, new_pg)); - - if (!mmp->user) { - mmp->user = get_uid(user); - mmp->num_pg = num_pg; - } else { - mmp->num_pg += num_pg; - } + mmp->num_pg += num_pg; return 0; } EXPORT_SYMBOL_GPL(mm_account_pinned_pages); -void mm_unaccount_pinned_pages(struct mmpin *mmp) +void mm_unaccount_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp) { - if (mmp->user) { - atomic_long_sub(mmp->num_pg, &mmp->user->locked_vm); - free_uid(mmp->user); - } + vm_unaccount_pinned(vm_account, mmp->num_pg); + vm_account_release(vm_account); } EXPORT_SYMBOL_GPL(mm_unaccount_pinned_pages); @@ -1361,9 +1344,12 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) BUILD_BUG_ON(sizeof(*uarg) > sizeof(skb->cb)); uarg = (void *)skb->cb; - uarg->mmp.user = NULL; + uarg->mmp.num_pg = 0; + vm_account_init(&sk->sk_vm_account, current, + current_user(), VM_ACCOUNT_USER); - if (mm_account_pinned_pages(&uarg->mmp, size)) { + if (mm_account_pinned_pages(&sk->sk_vm_account, &uarg->mmp, size)) { + vm_account_release(&sk->sk_vm_account); kfree_skb(skb); return NULL; } @@ -1416,7 +1402,8 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, next = (u32)atomic_read(&sk->sk_zckey); if ((u32)(uarg_zc->id + uarg_zc->len) == next) { - if (mm_account_pinned_pages(&uarg_zc->mmp, size)) + if (mm_account_pinned_pages(&sk->sk_vm_account, + &uarg_zc->mmp, size)) return NULL; uarg_zc->len++; uarg_zc->bytelen = bytelen; @@ -1466,7 +1453,7 @@ static void __msg_zerocopy_callback(struct ubuf_info_msgzc *uarg) u32 lo, hi; u16 len; - mm_unaccount_pinned_pages(&uarg->mmp); + mm_unaccount_pinned_pages(&sk->sk_vm_account, &uarg->mmp); /* if !len, there was only 1 call, and it was aborted * so do not queue a completion notification diff --git a/net/rds/message.c b/net/rds/message.c index b47e4f0..2138a70 100644 --- a/net/rds/message.c +++ b/net/rds/message.c @@ -99,7 +99,7 @@ static void rds_rm_zerocopy_callback(struct rds_sock *rs, struct list_head *head; unsigned long flags; - mm_unaccount_pinned_pages(&znotif->z_mmp); + mm_unaccount_pinned_pages(&rs->rs_sk.sk_vm_account, &znotif->z_mmp); q = &rs->rs_zcookie_queue; spin_lock_irqsave(&q->lock, flags); head = &q->zcookie_head; @@ -367,6 +367,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * int ret = 0; int length = iov_iter_count(from); struct rds_msg_zcopy_info *info; + struct vm_account *vm_account = &rm->m_rs->rs_sk.sk_vm_account; rm->m_inc.i_hdr.h_len = cpu_to_be32(iov_iter_count(from)); @@ -380,7 +381,9 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * return -ENOMEM; INIT_LIST_HEAD(&info->rs_zcookie_next); rm->data.op_mmp_znotifier = &info->znotif; - if (mm_account_pinned_pages(&rm->data.op_mmp_znotifier->z_mmp, + vm_account_init(vm_account, current, current_user(), VM_ACCOUNT_USER); + if (mm_account_pinned_pages(vm_account, + &rm->data.op_mmp_znotifier->z_mmp, length)) { ret = -ENOMEM; goto err; @@ -399,7 +402,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * for (i = 0; i < rm->data.op_nents; i++) put_page(sg_page(&rm->data.op_sg[i])); mmp = &rm->data.op_mmp_znotifier->z_mmp; - mm_unaccount_pinned_pages(mmp); + mm_unaccount_pinned_pages(vm_account, mmp); ret = -EFAULT; goto err; }