| Message ID | 20230522154558.2166815-1-aspsk@isovalent.com (mailing list archive) |
|---|---|
| State | Accepted |
| Delegated to: | BPF |
| Headers | show |
| Series | [bpf] bpf: fix a memory leak in the LRU and LRU_PERCPU hash maps | expand |
On Mon, May 22, 2023 at 03:45:58PM +0000, Anton Protopopov wrote: > The LRU and LRU_PERCPU maps allocate a new element on update before locking the > target hash table bucket. Right after that the maps try to lock the bucket. > If this fails, then maps return -EBUSY to the caller without releasing the > allocated element. This makes the element untracked: it doesn't belong to > either of free lists, and it doesn't belong to the hash table, so can't be > re-used; this eventually leads to the permanent -ENOMEM on LRU map updates, > which is unexpected. Fix this by returning the element to the local free list > if bucket locking fails. Fixes: 20b6cc34ea74 ("bpf: Avoid hashtab deadlock with map_locked") > Signed-off-by: Anton Protopopov <aspsk@isovalent.com> > --- > kernel/bpf/hashtab.c | 6 ++++-- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c > index 00c253b84bf5..9901efee4339 100644 > --- a/kernel/bpf/hashtab.c > +++ b/kernel/bpf/hashtab.c > @@ -1215,7 +1215,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value > > ret = htab_lock_bucket(htab, b, hash, &flags); > if (ret) > - return ret; > + goto err_lock_bucket; > > l_old = lookup_elem_raw(head, hash, key, key_size); > > @@ -1236,6 +1236,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value > err: > htab_unlock_bucket(htab, b, hash, flags); > > +err_lock_bucket: > if (ret) > htab_lru_push_free(htab, l_new); > else if (l_old) > @@ -1338,7 +1339,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, > > ret = htab_lock_bucket(htab, b, hash, &flags); > if (ret) > - return ret; > + goto err_lock_bucket; > > l_old = lookup_elem_raw(head, hash, key, key_size); > > @@ -1361,6 +1362,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, > ret = 0; > err: > htab_unlock_bucket(htab, b, hash, flags); > +err_lock_bucket: > if (l_new) > bpf_lru_push_free(&htab->lru, &l_new->lru_node); > return ret; > -- > 2.34.1 >
On 5/22/23 9:12 AM, Anton Protopopov wrote: > On Mon, May 22, 2023 at 03:45:58PM +0000, Anton Protopopov wrote: >> The LRU and LRU_PERCPU maps allocate a new element on update before locking the >> target hash table bucket. Right after that the maps try to lock the bucket. >> If this fails, then maps return -EBUSY to the caller without releasing the >> allocated element. This makes the element untracked: it doesn't belong to >> either of free lists, and it doesn't belong to the hash table, so can't be >> re-used; this eventually leads to the permanent -ENOMEM on LRU map updates, >> which is unexpected. Ouch. This is very bad. :( Excellent catch. Applied. I am thinking if a test could be written but it does not seem like anything after htab_lock_bucket() is traceable. Not sure if Song may have good idea? >> Fix this by returning the element to the local free list if bucket locking fails. > Fixes: 20b6cc34ea74 ("bpf: Avoid hashtab deadlock with map_locked") >
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 00c253b84bf5..9901efee4339 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -1215,7 +1215,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value ret = htab_lock_bucket(htab, b, hash, &flags); if (ret) - return ret; + goto err_lock_bucket; l_old = lookup_elem_raw(head, hash, key, key_size); @@ -1236,6 +1236,7 @@ static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value err: htab_unlock_bucket(htab, b, hash, flags); +err_lock_bucket: if (ret) htab_lru_push_free(htab, l_new); else if (l_old) @@ -1338,7 +1339,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, ret = htab_lock_bucket(htab, b, hash, &flags); if (ret) - return ret; + goto err_lock_bucket; l_old = lookup_elem_raw(head, hash, key, key_size); @@ -1361,6 +1362,7 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, ret = 0; err: htab_unlock_bucket(htab, b, hash, flags); +err_lock_bucket: if (l_new) bpf_lru_push_free(&htab->lru, &l_new->lru_node); return ret;
The LRU and LRU_PERCPU maps allocate a new element on update before locking the target hash table bucket. Right after that the maps try to lock the bucket. If this fails, then maps return -EBUSY to the caller without releasing the allocated element. This makes the element untracked: it doesn't belong to either of free lists, and it doesn't belong to the hash table, so can't be re-used; this eventually leads to the permanent -ENOMEM on LRU map updates, which is unexpected. Fix this by returning the element to the local free list if bucket locking fails. Signed-off-by: Anton Protopopov <aspsk@isovalent.com> --- kernel/bpf/hashtab.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)