| Message ID | 20200228044518.20314-1-gmayyyha@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | ceph: using POOL FULL flag instead of OSDMAP FULL flag | expand |
On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > --- > fs/ceph/file.c | 6 ++++-- > include/linux/ceph/osd_client.h | 2 ++ > include/linux/ceph/osdmap.h | 3 ++- > net/ceph/osd_client.c | 23 +++++++++++++---------- > 4 files changed, 21 insertions(+), 13 deletions(-) > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > index 7e0190b1f821..60ea1eed1b84 100644 > --- a/fs/ceph/file.c > +++ b/fs/ceph/file.c > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > } > > /* FIXME: not complete since it doesn't account for being at quota */ > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > + CEPH_POOL_FLAG_FULL)) { > err = -ENOSPC; > goto out; > } > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > } > > if (written >= 0) { > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > + CEPH_POOL_FLAG_NEARFULL)) Hi Yanhu, Have you considered pre-mimic clusters here? They are still supported (and will continue to be supported for the foreseeable future). Thanks, Ilya
On Fri, Feb 28, 2020 at 6:23 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > > --- > > fs/ceph/file.c | 6 ++++-- > > include/linux/ceph/osd_client.h | 2 ++ > > include/linux/ceph/osdmap.h | 3 ++- > > net/ceph/osd_client.c | 23 +++++++++++++---------- > > 4 files changed, 21 insertions(+), 13 deletions(-) > > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > > index 7e0190b1f821..60ea1eed1b84 100644 > > --- a/fs/ceph/file.c > > +++ b/fs/ceph/file.c > > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > } > > > > /* FIXME: not complete since it doesn't account for being at quota */ > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > + CEPH_POOL_FLAG_FULL)) { > > err = -ENOSPC; > > goto out; > > } > > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > } > > > > if (written >= 0) { > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > + CEPH_POOL_FLAG_NEARFULL)) > > Hi Yanhu, > > Have you considered pre-mimic clusters here? They are still supported > (and will continue to be supported for the foreseeable future). > > Thanks, > > Ilya I have tested it work on Luminous, I think it work too since ceph-v0.80(https://github.com/ceph/ceph/blob/b78644e7dee100e48dfeca32c9270a6b210d3003/src/osd/osd_types.h#L815) alread have pool FLAG_FULL. CephFS doesn't write synchronously even if CEPH_OSDMAP_NEARFULL is used, then should fixed by CEPH_POOL_FLAG_NEARFULL.
On Fri, Feb 28, 2020 at 12:41 PM Yanhu Cao <gmayyyha@gmail.com> wrote: > > On Fri, Feb 28, 2020 at 6:23 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > > > > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > > > --- > > > fs/ceph/file.c | 6 ++++-- > > > include/linux/ceph/osd_client.h | 2 ++ > > > include/linux/ceph/osdmap.h | 3 ++- > > > net/ceph/osd_client.c | 23 +++++++++++++---------- > > > 4 files changed, 21 insertions(+), 13 deletions(-) > > > > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > > > index 7e0190b1f821..60ea1eed1b84 100644 > > > --- a/fs/ceph/file.c > > > +++ b/fs/ceph/file.c > > > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > } > > > > > > /* FIXME: not complete since it doesn't account for being at quota */ > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > + CEPH_POOL_FLAG_FULL)) { > > > err = -ENOSPC; > > > goto out; > > > } > > > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > } > > > > > > if (written >= 0) { > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > Hi Yanhu, > > > > Have you considered pre-mimic clusters here? They are still supported > > (and will continue to be supported for the foreseeable future). > > > > Thanks, > > > > Ilya > > I have tested it work on Luminous, I think it work too since > ceph-v0.80(https://github.com/ceph/ceph/blob/b78644e7dee100e48dfeca32c9270a6b210d3003/src/osd/osd_types.h#L815) > alread have pool FLAG_FULL. But not FLAG_NEARFULL, which appeared in mimic. > > CephFS doesn't write synchronously even if CEPH_OSDMAP_NEARFULL is > used, then should fixed by CEPH_POOL_FLAG_NEARFULL. I'm not sure I follow. - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, + CEPH_POOL_FLAG_NEARFULL)) AFAICT this change would effectively disable this branch for pre-mimic clusters. Are you saying this branch is already broken? Thanks, Ilya
On Fri, Feb 28, 2020 at 10:02 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > On Fri, Feb 28, 2020 at 12:41 PM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > On Fri, Feb 28, 2020 at 6:23 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > > > On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > > > > > > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > > > > --- > > > > fs/ceph/file.c | 6 ++++-- > > > > include/linux/ceph/osd_client.h | 2 ++ > > > > include/linux/ceph/osdmap.h | 3 ++- > > > > net/ceph/osd_client.c | 23 +++++++++++++---------- > > > > 4 files changed, 21 insertions(+), 13 deletions(-) > > > > > > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > > > > index 7e0190b1f821..60ea1eed1b84 100644 > > > > --- a/fs/ceph/file.c > > > > +++ b/fs/ceph/file.c > > > > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > } > > > > > > > > /* FIXME: not complete since it doesn't account for being at quota */ > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > + CEPH_POOL_FLAG_FULL)) { > > > > err = -ENOSPC; > > > > goto out; > > > > } > > > > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > } > > > > > > > > if (written >= 0) { > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > > > Hi Yanhu, > > > > > > Have you considered pre-mimic clusters here? They are still supported > > > (and will continue to be supported for the foreseeable future). > > > > > > Thanks, > > > > > > Ilya > > > > I have tested it work on Luminous, I think it work too since > > ceph-v0.80(https://github.com/ceph/ceph/blob/b78644e7dee100e48dfeca32c9270a6b210d3003/src/osd/osd_types.h#L815) > > alread have pool FLAG_FULL. > > But not FLAG_NEARFULL, which appeared in mimic. FLAG_NEARFULL appeared in Luminous. > > > > > CephFS doesn't write synchronously even if CEPH_OSDMAP_NEARFULL is > > used, then should fixed by CEPH_POOL_FLAG_NEARFULL. > > I'm not sure I follow. > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > + CEPH_POOL_FLAG_NEARFULL)) > > AFAICT this change would effectively disable this branch for pre-mimic > clusters. Are you saying this branch is already broken? > > Thanks, > > Ilya CEPH_OSDMAP_NEARFULL is not set in Jewel, so it has no effect. And in Luminous version, this flag is cleared as a legacy and has no effect too.
On Mon, Mar 2, 2020 at 3:30 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > On Fri, Feb 28, 2020 at 10:02 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > On Fri, Feb 28, 2020 at 12:41 PM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > On Fri, Feb 28, 2020 at 6:23 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > > > > > On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > > > > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > > > > > > > > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > > > > > --- > > > > > fs/ceph/file.c | 6 ++++-- > > > > > include/linux/ceph/osd_client.h | 2 ++ > > > > > include/linux/ceph/osdmap.h | 3 ++- > > > > > net/ceph/osd_client.c | 23 +++++++++++++---------- > > > > > 4 files changed, 21 insertions(+), 13 deletions(-) > > > > > > > > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > > > > > index 7e0190b1f821..60ea1eed1b84 100644 > > > > > --- a/fs/ceph/file.c > > > > > +++ b/fs/ceph/file.c > > > > > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > > } > > > > > > > > > > /* FIXME: not complete since it doesn't account for being at quota */ > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > > + CEPH_POOL_FLAG_FULL)) { > > > > > err = -ENOSPC; > > > > > goto out; > > > > > } > > > > > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > > } > > > > > > > > > > if (written >= 0) { > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > > > > > Hi Yanhu, > > > > > > > > Have you considered pre-mimic clusters here? They are still supported > > > > (and will continue to be supported for the foreseeable future). > > > > > > > > Thanks, > > > > > > > > Ilya > > > > > > I have tested it work on Luminous, I think it work too since > > > ceph-v0.80(https://github.com/ceph/ceph/blob/b78644e7dee100e48dfeca32c9270a6b210d3003/src/osd/osd_types.h#L815) > > > alread have pool FLAG_FULL. > > > > But not FLAG_NEARFULL, which appeared in mimic. > FLAG_NEARFULL appeared in Luminous. Well, it appeared in mimic in v13.0.1 and was backported to luminous in v12.2.2. So technically, some luminous releases don't have it. > > > > > > > > > CephFS doesn't write synchronously even if CEPH_OSDMAP_NEARFULL is > > > used, then should fixed by CEPH_POOL_FLAG_NEARFULL. > > > > I'm not sure I follow. > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > + CEPH_POOL_FLAG_NEARFULL)) > > > > AFAICT this change would effectively disable this branch for pre-mimic > > clusters. Are you saying this branch is already broken? > > > > Thanks, > > > > Ilya > CEPH_OSDMAP_NEARFULL is not set in Jewel, so it has no effect. And in > Luminous version, this flag is cleared as a legacy and has no effect > too. Are you sure? What about this code in OSDMonitor::tick() that showed up in kraken in v11.0.1 and was backported to jewel in v10.2.4? if (!mon->pgmon()->pg_map.nearfull_osds.empty()) { ... add_flag(CEPH_OSDMAP_NEARFULL); } else if (osdmap.test_flag(CEPH_OSDMAP_NEARFULL)){ ... remove_flag(CEPH_OSDMAP_NEARFULL); } if (pending_inc.new_flags != -1 && (pending_inc.new_flags ^ osdmap.flags) & (CEPH_OSDMAP_FULL | CEPH_OSDMAP_NEARFULL)) { ... do_propose = true; It's there in v10.2.11 (the final jewel release). It's also there in hammer since v0.94.10... Thanks, Ilya
On Mon, Mar 2, 2020 at 6:09 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > On Mon, Mar 2, 2020 at 3:30 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > On Fri, Feb 28, 2020 at 10:02 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > > > On Fri, Feb 28, 2020 at 12:41 PM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > > > On Fri, Feb 28, 2020 at 6:23 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > > > > > > > On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > > > > > > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > > > > > > > > > > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > > > > > > --- > > > > > > fs/ceph/file.c | 6 ++++-- > > > > > > include/linux/ceph/osd_client.h | 2 ++ > > > > > > include/linux/ceph/osdmap.h | 3 ++- > > > > > > net/ceph/osd_client.c | 23 +++++++++++++---------- > > > > > > 4 files changed, 21 insertions(+), 13 deletions(-) > > > > > > > > > > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > > > > > > index 7e0190b1f821..60ea1eed1b84 100644 > > > > > > --- a/fs/ceph/file.c > > > > > > +++ b/fs/ceph/file.c > > > > > > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > > > } > > > > > > > > > > > > /* FIXME: not complete since it doesn't account for being at quota */ > > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > > > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > > > + CEPH_POOL_FLAG_FULL)) { > > > > > > err = -ENOSPC; > > > > > > goto out; > > > > > > } > > > > > > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > > > } > > > > > > > > > > > > if (written >= 0) { > > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > > > > > > > Hi Yanhu, > > > > > > > > > > Have you considered pre-mimic clusters here? They are still supported > > > > > (and will continue to be supported for the foreseeable future). > > > > > > > > > > Thanks, > > > > > > > > > > Ilya > > > > > > > > I have tested it work on Luminous, I think it work too since > > > > ceph-v0.80(https://github.com/ceph/ceph/blob/b78644e7dee100e48dfeca32c9270a6b210d3003/src/osd/osd_types.h#L815) > > > > alread have pool FLAG_FULL. > > > > > > But not FLAG_NEARFULL, which appeared in mimic. > > FLAG_NEARFULL appeared in Luminous. > > Well, it appeared in mimic in v13.0.1 and was backported to luminous > in v12.2.2. So technically, some luminous releases don't have it. > > > > > > > > > > > > > > CephFS doesn't write synchronously even if CEPH_OSDMAP_NEARFULL is > > > > used, then should fixed by CEPH_POOL_FLAG_NEARFULL. > > > > > > I'm not sure I follow. > > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > > > AFAICT this change would effectively disable this branch for pre-mimic > > > clusters. Are you saying this branch is already broken? > > > > > > Thanks, > > > > > > Ilya > > CEPH_OSDMAP_NEARFULL is not set in Jewel, so it has no effect. And in > > Luminous version, this flag is cleared as a legacy and has no effect > > too. > > Are you sure? What about this code in OSDMonitor::tick() that showed > up in kraken in v11.0.1 and was backported to jewel in v10.2.4? > > if (!mon->pgmon()->pg_map.nearfull_osds.empty()) { > ... > add_flag(CEPH_OSDMAP_NEARFULL); > } else if (osdmap.test_flag(CEPH_OSDMAP_NEARFULL)){ > ... > remove_flag(CEPH_OSDMAP_NEARFULL); > } > if (pending_inc.new_flags != -1 && > (pending_inc.new_flags ^ osdmap.flags) & (CEPH_OSDMAP_FULL | > CEPH_OSDMAP_NEARFULL)) { > ... > do_propose = true; > > It's there in v10.2.11 (the final jewel release). It's also there > in hammer since v0.94.10... > > Thanks, > > Ilya Sorry for not seeing all version changes, I will submit plus CEPH_OSDMAP_FULL/NEARFULL. How to check if the feature is backported?
On Mon, Mar 2, 2020 at 12:02 PM Yanhu Cao <gmayyyha@gmail.com> wrote: > > On Mon, Mar 2, 2020 at 6:09 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > On Mon, Mar 2, 2020 at 3:30 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > On Fri, Feb 28, 2020 at 10:02 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > > > > > On Fri, Feb 28, 2020 at 12:41 PM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > > > > > On Fri, Feb 28, 2020 at 6:23 PM Ilya Dryomov <idryomov@gmail.com> wrote: > > > > > > > > > > > > On Fri, Feb 28, 2020 at 5:45 AM Yanhu Cao <gmayyyha@gmail.com> wrote: > > > > > > > > > > > > > > OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. > > > > > > > > > > > > > > Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> > > > > > > > --- > > > > > > > fs/ceph/file.c | 6 ++++-- > > > > > > > include/linux/ceph/osd_client.h | 2 ++ > > > > > > > include/linux/ceph/osdmap.h | 3 ++- > > > > > > > net/ceph/osd_client.c | 23 +++++++++++++---------- > > > > > > > 4 files changed, 21 insertions(+), 13 deletions(-) > > > > > > > > > > > > > > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > > > > > > > index 7e0190b1f821..60ea1eed1b84 100644 > > > > > > > --- a/fs/ceph/file.c > > > > > > > +++ b/fs/ceph/file.c > > > > > > > @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > > > > } > > > > > > > > > > > > > > /* FIXME: not complete since it doesn't account for being at quota */ > > > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { > > > > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > > > > + CEPH_POOL_FLAG_FULL)) { > > > > > > > err = -ENOSPC; > > > > > > > goto out; > > > > > > > } > > > > > > > @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) > > > > > > > } > > > > > > > > > > > > > > if (written >= 0) { > > > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > > > > > > > > > Hi Yanhu, > > > > > > > > > > > > Have you considered pre-mimic clusters here? They are still supported > > > > > > (and will continue to be supported for the foreseeable future). > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Ilya > > > > > > > > > > I have tested it work on Luminous, I think it work too since > > > > > ceph-v0.80(https://github.com/ceph/ceph/blob/b78644e7dee100e48dfeca32c9270a6b210d3003/src/osd/osd_types.h#L815) > > > > > alread have pool FLAG_FULL. > > > > > > > > But not FLAG_NEARFULL, which appeared in mimic. > > > FLAG_NEARFULL appeared in Luminous. > > > > Well, it appeared in mimic in v13.0.1 and was backported to luminous > > in v12.2.2. So technically, some luminous releases don't have it. > > > > > > > > > > > > > > > > > > > CephFS doesn't write synchronously even if CEPH_OSDMAP_NEARFULL is > > > > > used, then should fixed by CEPH_POOL_FLAG_NEARFULL. > > > > > > > > I'm not sure I follow. > > > > > > > > - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) > > > > + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, > > > > + CEPH_POOL_FLAG_NEARFULL)) > > > > > > > > AFAICT this change would effectively disable this branch for pre-mimic > > > > clusters. Are you saying this branch is already broken? > > > > > > > > Thanks, > > > > > > > > Ilya > > > CEPH_OSDMAP_NEARFULL is not set in Jewel, so it has no effect. And in > > > Luminous version, this flag is cleared as a legacy and has no effect > > > too. > > > > Are you sure? What about this code in OSDMonitor::tick() that showed > > up in kraken in v11.0.1 and was backported to jewel in v10.2.4? > > > > if (!mon->pgmon()->pg_map.nearfull_osds.empty()) { > > ... > > add_flag(CEPH_OSDMAP_NEARFULL); > > } else if (osdmap.test_flag(CEPH_OSDMAP_NEARFULL)){ > > ... > > remove_flag(CEPH_OSDMAP_NEARFULL); > > } > > if (pending_inc.new_flags != -1 && > > (pending_inc.new_flags ^ osdmap.flags) & (CEPH_OSDMAP_FULL | > > CEPH_OSDMAP_NEARFULL)) { > > ... > > do_propose = true; > > > > It's there in v10.2.11 (the final jewel release). It's also there > > in hammer since v0.94.10... > > > > Thanks, > > > > Ilya > > Sorry for not seeing all version changes, I will submit plus > CEPH_OSDMAP_FULL/NEARFULL. > How to check if the feature is backported? Look through release notes and git history. It's the only source of truth ;) Thanks, Ilya
diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 7e0190b1f821..60ea1eed1b84 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -1482,7 +1482,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) } /* FIXME: not complete since it doesn't account for being at quota */ - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_FULL)) { + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, + CEPH_POOL_FLAG_FULL)) { err = -ENOSPC; goto out; } @@ -1575,7 +1576,8 @@ static ssize_t ceph_write_iter(struct kiocb *iocb, struct iov_iter *from) } if (written >= 0) { - if (ceph_osdmap_flag(&fsc->client->osdc, CEPH_OSDMAP_NEARFULL)) + if (pool_flag(&fsc->client->osdc, ci->i_layout.pool_id, + CEPH_POOL_FLAG_NEARFULL)) iocb->ki_flags |= IOCB_DSYNC; written = generic_write_sync(iocb, written); } diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h index 5a62dbd3f4c2..be9007b93862 100644 --- a/include/linux/ceph/osd_client.h +++ b/include/linux/ceph/osd_client.h @@ -375,6 +375,8 @@ static inline bool ceph_osdmap_flag(struct ceph_osd_client *osdc, int flag) return osdc->osdmap->flags & flag; } +bool pool_flag(struct ceph_osd_client *osdc, s64 pool_id, int flag); + extern int ceph_osdc_setup(void); extern void ceph_osdc_cleanup(void); diff --git a/include/linux/ceph/osdmap.h b/include/linux/ceph/osdmap.h index e081b56f1c1d..88faacc11f55 100644 --- a/include/linux/ceph/osdmap.h +++ b/include/linux/ceph/osdmap.h @@ -36,7 +36,8 @@ int ceph_spg_compare(const struct ceph_spg *lhs, const struct ceph_spg *rhs); #define CEPH_POOL_FLAG_HASHPSPOOL (1ULL << 0) /* hash pg seed and pool id together */ -#define CEPH_POOL_FLAG_FULL (1ULL << 1) /* pool is full */ +#define CEPH_POOL_FLAG_FULL (1ULL << 1) /* pool is full */ +#define CEPH_POOL_FLAG_NEARFULL (1ULL << 11) /* pool is nearfull */ struct ceph_pg_pool_info { struct rb_node node; diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index b68b376d8c2f..9ad2b96c3e78 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -1447,9 +1447,9 @@ static void unlink_request(struct ceph_osd *osd, struct ceph_osd_request *req) atomic_dec(&osd->o_osdc->num_homeless); } -static bool __pool_full(struct ceph_pg_pool_info *pi) +static bool __pool_flag(struct ceph_pg_pool_info *pi, int flag) { - return pi->flags & CEPH_POOL_FLAG_FULL; + return pi->flags & flag; } static bool have_pool_full(struct ceph_osd_client *osdc) @@ -1460,14 +1460,14 @@ static bool have_pool_full(struct ceph_osd_client *osdc) struct ceph_pg_pool_info *pi = rb_entry(n, struct ceph_pg_pool_info, node); - if (__pool_full(pi)) + if (__pool_flag(pi, CEPH_POOL_FLAG_FULL)) return true; } return false; } -static bool pool_full(struct ceph_osd_client *osdc, s64 pool_id) +bool pool_flag(struct ceph_osd_client *osdc, s64 pool_id, int flag) { struct ceph_pg_pool_info *pi; @@ -1475,8 +1475,10 @@ static bool pool_full(struct ceph_osd_client *osdc, s64 pool_id) if (!pi) return false; - return __pool_full(pi); + return __pool_flag(pi, flag); } +EXPORT_SYMBOL(pool_flag); + /* * Returns whether a request should be blocked from being sent @@ -1489,7 +1491,7 @@ static bool target_should_be_paused(struct ceph_osd_client *osdc, bool pauserd = ceph_osdmap_flag(osdc, CEPH_OSDMAP_PAUSERD); bool pausewr = ceph_osdmap_flag(osdc, CEPH_OSDMAP_PAUSEWR) || ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || - __pool_full(pi); + __pool_flag(pi, CEPH_POOL_FLAG_FULL); WARN_ON(pi->id != t->target_oloc.pool); return ((t->flags & CEPH_OSD_FLAG_READ) && pauserd) || @@ -2320,7 +2322,8 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked) !(req->r_flags & (CEPH_OSD_FLAG_FULL_TRY | CEPH_OSD_FLAG_FULL_FORCE)) && (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || - pool_full(osdc, req->r_t.base_oloc.pool))) { + pool_flag(osdc, req->r_t.base_oloc.pool, + CEPH_POOL_FLAG_FULL))) { dout("req %p full/pool_full\n", req); if (ceph_test_opt(osdc->client, ABORT_ON_FULL)) { err = -ENOSPC; @@ -2539,7 +2542,7 @@ static int abort_on_full_fn(struct ceph_osd_request *req, void *arg) if ((req->r_flags & CEPH_OSD_FLAG_WRITE) && (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || - pool_full(osdc, req->r_t.base_oloc.pool))) { + pool_flag(osdc, req->r_t.base_oloc.pool, CEPH_POOL_FLAG_FULL))) { if (!*victims) { update_epoch_barrier(osdc, osdc->osdmap->epoch); *victims = true; @@ -3707,7 +3710,7 @@ static void set_pool_was_full(struct ceph_osd_client *osdc) struct ceph_pg_pool_info *pi = rb_entry(n, struct ceph_pg_pool_info, node); - pi->was_full = __pool_full(pi); + pi->was_full = __pool_flag(pi, CEPH_POOL_FLAG_FULL); } } @@ -3719,7 +3722,7 @@ static bool pool_cleared_full(struct ceph_osd_client *osdc, s64 pool_id) if (!pi) return false; - return pi->was_full && !__pool_full(pi); + return pi->was_full && !__pool_flag(pi, CEPH_POOL_FLAG_FULL); } static enum calc_target_result
OSDMAP_FULL and OSDMAP_NEARFULL are deprecated since mimic. Signed-off-by: Yanhu Cao <gmayyyha@gmail.com> --- fs/ceph/file.c | 6 ++++-- include/linux/ceph/osd_client.h | 2 ++ include/linux/ceph/osdmap.h | 3 ++- net/ceph/osd_client.c | 23 +++++++++++++---------- 4 files changed, 21 insertions(+), 13 deletions(-)