| Message ID | 1527702222-8232-8-git-send-email-idryomov@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, 2018-05-30 at 19:43 +0200, Ilya Dryomov wrote: > The intent behind making it a per-request setting was that it would be > set for writes, but not for reads. As it is, the flag is set for all > fs/ceph requests except for pool perm check stat request (technically > a read). > > ceph_osdc_abort_on_full() skips reads since the previous commit and > I don't see a use case for marking individual requests. > > Signed-off-by: Ilya Dryomov <idryomov@gmail.com> I'm assuming that it's not possible for (e.g.) an RBD and CephFS client to share an osdc? We'd almost certainly want different semantics in those two cases, I'd think. Looking at the code, it seems like it's not possible for now. If we think that might be something to pursue in the future, then we may need to rethink this. > --- > fs/ceph/addr.c | 1 - > fs/ceph/file.c | 1 - > fs/ceph/super.c | 2 ++ > include/linux/ceph/osd_client.h | 2 +- > net/ceph/osd_client.c | 9 ++++----- > 5 files changed, 7 insertions(+), 8 deletions(-) > > diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c > index 5f7ad3d0df2e..ca0d5510ed50 100644 > --- a/fs/ceph/addr.c > +++ b/fs/ceph/addr.c > @@ -1935,7 +1935,6 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, > err = ceph_osdc_start_request(&fsc->client->osdc, rd_req, false); > > wr_req->r_mtime = ci->vfs_inode.i_mtime; > - wr_req->r_abort_on_full = true; > err2 = ceph_osdc_start_request(&fsc->client->osdc, wr_req, false); > > if (!err) > diff --git a/fs/ceph/file.c b/fs/ceph/file.c > index cf0e45b10121..6b9f7f3cd237 100644 > --- a/fs/ceph/file.c > +++ b/fs/ceph/file.c > @@ -895,7 +895,6 @@ static void ceph_aio_retry_work(struct work_struct *work) > req->r_callback = ceph_aio_complete_req; > req->r_inode = inode; > req->r_priv = aio_req; > - req->r_abort_on_full = true; > > ret = ceph_osdc_start_request(req->r_osdc, req, false); > out: > diff --git a/fs/ceph/super.c b/fs/ceph/super.c > index a092cdb69288..cad046aa4fd0 100644 > --- a/fs/ceph/super.c > +++ b/fs/ceph/super.c > @@ -616,7 +616,9 @@ static struct ceph_fs_client *create_fs_client(struct ceph_mount_options *fsopt, > err = PTR_ERR(fsc->client); > goto fail; > } > + > fsc->client->extra_mon_dispatch = extra_mon_dispatch; > + fsc->client->osdc.abort_on_full = true; > > if (!fsopt->mds_namespace) { > ceph_monc_want_map(&fsc->client->monc, CEPH_SUB_MDSMAP, > diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h > index d4191bde95a4..0d6ee04b4c41 100644 > --- a/include/linux/ceph/osd_client.h > +++ b/include/linux/ceph/osd_client.h > @@ -202,7 +202,6 @@ struct ceph_osd_request { > struct timespec r_mtime; /* ditto */ > u64 r_data_offset; /* ditto */ > bool r_linger; /* don't resend on failure */ > - bool r_abort_on_full; /* return ENOSPC when full */ > > /* internal */ > unsigned long r_stamp; /* jiffies, send or check time */ > @@ -348,6 +347,7 @@ struct ceph_osd_client { > struct rb_root linger_map_checks; > atomic_t num_requests; > atomic_t num_homeless; > + bool abort_on_full; /* abort w/ ENOSPC when full */ > int abort_err; > struct delayed_work timeout_work; > struct delayed_work osds_timeout_work; > diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c > index 3d055529189c..05c4d27d25fe 100644 > --- a/net/ceph/osd_client.c > +++ b/net/ceph/osd_client.c > @@ -1030,7 +1030,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc, > truncate_size, truncate_seq); > } > > - req->r_abort_on_full = true; > req->r_flags = flags; > req->r_base_oloc.pool = layout->pool_id; > req->r_base_oloc.pool_ns = ceph_try_get_string(layout->pool_ns); > @@ -2239,7 +2238,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked) > (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || > pool_full(osdc, req->r_t.base_oloc.pool))) { > dout("req %p full/pool_full\n", req); > - if (req->r_abort_on_full) { > + if (osdc->abort_on_full) { > err = -ENOSPC; > } else { > pr_warn_ratelimited("FULL or reached pool quota\n"); > @@ -2446,8 +2445,7 @@ static int abort_on_full_fn(struct ceph_osd_request *req, void *arg) > struct ceph_osd_client *osdc = req->r_osdc; > bool *victims = arg; > > - if (req->r_abort_on_full && > - (req->r_flags & CEPH_OSD_FLAG_WRITE) && > + if ((req->r_flags & CEPH_OSD_FLAG_WRITE) && > (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || > pool_full(osdc, req->r_t.base_oloc.pool))) { > if (!*victims) { > @@ -2470,7 +2468,8 @@ static void ceph_osdc_abort_on_full(struct ceph_osd_client *osdc) > { > bool victims = false; > > - if (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || have_pool_full(osdc)) > + if (osdc->abort_on_full && > + (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || have_pool_full(osdc))) > for_each_request(osdc, abort_on_full_fn, &victims); > } >
On Wed, May 30, 2018 at 8:50 PM, Jeff Layton <jlayton@redhat.com> wrote: > On Wed, 2018-05-30 at 19:43 +0200, Ilya Dryomov wrote: >> The intent behind making it a per-request setting was that it would be >> set for writes, but not for reads. As it is, the flag is set for all >> fs/ceph requests except for pool perm check stat request (technically >> a read). >> >> ceph_osdc_abort_on_full() skips reads since the previous commit and >> I don't see a use case for marking individual requests. >> >> Signed-off-by: Ilya Dryomov <idryomov@gmail.com> > > I'm assuming that it's not possible for (e.g.) an RBD and CephFS client > to share an osdc? We'd almost certainly want different semantics in > those two cases, I'd think. > > Looking at the code, it seems like it's not possible for now. If we > think that might be something to pursue in the future, then we may need > to rethink this. No, it is not and has never been possible. Thanks, Ilya -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 5f7ad3d0df2e..ca0d5510ed50 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -1935,7 +1935,6 @@ static int __ceph_pool_perm_get(struct ceph_inode_info *ci, err = ceph_osdc_start_request(&fsc->client->osdc, rd_req, false); wr_req->r_mtime = ci->vfs_inode.i_mtime; - wr_req->r_abort_on_full = true; err2 = ceph_osdc_start_request(&fsc->client->osdc, wr_req, false); if (!err) diff --git a/fs/ceph/file.c b/fs/ceph/file.c index cf0e45b10121..6b9f7f3cd237 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -895,7 +895,6 @@ static void ceph_aio_retry_work(struct work_struct *work) req->r_callback = ceph_aio_complete_req; req->r_inode = inode; req->r_priv = aio_req; - req->r_abort_on_full = true; ret = ceph_osdc_start_request(req->r_osdc, req, false); out: diff --git a/fs/ceph/super.c b/fs/ceph/super.c index a092cdb69288..cad046aa4fd0 100644 --- a/fs/ceph/super.c +++ b/fs/ceph/super.c @@ -616,7 +616,9 @@ static struct ceph_fs_client *create_fs_client(struct ceph_mount_options *fsopt, err = PTR_ERR(fsc->client); goto fail; } + fsc->client->extra_mon_dispatch = extra_mon_dispatch; + fsc->client->osdc.abort_on_full = true; if (!fsopt->mds_namespace) { ceph_monc_want_map(&fsc->client->monc, CEPH_SUB_MDSMAP, diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h index d4191bde95a4..0d6ee04b4c41 100644 --- a/include/linux/ceph/osd_client.h +++ b/include/linux/ceph/osd_client.h @@ -202,7 +202,6 @@ struct ceph_osd_request { struct timespec r_mtime; /* ditto */ u64 r_data_offset; /* ditto */ bool r_linger; /* don't resend on failure */ - bool r_abort_on_full; /* return ENOSPC when full */ /* internal */ unsigned long r_stamp; /* jiffies, send or check time */ @@ -348,6 +347,7 @@ struct ceph_osd_client { struct rb_root linger_map_checks; atomic_t num_requests; atomic_t num_homeless; + bool abort_on_full; /* abort w/ ENOSPC when full */ int abort_err; struct delayed_work timeout_work; struct delayed_work osds_timeout_work; diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index 3d055529189c..05c4d27d25fe 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -1030,7 +1030,6 @@ struct ceph_osd_request *ceph_osdc_new_request(struct ceph_osd_client *osdc, truncate_size, truncate_seq); } - req->r_abort_on_full = true; req->r_flags = flags; req->r_base_oloc.pool = layout->pool_id; req->r_base_oloc.pool_ns = ceph_try_get_string(layout->pool_ns); @@ -2239,7 +2238,7 @@ static void __submit_request(struct ceph_osd_request *req, bool wrlocked) (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || pool_full(osdc, req->r_t.base_oloc.pool))) { dout("req %p full/pool_full\n", req); - if (req->r_abort_on_full) { + if (osdc->abort_on_full) { err = -ENOSPC; } else { pr_warn_ratelimited("FULL or reached pool quota\n"); @@ -2446,8 +2445,7 @@ static int abort_on_full_fn(struct ceph_osd_request *req, void *arg) struct ceph_osd_client *osdc = req->r_osdc; bool *victims = arg; - if (req->r_abort_on_full && - (req->r_flags & CEPH_OSD_FLAG_WRITE) && + if ((req->r_flags & CEPH_OSD_FLAG_WRITE) && (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || pool_full(osdc, req->r_t.base_oloc.pool))) { if (!*victims) { @@ -2470,7 +2468,8 @@ static void ceph_osdc_abort_on_full(struct ceph_osd_client *osdc) { bool victims = false; - if (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || have_pool_full(osdc)) + if (osdc->abort_on_full && + (ceph_osdmap_flag(osdc, CEPH_OSDMAP_FULL) || have_pool_full(osdc))) for_each_request(osdc, abort_on_full_fn, &victims); }
The intent behind making it a per-request setting was that it would be set for writes, but not for reads. As it is, the flag is set for all fs/ceph requests except for pool perm check stat request (technically a read). ceph_osdc_abort_on_full() skips reads since the previous commit and I don't see a use case for marking individual requests. Signed-off-by: Ilya Dryomov <idryomov@gmail.com> --- fs/ceph/addr.c | 1 - fs/ceph/file.c | 1 - fs/ceph/super.c | 2 ++ include/linux/ceph/osd_client.h | 2 +- net/ceph/osd_client.c | 9 ++++----- 5 files changed, 7 insertions(+), 8 deletions(-)