* [PATCH] libceph: add ignore cache/overlay flag if got redirect reply
@ 2020-05-18 8:03 Jerry Lee
2020-05-19 9:14 ` Ilya Dryomov
0 siblings, 1 reply; 5+ messages in thread
From: Jerry Lee @ 2020-05-18 8:03 UTC (permalink / raw
To: ceph-devel, Ilya Dryomov, Jeff Layton
osd client should ignore cache/overlay flag if got redirect reply.
Otherwise, the client hangs when the cache tier is in forward mode.
Similar issues:
https://tracker.ceph.com/issues/23296
https://tracker.ceph.com/issues/36406
Signed-off-by: Jerry Lee <leisurelysw24@gmail.com>
---
net/ceph/osd_client.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 998e26b..1d4973f 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -3649,7 +3649,9 @@ static void handle_reply(struct ceph_osd *osd,
struct ceph_msg *msg)
* supported.
*/
req->r_t.target_oloc.pool = m.redirect.oloc.pool;
- req->r_flags |= CEPH_OSD_FLAG_REDIRECTED;
+ req->r_flags |= CEPH_OSD_FLAG_REDIRECTED |
+ CEPH_OSD_FLAG_IGNORE_OVERLAY |
+ CEPH_OSD_FLAG_IGNORE_CACHE;
req->r_tid = 0;
__submit_request(req, false);
goto out_unlock_osdc;
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] libceph: add ignore cache/overlay flag if got redirect reply
2020-05-18 8:03 [PATCH] libceph: add ignore cache/overlay flag if got redirect reply Jerry Lee
@ 2020-05-19 9:14 ` Ilya Dryomov
2020-05-19 10:30 ` Jerry Lee
0 siblings, 1 reply; 5+ messages in thread
From: Ilya Dryomov @ 2020-05-19 9:14 UTC (permalink / raw
To: Jerry Lee; +Cc: ceph-devel, Jeff Layton
On Mon, May 18, 2020 at 10:03 AM Jerry Lee <leisurelysw24@gmail.com> wrote:
>
> osd client should ignore cache/overlay flag if got redirect reply.
> Otherwise, the client hangs when the cache tier is in forward mode.
>
> Similar issues:
> https://tracker.ceph.com/issues/23296
> https://tracker.ceph.com/issues/36406
>
> Signed-off-by: Jerry Lee <leisurelysw24@gmail.com>
> ---
> net/ceph/osd_client.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 998e26b..1d4973f 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -3649,7 +3649,9 @@ static void handle_reply(struct ceph_osd *osd,
> struct ceph_msg *msg)
> * supported.
> */
> req->r_t.target_oloc.pool = m.redirect.oloc.pool;
> - req->r_flags |= CEPH_OSD_FLAG_REDIRECTED;
> + req->r_flags |= CEPH_OSD_FLAG_REDIRECTED |
> + CEPH_OSD_FLAG_IGNORE_OVERLAY |
> + CEPH_OSD_FLAG_IGNORE_CACHE;
> req->r_tid = 0;
> __submit_request(req, false);
> goto out_unlock_osdc;
Hi Jerry,
Looks good (although the patch was whitespace damaged). I've fixed
it up, but check out Documentation/process/email-clients.rst.
Also, out of curiosity, are you actually using the forward cache mode?
Thanks,
Ilya
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] libceph: add ignore cache/overlay flag if got redirect reply
2020-05-19 9:14 ` Ilya Dryomov
@ 2020-05-19 10:30 ` Jerry Lee
2020-05-19 13:32 ` Ilya Dryomov
0 siblings, 1 reply; 5+ messages in thread
From: Jerry Lee @ 2020-05-19 10:30 UTC (permalink / raw
To: Ilya Dryomov; +Cc: ceph-devel, Jeff Layton
On Tue, 19 May 2020 at 17:14, Ilya Dryomov <idryomov@gmail.com> wrote:
>
> On Mon, May 18, 2020 at 10:03 AM Jerry Lee <leisurelysw24@gmail.com> wrote:
> >
> > osd client should ignore cache/overlay flag if got redirect reply.
> > Otherwise, the client hangs when the cache tier is in forward mode.
> >
> > Similar issues:
> > https://tracker.ceph.com/issues/23296
> > https://tracker.ceph.com/issues/36406
> >
> > Signed-off-by: Jerry Lee <leisurelysw24@gmail.com>
> > ---
> > net/ceph/osd_client.c | 4 +++-
> > 1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> > index 998e26b..1d4973f 100644
> > --- a/net/ceph/osd_client.c
> > +++ b/net/ceph/osd_client.c
> > @@ -3649,7 +3649,9 @@ static void handle_reply(struct ceph_osd *osd,
> > struct ceph_msg *msg)
> > * supported.
> > */
> > req->r_t.target_oloc.pool = m.redirect.oloc.pool;
> > - req->r_flags |= CEPH_OSD_FLAG_REDIRECTED;
> > + req->r_flags |= CEPH_OSD_FLAG_REDIRECTED |
> > + CEPH_OSD_FLAG_IGNORE_OVERLAY |
> > + CEPH_OSD_FLAG_IGNORE_CACHE;
> > req->r_tid = 0;
> > __submit_request(req, false);
> > goto out_unlock_osdc;
>
> Hi Jerry,
>
> Looks good (although the patch was whitespace damaged). I've fixed
> it up, but check out Documentation/process/email-clients.rst.
Thanks for sharing the doc!
>
> Also, out of curiosity, are you actually using the forward cache mode?
No, we accidentally found the issue when removing a writeback cache.
The kernel client got blocked when the cache mode switched from
writeback to forward and waited for the cache pool to be flushed.
BTW, a warning (Error EPERM: 'forward' is not a well-supported cache
mode and may corrupt your data.) is shown when the cache mode is
changed to forward mode. Does it mean that the data integrity and IO
ordering cannot be ensured in this mode?
Thanks!
- Jerry
>
> Thanks,
>
> Ilya
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] libceph: add ignore cache/overlay flag if got redirect reply
2020-05-19 10:30 ` Jerry Lee
@ 2020-05-19 13:32 ` Ilya Dryomov
2020-05-20 2:35 ` Jerry Lee
0 siblings, 1 reply; 5+ messages in thread
From: Ilya Dryomov @ 2020-05-19 13:32 UTC (permalink / raw
To: Jerry Lee; +Cc: ceph-devel, Jeff Layton
On Tue, May 19, 2020 at 12:30 PM Jerry Lee <leisurelysw24@gmail.com> wrote:
>
> On Tue, 19 May 2020 at 17:14, Ilya Dryomov <idryomov@gmail.com> wrote:
> >
> > On Mon, May 18, 2020 at 10:03 AM Jerry Lee <leisurelysw24@gmail.com> wrote:
> > >
> > > osd client should ignore cache/overlay flag if got redirect reply.
> > > Otherwise, the client hangs when the cache tier is in forward mode.
> > >
> > > Similar issues:
> > > https://tracker.ceph.com/issues/23296
> > > https://tracker.ceph.com/issues/36406
> > >
> > > Signed-off-by: Jerry Lee <leisurelysw24@gmail.com>
> > > ---
> > > net/ceph/osd_client.c | 4 +++-
> > > 1 file changed, 3 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> > > index 998e26b..1d4973f 100644
> > > --- a/net/ceph/osd_client.c
> > > +++ b/net/ceph/osd_client.c
> > > @@ -3649,7 +3649,9 @@ static void handle_reply(struct ceph_osd *osd,
> > > struct ceph_msg *msg)
> > > * supported.
> > > */
> > > req->r_t.target_oloc.pool = m.redirect.oloc.pool;
> > > - req->r_flags |= CEPH_OSD_FLAG_REDIRECTED;
> > > + req->r_flags |= CEPH_OSD_FLAG_REDIRECTED |
> > > + CEPH_OSD_FLAG_IGNORE_OVERLAY |
> > > + CEPH_OSD_FLAG_IGNORE_CACHE;
> > > req->r_tid = 0;
> > > __submit_request(req, false);
> > > goto out_unlock_osdc;
> >
> > Hi Jerry,
> >
> > Looks good (although the patch was whitespace damaged). I've fixed
> > it up, but check out Documentation/process/email-clients.rst.
> Thanks for sharing the doc!
> >
> > Also, out of curiosity, are you actually using the forward cache mode?
> No, we accidentally found the issue when removing a writeback cache.
> The kernel client got blocked when the cache mode switched from
> writeback to forward and waited for the cache pool to be flushed.
>
> BTW, a warning (Error EPERM: 'forward' is not a well-supported cache
> mode and may corrupt your data.) is shown when the cache mode is
> changed to forward mode. Does it mean that the data integrity and IO
> ordering cannot be ensured in this mode?
Yes. The problem with redirects is that they can mess up the order
of requests. The forward mode is based on redirects and therefore
inherently flawed.
Use proxy and readproxy modes instead of forward and readforward.
Thanks,
Ilya
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] libceph: add ignore cache/overlay flag if got redirect reply
2020-05-19 13:32 ` Ilya Dryomov
@ 2020-05-20 2:35 ` Jerry Lee
0 siblings, 0 replies; 5+ messages in thread
From: Jerry Lee @ 2020-05-20 2:35 UTC (permalink / raw
To: Ilya Dryomov; +Cc: ceph-devel, Jeff Layton
On Tue, 19 May 2020 at 21:32, Ilya Dryomov <idryomov@gmail.com> wrote:
>
> On Tue, May 19, 2020 at 12:30 PM Jerry Lee <leisurelysw24@gmail.com> wrote:
> >
> > On Tue, 19 May 2020 at 17:14, Ilya Dryomov <idryomov@gmail.com> wrote:
> > >
> > > On Mon, May 18, 2020 at 10:03 AM Jerry Lee <leisurelysw24@gmail.com> wrote:
> > > >
> > > > osd client should ignore cache/overlay flag if got redirect reply.
> > > > Otherwise, the client hangs when the cache tier is in forward mode.
> > > >
> > > > Similar issues:
> > > > https://tracker.ceph.com/issues/23296
> > > > https://tracker.ceph.com/issues/36406
> > > >
> > > > Signed-off-by: Jerry Lee <leisurelysw24@gmail.com>
> > > > ---
> > > > net/ceph/osd_client.c | 4 +++-
> > > > 1 file changed, 3 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> > > > index 998e26b..1d4973f 100644
> > > > --- a/net/ceph/osd_client.c
> > > > +++ b/net/ceph/osd_client.c
> > > > @@ -3649,7 +3649,9 @@ static void handle_reply(struct ceph_osd *osd,
> > > > struct ceph_msg *msg)
> > > > * supported.
> > > > */
> > > > req->r_t.target_oloc.pool = m.redirect.oloc.pool;
> > > > - req->r_flags |= CEPH_OSD_FLAG_REDIRECTED;
> > > > + req->r_flags |= CEPH_OSD_FLAG_REDIRECTED |
> > > > + CEPH_OSD_FLAG_IGNORE_OVERLAY |
> > > > + CEPH_OSD_FLAG_IGNORE_CACHE;
> > > > req->r_tid = 0;
> > > > __submit_request(req, false);
> > > > goto out_unlock_osdc;
> > >
> > > Hi Jerry,
> > >
> > > Looks good (although the patch was whitespace damaged). I've fixed
> > > it up, but check out Documentation/process/email-clients.rst.
> > Thanks for sharing the doc!
> > >
> > > Also, out of curiosity, are you actually using the forward cache mode?
> > No, we accidentally found the issue when removing a writeback cache.
> > The kernel client got blocked when the cache mode switched from
> > writeback to forward and waited for the cache pool to be flushed.
> >
> > BTW, a warning (Error EPERM: 'forward' is not a well-supported cache
> > mode and may corrupt your data.) is shown when the cache mode is
> > changed to forward mode. Does it mean that the data integrity and IO
> > ordering cannot be ensured in this mode?
>
> Yes. The problem with redirects is that they can mess up the order
> of requests. The forward mode is based on redirects and therefore
> inherently flawed.
>
> Use proxy and readproxy modes instead of forward and readforward.
>
Thanks for the clarification. I refer to the mimic version
cache-tering configuration guide which states that forward mode is
configured when removing a writeback cache. However, in the
up-to-date doc (master), proxy mode is recommended. I'll use proxy
mode instead.
Thanks,
- Jerry
> Thanks,
>
> Ilya
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-05-20 2:36 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-05-18 8:03 [PATCH] libceph: add ignore cache/overlay flag if got redirect reply Jerry Lee
2020-05-19 9:14 ` Ilya Dryomov
2020-05-19 10:30 ` Jerry Lee
2020-05-19 13:32 ` Ilya Dryomov
2020-05-20 2:35 ` Jerry Lee
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.