| Message ID | 20211127101132.486806-10-yukuai3@huawei.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | support concurrent sync io for bfq on a specail occasion | expand |
> Il giorno 27 nov 2021, alle ore 11:11, Yu Kuai <yukuai3@huawei.com> ha scritto: > > Currently 'num_groups_with_pending_reqs' won't be decreased when > the group doesn't have any pending requests, while any child group > have any pending requests. The decrement is delayed to when all the > child groups doesn't have any pending requests. > > For example: > 1) t1 issue sync io on root group, t2 and t3 issue sync io on the same > child group. num_groups_with_pending_reqs is 2 now. > 2) t1 stopped, num_groups_with_pending_reqs is still 2. io from t2 and > t3 still can't be handled concurrently. > > Fix the problem by decreasing 'num_groups_with_pending_reqs' > immediately upon the deactivation of last entity of the group. > I don't understand this patch clearly. I understand your proposal not to count a group as with pending requests, in case no child process of the group has IO, but only its child groups have pending requests. So, entities here are only queues for this patch? If they are only queues, I think it is still incorrect to remove the group from the count of groups with pending IO when all its child queues are deactivated, because there may still be unfinished IO for those queues. Am I missing something? Thanks, Paolo > Signed-off-by: Yu Kuai <yukuai3@huawei.com> > --- > block/bfq-iosched.c | 58 ++++++++++++++++----------------------------- > block/bfq-iosched.h | 16 ++++++------- > 2 files changed, 29 insertions(+), 45 deletions(-) > > diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c > index 4239b3996e23..55925e1ee85d 100644 > --- a/block/bfq-iosched.c > +++ b/block/bfq-iosched.c > @@ -873,6 +873,26 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd, > bfq_put_queue(bfqq); > } > > +static void decrease_groups_with_pending_reqs(struct bfq_data *bfqd, > + struct bfq_queue *bfqq) > +{ > +#ifdef CONFIG_BFQ_GROUP_IOSCHED > + struct bfq_entity *entity = bfqq->entity.parent; > + struct bfq_group *bfqg = container_of(entity, struct bfq_group, entity); > + > + /* > + * The decrement of num_groups_with_pending_reqs is performed > + * immediately upon the deactivation of last entity that have pending > + * requests > + */ > + if (!bfqg->num_entities_with_pending_reqs && > + entity->in_groups_with_pending_reqs) { > + entity->in_groups_with_pending_reqs = false; > + bfqd->num_groups_with_pending_reqs--; > + } > +#endif > +} > + > /* > * Invoke __bfq_weights_tree_remove on bfqq and decrement the number > * of active groups for each queue's inactive parent entity. > @@ -880,46 +900,10 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd, > void bfq_weights_tree_remove(struct bfq_data *bfqd, > struct bfq_queue *bfqq) > { > - struct bfq_entity *entity = bfqq->entity.parent; > - > bfqq->ref++; > __bfq_weights_tree_remove(bfqd, bfqq, > &bfqd->queue_weights_tree); > - > - for_each_entity(entity) { > - struct bfq_sched_data *sd = entity->my_sched_data; > - > - if (sd && (sd->next_in_service || sd->in_service_entity)) { > - /* > - * entity is still active, because either > - * next_in_service or in_service_entity is not > - * NULL (see the comments on the definition of > - * next_in_service for details on why > - * in_service_entity must be checked too). > - * > - * As a consequence, its parent entities are > - * active as well, and thus this loop must > - * stop here. > - */ > - break; > - } > - > - /* > - * The decrement of num_groups_with_pending_reqs is > - * not performed immediately upon the deactivation of > - * entity, but it is delayed to when it also happens > - * that the first leaf descendant bfqq of entity gets > - * all its pending requests completed. The following > - * instructions perform this delayed decrement, if > - * needed. See the comments on > - * num_groups_with_pending_reqs for details. > - */ > - if (entity->in_groups_with_pending_reqs) { > - entity->in_groups_with_pending_reqs = false; > - bfqd->num_groups_with_pending_reqs--; > - } > - } > - > + decrease_groups_with_pending_reqs(bfqd, bfqq); > bfq_put_queue(bfqq); > } > > diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h > index df08bff89a70..7ae11f62900b 100644 > --- a/block/bfq-iosched.h > +++ b/block/bfq-iosched.h > @@ -493,7 +493,7 @@ struct bfq_data { > struct rb_root_cached queue_weights_tree; > > /* > - * Number of groups with at least one descendant process that > + * Number of groups with at least one process that > * has at least one request waiting for completion. Note that > * this accounts for also requests already dispatched, but not > * yet completed. Therefore this number of groups may differ > @@ -506,14 +506,14 @@ struct bfq_data { > * bfq_better_to_idle(). > * > * However, it is hard to compute this number exactly, for > - * groups with multiple descendant processes. Consider a group > - * that is inactive, i.e., that has no descendant process with > + * groups with multiple processes. Consider a group > + * that is inactive, i.e., that has no process with > * pending I/O inside BFQ queues. Then suppose that > * num_groups_with_pending_reqs is still accounting for this > - * group, because the group has descendant processes with some > + * group, because the group has processes with some > * I/O request still in flight. num_groups_with_pending_reqs > * should be decremented when the in-flight request of the > - * last descendant process is finally completed (assuming that > + * last process is finally completed (assuming that > * nothing else has changed for the group in the meantime, in > * terms of composition of the group and active/inactive state of child > * groups and processes). To accomplish this, an additional > @@ -522,7 +522,7 @@ struct bfq_data { > * we resort to the following tradeoff between simplicity and > * accuracy: for an inactive group that is still counted in > * num_groups_with_pending_reqs, we decrement > - * num_groups_with_pending_reqs when the first descendant > + * num_groups_with_pending_reqs when the last > * process of the group remains with no request waiting for > * completion. > * > @@ -530,12 +530,12 @@ struct bfq_data { > * carefulness: to avoid multiple decrements, we flag a group, > * more precisely an entity representing a group, as still > * counted in num_groups_with_pending_reqs when it becomes > - * inactive. Then, when the first descendant queue of the > + * inactive. Then, when the last queue of the > * entity remains with no request waiting for completion, > * num_groups_with_pending_reqs is decremented, and this flag > * is reset. After this flag is reset for the entity, > * num_groups_with_pending_reqs won't be decremented any > - * longer in case a new descendant queue of the entity remains > + * longer in case a new queue of the entity remains > * with no request waiting for completion. > */ > unsigned int num_groups_with_pending_reqs; > -- > 2.31.1 >
在 2021/12/10 18:21, Paolo Valente 写道: > >> Il giorno 27 nov 2021, alle ore 11:11, Yu Kuai<yukuai3@huawei.com> ha scritto: >> >> Currently 'num_groups_with_pending_reqs' won't be decreased when >> the group doesn't have any pending requests, while any child group >> have any pending requests. The decrement is delayed to when all the >> child groups doesn't have any pending requests. >> >> For example: >> 1) t1 issue sync io on root group, t2 and t3 issue sync io on the same >> child group. num_groups_with_pending_reqs is 2 now. >> 2) t1 stopped, num_groups_with_pending_reqs is still 2. io from t2 and >> t3 still can't be handled concurrently. >> >> Fix the problem by decreasing 'num_groups_with_pending_reqs' >> immediately upon the deactivation of last entity of the group. >> > I don't understand this patch clearly. > > I understand your proposal not to count a group as with pending requests, in case no child process of the group has IO, but only its child groups have pending requests. > > So, entities here are only queues for this patch? > > If they are only queues, I think it is still incorrect to remove the group from the count of groups with pending IO when all its child queues are deactivated, because there may still be unfinished IO for those queues. Hi, Paolo bfq_weights_tree_remove() will be called when all requests are completed in bfq_queue, thus I recored how many queues have pending requests through weights tree insertion and removal.(Details in patch 7) Thus when calling bfq_weights_tree_remove() for bfqq, I can check if there are no queues have pending requests for parent bfqg: if (!bfqg->num_entities_with_pending_reqs && -> no queues with pending reqs entity->in_groups_with_pending_reqs) { -> the group is counted Thanks, Kuai > > Am I missing something? > > Thanks, > Paolo >
> Il giorno 11 dic 2021, alle ore 03:10, yukuai (C) <yukuai3@huawei.com> ha scritto: > > 在 2021/12/10 18:21, Paolo Valente 写道: >>> Il giorno 27 nov 2021, alle ore 11:11, Yu Kuai<yukuai3@huawei.com> ha scritto: >>> >>> Currently 'num_groups_with_pending_reqs' won't be decreased when >>> the group doesn't have any pending requests, while any child group >>> have any pending requests. The decrement is delayed to when all the >>> child groups doesn't have any pending requests. >>> >>> For example: >>> 1) t1 issue sync io on root group, t2 and t3 issue sync io on the same >>> child group. num_groups_with_pending_reqs is 2 now. >>> 2) t1 stopped, num_groups_with_pending_reqs is still 2. io from t2 and >>> t3 still can't be handled concurrently. >>> >>> Fix the problem by decreasing 'num_groups_with_pending_reqs' >>> immediately upon the deactivation of last entity of the group. >>> >> I don't understand this patch clearly. >> I understand your proposal not to count a group as with pending requests, in case no child process of the group has IO, but only its child groups have pending requests. >> So, entities here are only queues for this patch? >> If they are only queues, I think it is still incorrect to remove the group from the count of groups with pending IO when all its child queues are deactivated, because there may still be unfinished IO for those queues. > > Hi, Paolo > > bfq_weights_tree_remove() will be called when all requests are completed > in bfq_queue, thus I recored how many queues have pending requests > through weights tree insertion and removal.(Details in patch 7) > > Thus when calling bfq_weights_tree_remove() for bfqq, I can check if > there are no queues have pending requests for parent bfqg: > > if (!bfqg->num_entities_with_pending_reqs && -> no queues with pending reqs > entity->in_groups_with_pending_reqs) { -> the group is counted > Ok, I got confused because you use the term deactivation. Yet you seem to decrement the counter at the right time. Maybe fix that term, in commit messages and comments. Thanks, Paolo > Thanks, > Kuai >> Am I missing something? >> Thanks, >> Paolo
在 2021/12/17 0:34, Paolo Valente 写道: > > >> Il giorno 11 dic 2021, alle ore 03:10, yukuai (C) <yukuai3@huawei.com> ha scritto: >> >> 在 2021/12/10 18:21, Paolo Valente 写道: >>>> Il giorno 27 nov 2021, alle ore 11:11, Yu Kuai<yukuai3@huawei.com> ha scritto: >>>> >>>> Currently 'num_groups_with_pending_reqs' won't be decreased when >>>> the group doesn't have any pending requests, while any child group >>>> have any pending requests. The decrement is delayed to when all the >>>> child groups doesn't have any pending requests. >>>> >>>> For example: >>>> 1) t1 issue sync io on root group, t2 and t3 issue sync io on the same >>>> child group. num_groups_with_pending_reqs is 2 now. >>>> 2) t1 stopped, num_groups_with_pending_reqs is still 2. io from t2 and >>>> t3 still can't be handled concurrently. >>>> >>>> Fix the problem by decreasing 'num_groups_with_pending_reqs' >>>> immediately upon the deactivation of last entity of the group. >>>> >>> I don't understand this patch clearly. >>> I understand your proposal not to count a group as with pending requests, in case no child process of the group has IO, but only its child groups have pending requests. >>> So, entities here are only queues for this patch? >>> If they are only queues, I think it is still incorrect to remove the group from the count of groups with pending IO when all its child queues are deactivated, because there may still be unfinished IO for those queues. >> >> Hi, Paolo >> >> bfq_weights_tree_remove() will be called when all requests are completed >> in bfq_queue, thus I recored how many queues have pending requests >> through weights tree insertion and removal.(Details in patch 7) >> >> Thus when calling bfq_weights_tree_remove() for bfqq, I can check if >> there are no queues have pending requests for parent bfqg: >> >> if (!bfqg->num_entities_with_pending_reqs && -> no queues with pending reqs >> entity->in_groups_with_pending_reqs) { -> the group is counted >> > > Ok, I got confused because you use the term deactivation. Yet you > seem to decrement the counter at the right time. Maybe fix that term, > in commit messages and comments. Ok, I'll fix that term, and thanks for taking time reviewing these patches, > > Thanks, > Paolo > >> Thanks, >> Kuai >>> Am I missing something? >>> Thanks, >>> Paolo > > . >
diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c index 4239b3996e23..55925e1ee85d 100644 --- a/block/bfq-iosched.c +++ b/block/bfq-iosched.c @@ -873,6 +873,26 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd, bfq_put_queue(bfqq); } +static void decrease_groups_with_pending_reqs(struct bfq_data *bfqd, + struct bfq_queue *bfqq) +{ +#ifdef CONFIG_BFQ_GROUP_IOSCHED + struct bfq_entity *entity = bfqq->entity.parent; + struct bfq_group *bfqg = container_of(entity, struct bfq_group, entity); + + /* + * The decrement of num_groups_with_pending_reqs is performed + * immediately upon the deactivation of last entity that have pending + * requests + */ + if (!bfqg->num_entities_with_pending_reqs && + entity->in_groups_with_pending_reqs) { + entity->in_groups_with_pending_reqs = false; + bfqd->num_groups_with_pending_reqs--; + } +#endif +} + /* * Invoke __bfq_weights_tree_remove on bfqq and decrement the number * of active groups for each queue's inactive parent entity. @@ -880,46 +900,10 @@ void __bfq_weights_tree_remove(struct bfq_data *bfqd, void bfq_weights_tree_remove(struct bfq_data *bfqd, struct bfq_queue *bfqq) { - struct bfq_entity *entity = bfqq->entity.parent; - bfqq->ref++; __bfq_weights_tree_remove(bfqd, bfqq, &bfqd->queue_weights_tree); - - for_each_entity(entity) { - struct bfq_sched_data *sd = entity->my_sched_data; - - if (sd && (sd->next_in_service || sd->in_service_entity)) { - /* - * entity is still active, because either - * next_in_service or in_service_entity is not - * NULL (see the comments on the definition of - * next_in_service for details on why - * in_service_entity must be checked too). - * - * As a consequence, its parent entities are - * active as well, and thus this loop must - * stop here. - */ - break; - } - - /* - * The decrement of num_groups_with_pending_reqs is - * not performed immediately upon the deactivation of - * entity, but it is delayed to when it also happens - * that the first leaf descendant bfqq of entity gets - * all its pending requests completed. The following - * instructions perform this delayed decrement, if - * needed. See the comments on - * num_groups_with_pending_reqs for details. - */ - if (entity->in_groups_with_pending_reqs) { - entity->in_groups_with_pending_reqs = false; - bfqd->num_groups_with_pending_reqs--; - } - } - + decrease_groups_with_pending_reqs(bfqd, bfqq); bfq_put_queue(bfqq); } diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h index df08bff89a70..7ae11f62900b 100644 --- a/block/bfq-iosched.h +++ b/block/bfq-iosched.h @@ -493,7 +493,7 @@ struct bfq_data { struct rb_root_cached queue_weights_tree; /* - * Number of groups with at least one descendant process that + * Number of groups with at least one process that * has at least one request waiting for completion. Note that * this accounts for also requests already dispatched, but not * yet completed. Therefore this number of groups may differ @@ -506,14 +506,14 @@ struct bfq_data { * bfq_better_to_idle(). * * However, it is hard to compute this number exactly, for - * groups with multiple descendant processes. Consider a group - * that is inactive, i.e., that has no descendant process with + * groups with multiple processes. Consider a group + * that is inactive, i.e., that has no process with * pending I/O inside BFQ queues. Then suppose that * num_groups_with_pending_reqs is still accounting for this - * group, because the group has descendant processes with some + * group, because the group has processes with some * I/O request still in flight. num_groups_with_pending_reqs * should be decremented when the in-flight request of the - * last descendant process is finally completed (assuming that + * last process is finally completed (assuming that * nothing else has changed for the group in the meantime, in * terms of composition of the group and active/inactive state of child * groups and processes). To accomplish this, an additional @@ -522,7 +522,7 @@ struct bfq_data { * we resort to the following tradeoff between simplicity and * accuracy: for an inactive group that is still counted in * num_groups_with_pending_reqs, we decrement - * num_groups_with_pending_reqs when the first descendant + * num_groups_with_pending_reqs when the last * process of the group remains with no request waiting for * completion. * @@ -530,12 +530,12 @@ struct bfq_data { * carefulness: to avoid multiple decrements, we flag a group, * more precisely an entity representing a group, as still * counted in num_groups_with_pending_reqs when it becomes - * inactive. Then, when the first descendant queue of the + * inactive. Then, when the last queue of the * entity remains with no request waiting for completion, * num_groups_with_pending_reqs is decremented, and this flag * is reset. After this flag is reset for the entity, * num_groups_with_pending_reqs won't be decremented any - * longer in case a new descendant queue of the entity remains + * longer in case a new queue of the entity remains * with no request waiting for completion. */ unsigned int num_groups_with_pending_reqs;
Currently 'num_groups_with_pending_reqs' won't be decreased when the group doesn't have any pending requests, while any child group have any pending requests. The decrement is delayed to when all the child groups doesn't have any pending requests. For example: 1) t1 issue sync io on root group, t2 and t3 issue sync io on the same child group. num_groups_with_pending_reqs is 2 now. 2) t1 stopped, num_groups_with_pending_reqs is still 2. io from t2 and t3 still can't be handled concurrently. Fix the problem by decreasing 'num_groups_with_pending_reqs' immediately upon the deactivation of last entity of the group. Signed-off-by: Yu Kuai <yukuai3@huawei.com> --- block/bfq-iosched.c | 58 ++++++++++++++++----------------------------- block/bfq-iosched.h | 16 ++++++------- 2 files changed, 29 insertions(+), 45 deletions(-)