Repository navigation
Conversation
The uniform assignor balances the total number of partitions per member but not the partitions of each topic: with homogeneous subscriptions, a member can get all the partitions of a topic while the others get none of them. uniform2, the next version of the uniform assignor, spreads every topic over its subscribers. It reads the group, decides how many partitions of every topic each member gets in three steps, keep, hand out and balance, and then places the partitions. This stack adds it one step at a time, and the assignor itself last. This patch adds the model of the group that every step reads: - `GroupModel` numbers the members in member id order and the topics in topic id order, so that every decision depends on the content of the input only. It drops the topics that do not exist, and groups the members with the same subscription into cohorts. - `CurrentAssignment` reads the owner of every partition. What the members hold beyond that is stale: partitions of topics they do not subscribe to or which no longer exist, partitions beyond the partition count, and empty sets. - `AssignmentTestUtils` builds groups for the tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
uniform2 decides how many partitions of every topic each member gets, its shares, in three steps, keep, hand out and balance, and then places the partitions. A topic with P partitions and N subscribers gives P / N base partitions to each of them, and its P % N extra partitions to as many distinct subscribers. This patch adds the shares and their first two steps: - `Shares` holds the base and extra partitions of every topic, the members getting the extra partitions, and the sizes of the members. In the keep step, a member owning more than the base partitions of a topic keeps one of its extra partitions, the members with the smallest assignments first when the owners outnumber the extra partitions. - `HandOut` gives the extra partitions that nobody kept, one at a time, to subscribers without one of the topic, among the ones with the smallest assignments: the topics with the fewest subscribers first, and round robin within a cohort. - `TestGroup` builds the model, the current assignment and the shares of a group for the tests of the steps. The balance step comes in a later patch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #23732: only the last commit belongs to this PR.
uniform2 decides how many partitions of every topic each member gets,
its shares, in three steps, keep, hand out and balance, and then places
the partitions. A topic with P partitions and N subscribers gives P / N
base partitions to each of them, and its P % N extra partitions to as
many distinct subscribers.
This patch adds the shares and their first two steps:
Sharesholds the base and extra partitions of every topic, themembers getting the extra partitions, and the sizes of the members.
In the keep step, a member owning more than the base partitions of a
topic keeps one of its extra partitions, the members with the
smallest assignments first when the owners outnumber the extra
partitions.
HandOutgives the extra partitions that nobody kept, one at a time,to subscribers without one of the topic, among the ones with the
smallest assignments: the topics with the fewest subscribers first,
and round robin within a cohort.
TestGroupbuilds the model, the current assignment and the sharesof a group for the tests of the steps.
The balance step comes in a later patch.