i use kafka high level consumer api, , @ same time disable auto commit of offsets. tried achieve through following steps.
1) auto.commit.enable = false 2) offsets.storage = kafka 3) dual.commit.enabled = false i created offset manager, periodically creates offsetcommit request kafka , commits offset.
still have following questions
1) high level consumer api automatically fetches offset kafka storage , initializes offset? or should use simple consumer api achieve this?
2) kafka based storage offsets repicated across brokers? or maintained on 1 broker?
i created offset manager, periodically creates offsetcommit request kafka , commits offset.
you need not if using high level consumer provides methods commit offsets manually, javadoc (under manual offset control) provides examples on how that.
1) high level consumer api automatically fetches offset kafka storage , initializes offset? or should use simple consumer api achieve this?
high level consumer take care of fetching last committed offset when restart it, can resume consuming left off.
2) kafka based storage offsets repicated across brokers? or maintained on 1 broker?
kafka stores consumer offsets in internal topic named __consumer_offsets , default replication factor set 3 50 partitions. replicated across 3 brokers. can find more info on configuration in broker config, start offset or offsets.
