Skip to content

feat(go): implement a binary reader#2986

Open
chengxilo wants to merge 6 commits intoapache:masterfrom
chengxilo:refactor/go-binary-reader
Open

feat(go): implement a binary reader#2986
chengxilo wants to merge 6 commits intoapache:masterfrom
chengxilo:refactor/go-binary-reader

Conversation

@chengxilo
Copy link
Contributor

Which issue does this PR close?

N/A

Rationale

Iggy, when read/write binary, use little endian. When there is a string, it has a prefix provide its length, and the actually string body. So instead of using the binary.LittleEndian.Uint32(payload[processIDPos : processIDPos+4]) and use position ++ , why not just wrap everything in a reader to make it simple and clear.

As the Stats deserialize function need to be updated, I think this change would be helpful. #2964 (comment)

Here is a example that shows the difference:
Before:

func (stats *TcpStats) Deserialize(payload []byte) error {
stats.ProcessId = binary.LittleEndian.Uint32(payload[processIDPos : processIDPos+4])
stats.CpuUsage = math.Float32frombits(binary.LittleEndian.Uint32(payload[cpuUsagePos : cpuUsagePos+4]))
stats.TotalCpuUsage = math.Float32frombits(binary.LittleEndian.Uint32(payload[totalCpuUsagePos : totalCpuUsagePos+4]))
stats.MemoryUsage = binary.LittleEndian.Uint64(payload[memoryUsagePos : memoryUsagePos+8])
stats.TotalMemory = binary.LittleEndian.Uint64(payload[totalMemoryPos : totalMemoryPos+8])
stats.AvailableMemory = binary.LittleEndian.Uint64(payload[availableMemoryPos : availableMemoryPos+8])
stats.RunTime = binary.LittleEndian.Uint64(payload[runTimePos : runTimePos+8])
stats.StartTime = binary.LittleEndian.Uint64(payload[startTimePos : startTimePos+8])
stats.ReadBytes = binary.LittleEndian.Uint64(payload[readBytesPos : readBytesPos+8])
stats.WrittenBytes = binary.LittleEndian.Uint64(payload[writtenBytesPos : writtenBytesPos+8])
stats.MessagesSizeBytes = binary.LittleEndian.Uint64(payload[messagesSizeBytesPos : messagesSizeBytesPos+8])
stats.StreamsCount = binary.LittleEndian.Uint32(payload[streamsCountPos : streamsCountPos+4])
stats.TopicsCount = binary.LittleEndian.Uint32(payload[topicsCountPos : topicsCountPos+4])
stats.PartitionsCount = binary.LittleEndian.Uint32(payload[partitionsCountPos : partitionsCountPos+4])
stats.SegmentsCount = binary.LittleEndian.Uint32(payload[segmentsCountPos : segmentsCountPos+4])
stats.MessagesCount = binary.LittleEndian.Uint64(payload[messagesCountPos : messagesCountPos+8])
stats.ClientsCount = binary.LittleEndian.Uint32(payload[clientsCountPos : clientsCountPos+4])
stats.ConsumerGroupsCount = binary.LittleEndian.Uint32(payload[consumerGroupsCountPos : consumerGroupsCountPos+4])
position := consumerGroupsCountPos + 4
hostnameLength := int(binary.LittleEndian.Uint32(payload[position : position+4]))
stats.Hostname = string(payload[position+4 : position+4+hostnameLength])
position += 4 + hostnameLength
osNameLength := int(binary.LittleEndian.Uint32(payload[position : position+4]))
stats.OsName = string(payload[position+4 : position+4+osNameLength])
position += 4 + osNameLength
osVersionLength := int(binary.LittleEndian.Uint32(payload[position : position+4]))
stats.OsVersion = string(payload[position+4 : position+4+osVersionLength])
position += 4 + osVersionLength
kernelVersionLength := int(binary.LittleEndian.Uint32(payload[position : position+4]))
stats.KernelVersion = string(payload[position+4 : position+4+kernelVersionLength])
return nil
}

After

func (s *Stats) UnmarshalBinary(payload []byte) error {
	r := newReader(payload)
	s.ProcessId = r.u32()
	s.CpuUsage = r.f32()
	s.TotalCpuUsage = r.f32()
	s.MemoryUsage = r.u64()
	s.TotalMemory = r.u64()
	s.AvailableMemory = r.u64()
	s.RunTime = r.u64()
	s.StartTime = r.u64()
	s.ReadBytes = r.u64()
	s.WrittenBytes = r.u64()
	s.MessagesSizeBytes = r.u64()
	s.StreamsCount = r.u32()
	s.TopicsCount = r.u32()
	s.PartitionsCount = r.u32()
	s.SegmentsCount = r.u32()
	s.MessagesCount = r.u64()
	s.ClientsCount = r.u32()
	s.ConsumerGroupsCount = r.u32()
	s.Hostname = r.u32LenStr()
	s.OsName = r.u32LenStr()
	s.OsVersion = r.u32LenStr()
	s.KernelVersion = r.u32LenStr()
	return r.Err()
}

What changed?

A reader for byte slice is created.

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

  1. smart Claude Sonnet 4.6 and my rotten brain
  2. entire implementation, manually modified some part to make sure the error output readablity.
  3. Reviewed the tests and code, actually modified some logic of reading binary and works well.
  4. yes

@codecov
Copy link

codecov bot commented Mar 20, 2026

Codecov Report

❌ Patch coverage is 82.60870% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.00%. Comparing base (f0e8578) to head (bcf7a53).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
foreign/go/internal/codec/reader.go 82.60% 8 Missing and 8 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2986      +/-   ##
============================================
+ Coverage     71.81%   72.00%   +0.18%     
+ Complexity      930      925       -5     
============================================
  Files          1116     1114       -2     
  Lines         92616    92426     -190     
  Branches      70139    69885     -254     
============================================
+ Hits          66512    66547      +35     
+ Misses        23543    23314     -229     
- Partials       2561     2565       +4     
Flag Coverage Δ
go 37.28% <82.60%> (+0.90%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
foreign/go/internal/codec/reader.go 82.60% <82.60%> (ø)

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@chengxilo chengxilo requested a review from ryankert01 March 20, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants