feat(table): add support for merge-on-read delete

[alexandre-normand]

This adds support for merge-on-read deletes. It offers an alternative to the copy-on-write to generate position delete files instead of rewriting existing data files.

I'm not very confident in the elegance of my solution as I'm still new to the internals of iceberg-go but the high-level is:

• Reuse the classification code from the existing delete implementation to get the list of files of dropped files vs files with partial deletes
• Reuse the arrow scanning facilities to filter records from the data files with partial deletes and emit position delete records with file path and position.
  • This is done by reusing the pipeline code and function and making the first stage in the pipeline one to enrich the RecordBatch with the file Path and position before the original position is lost due to filtering.
  • After filtering, the RecordBatch is projected to the position delete schema (i.e. the original schema fields are dropped)
• Once we have filtered PositionDelete records that need to be emitted, we reuse the record to file writing to generate position delete files.

Testing

Integration tests were added to exercise the partitioned and unpartitioned paths and the data is such that it's meant to actually produce a position delete file rather than just go through the quick path that drops an entire file because all records are gone.

Indirect fixes

While working on this change and adding the testing for the partitioned table deletions, I realized that the manifest evaluation when the filter affected a field that was part of a partition spec was not built correctly. It needed to use similar code as what's done during scanning to build projections and build a manifest evaluator per partition id. This is fixed in this PR but this technically also applies to copy-on-write and overwrite paths so the fix goes beyond the scope of the merge-on-read.

Fixes #487.

[laskoviymishka]

Overall, this looks correct (I mostly compared it with Iceberg Java), but I think a couple of things need attention:

• panic/recover is used in normal write-path error handling;
• position-delete fanout needs additional tests;
• missing focused local unit tests for invariant-heavy code (enrichRecordsWithPosDeleteFields, position-delete fanout).

Regarding the API change: the ToDataFile change is internal (package scope), but the ManifestWriter.ToManifestFile signature change is public and should be explicitly called out in the compatibility/release notes.

I would advocate for not changing this API and instead adding a new one with the extra argument, but this decision, in my opinion, is not critical.

[zeroshade]

this is looking good to me. There's a conflict to resolve in README.md, and I'll wait for @laskoviymishka to give feedback before merging.

[alexandre-normand]

Here's the updated status of the TODO list:

Blockers (must fix before merge)

• JSON syntax error in TestDeleteMergeOnReadPartitioned
  Missing comma in array.TableFromJSON.
  The partitioned integration test currently panics and never runs.
• summary() does not account for positionDeleteFiles (snapshot_producers.go)
  appendPositionDeleteFile() was added, but summary() was not updated.
  MoR delete snapshots currently report zero delete metrics.

Follow-ups (can be addressed separately)

• Partition routing via path.Split instead of (spec, partition) pair
  (writePositionDeletesForFiles, processBatch)
• Position delete files always written under CurrentSpec()
  (positionDeleteRecordsToDataFiles)
  At minimum, should be documented as safe only for single-spec tables.
• writeUUID not threaded from commitUUID
  (writePositionDeletesForFiles)
• dropFile=true sentinel batch uses data schema instead of PositionalDeleteArrowSchema
  (producePosDeletesFromTask)
  Harmless today, but a latent schema mismatch.
• No sort by (file_path, pos)
  Performance gap vs Java’s SortedPositionDeleteWriter.
• referenced_data_file not populated in delete manifest entries
  Optional spec optimization.

Thanks for the in-depth review, @laskoviymishka .

• For the lack of sorting, this would be a bigger effort because arrow-go doesn't yet have support for sorting (related issue which, I think, would be the right place to put that code in.

• For the referenced_data_file, I'm going to call this out of scope for this one. I'm a little out of breath from this work and the deep in the trenches hunting of the long tail of missing little pieces and I think this is probably a good milestone to call this one done?

[laskoviymishka]

Reviewed the latest changes — the core logic looks correct:

• MoR positional delete pipeline ordering is sound (positions assigned before filtering, remain valid through projection)
• Manifest evaluator fix for partition spec evolution is a genuine correctness improvement, applies to CoW paths too
• pos type fix (Int32 → Int64) aligns with the Iceberg spec

Two minor issues noted but will address in follow-up PRs to keep this moving:

1. Arrow array refcount leak in enrichRecordsWithPosDeleteFields — NewArray() results are never Release()d after NewRecordBatch retains them
2. Goroutine leak from iter.Pull in the partitioned path of positionDeleteRecordsToDataFiles — stopCount not called in the partitioned branch (same pattern from #718)

Neither blocks correctness for typical use. LGTM to merge.

[zeroshade]

Thanks to both of you!

[alexandre-normand]

Thanks for the review, @laskoviymishka and @zeroshade !

[ferhatelmas]

I think there is a regression here to fix the leak #766

[ferhatelmas]

I think order matters here: #767