docs: add upstream sync process documentation

Document the three-PR workflow used to sync to a newer upstream
apache/datafusion version: bump crate deps + fix breakage, consolidate
transitive deps, then fill API and documentation gaps via
/check-upstream. Cross-reference from dev/release/README.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: timsaucer

dev/release/README.md

@@ -33,6 +33,12 @@ release branch without blocking ongoing development in the `main` branch.
 We can cherry-pick commits from the `main` branch into `branch-53` as needed and then create new patch releases
 from that branch.
 
+## Upstream Sync
+
+Between releases the `main` branch is periodically synced to a newer upstream `apache/datafusion` version. This is
+broken into a three-PR workflow (bump + fix breakage, consolidate transitive deps, fill API and documentation gaps).
+See [`upstream-sync.md`](upstream-sync.md) for the full process.
+
 ## Detailed Guide
 
 ### Pre-requisites

dev/release/upstream-sync.md

@@ -0,0 +1,134 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Upstream Sync Process
+
+This document describes how to sync `datafusion-python` to a new version of the
+upstream `apache/datafusion` Rust crates. This is a recurring task: between
+official releases the `main` branch tracks DataFusion via crates.io or GitHub
+dependencies, and we periodically bump those dependencies to pick up new
+features and bug fixes.
+
+The work is broken into **three sequential PRs** rather than landing as one
+large change. Splitting reviews along these lines keeps each PR focused, makes
+breakage easier to bisect, and lets reviewers concentrate on one concern at a
+time.
+
+## PR 1: Bump DataFusion crate dependencies and fix breakage
+
+**Goal:** update the upstream `datafusion` crate version and make the project
+build, test, and lint cleanly against it.
+
+1. Update the `datafusion` dependency in the root `Cargo.toml` (workspace
+   section and dependencies). Any downstream `datafusion-*` crates pinned in
+   `crates/core/Cargo.toml` should move to the matching version.
+2. Run `cargo update -p datafusion` (or `cargo update` for a broader refresh)
+   so `Cargo.lock` reflects the new pin.
+3. Run the standard build and test commands and address compilation errors,
+   API renames, signature changes, and behavior changes:
+   - `cargo build`
+   - `cargo test`
+   - `pytest`
+   - `pre-commit run --all-files`
+4. Fix only what's needed to restore green CI. Resist the urge to bundle
+   unrelated cleanups — those belong in their own PR.
+5. If a breaking change in upstream requires a user-facing API change in
+   `datafusion-python`, add the `api change` label and document the change
+   in the PR description so it surfaces in the changelog.
+
+**Reference PRs:** [#1311](https://github.com/apache/datafusion-python/pull/1311)
+(DF51), [#1337](https://github.com/apache/datafusion-python/pull/1337) (DF52).
+
+## PR 2: Consolidate transitive dependencies
+
+**Goal:** after the upstream bump, the dependency tree may have multiple
+versions of the same transitive crate (for example, two `arrow` versions, two
+`object_store` versions). Reconcile these so we ship a single coherent set.
+
+1. Inspect the lockfile for duplicates:
+   ```bash
+   cargo tree --duplicates
+   ```
+2. For each duplicate that matters (Arrow, `object_store`, `parquet`,
+   `tokio`, `arrow-flight`, etc.), update our direct dependency declarations
+   in `Cargo.toml` to versions compatible with what upstream DataFusion now
+   pulls in. The goal is one version of each ecosystem-critical crate.
+3. Re-run `cargo update` and re-run the full test matrix. Some duplicates are
+   benign (small leaf crates with no FFI surface) and can be left alone if
+   reconciliation would force a much larger change. Use judgment.
+4. If consolidating forces a behavioral change visible to users (for example,
+   a newer `pyarrow`-compatible Arrow version), call it out in the PR
+   description.
+
+Keeping this work separate from PR 1 means PR 1 stays a "make it compile"
+review and PR 2 stays a "tidy the dependency graph" review.
+
+## PR 3: Fill API and documentation gaps
+
+**Goal:** with the upstream version locked in, identify new APIs that landed
+upstream and decide whether to expose them, and update agent-facing
+documentation so it still matches the surface we ship.
+
+1. Run the `check-upstream` skill (`.ai/skills/check-upstream/SKILL.md`) to
+   diff the upstream Rust API against what's exposed in
+   `python/datafusion/`. The skill covers scalar/aggregate/window/table
+   functions, `DataFrame` methods, `SessionContext` methods, and FFI types.
+   Invoke it from the assistant with `/check-upstream` (optionally scoped to
+   one area, e.g. `/check-upstream scalar functions`).
+2. For each gap, decide whether to:
+   - Expose it now (small, obvious additions can land in this PR).
+   - File a tracking issue (anything non-trivial — separate PR per feature
+     keeps reviews focused).
+   - Skip it (internal-only or already covered by an existing API; record
+     the decision in the "Evaluated and not requiring exposure" sections of
+     the skill so future runs don't re-flag it).
+3. Cross-reference the user-facing skill at
+   [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md)
+   against the current public API. Look for stale function names, missing
+   newly exposed APIs, and examples that drifted from current behavior.
+   Update `SKILL.md` and the relevant RST pages under
+   `docs/source/user-guide/common-operations/` accordingly. (An
+   `audit-skill-md` skill is planned to automate this step — once it lands
+   under `.ai/skills/audit-skill-md/`, invoke it via `/audit-skill-md`.)
+4. If new aggregate or window functions were exposed in step 2, also update:
+   - `docs/source/user-guide/common-operations/aggregations.rst`
+   - `docs/source/user-guide/common-operations/windows.rst`
+
+PR 3 is the natural place to land small Pythonic-interface improvements
+discovered during the audit. Larger reshapes should still get their own PR.
+
+## Why three PRs
+
+- **Bisectable.** If a regression appears, `git bisect` lands on the
+  responsible PR (compile fix, dependency consolidation, or API addition)
+  rather than a single mega-commit.
+- **Reviewable.** Each PR has a single concern. Reviewers reading PR 1 don't
+  need to also reason about whether new APIs are well-named.
+- **Skippable.** Some upstream syncs are pure version bumps with no new APIs
+  worth exposing. PR 3 can be empty or merged as a no-op if the audit comes
+  back clean.
+
+## Related documents
+
+- [`README.md`](README.md) — the broader release process (this sync work
+  feeds into the next official release).
+- [`.ai/skills/check-upstream/SKILL.md`](../../.ai/skills/check-upstream/SKILL.md)
+  — API coverage audit.
+- [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md)
+  — user-facing agent guide kept in sync via PR 3.