Remove Kafka trigger feature entirely

[stuartc]

User story

As a maintainer, I want to remove the Kafka trigger feature from Lightning so that we reduce codebase complexity, dependency surface area, and maintenance burden for a feature that remains in beta with limited adoption.

Details

The Kafka trigger feature was introduced in July 2024 (PR #2270) as an experimental trigger type alongside Webhook and Cron. It remains marked as BETA and is gated behind the KAFKA_TRIGGERS_ENABLED feature flag (default: false). Removing it means deleting all Kafka-specific code, database artifacts, dependencies, configuration, and UI components.

Pre-work: production audit (must be done before implementation begins)

The scope of the migration depends on whether any production data uses Kafka. Run the following queries against production to determine the blast radius:

-- Count workflows with Kafka triggers
SELECT COUNT(DISTINCT w.id)
FROM triggers t JOIN workflows w ON w.id = t.workflow_id
WHERE t.type = 'kafka';

-- List affected projects, workflows, and trigger status
SELECT p.name AS project_name, w.name AS workflow_name, t.id AS trigger_id, t.enabled
FROM triggers t
JOIN workflows w ON w.id = t.workflow_id
JOIN projects p ON p.id = w.project_id
WHERE t.type = 'kafka'
ORDER BY p.name, w.name;

-- Count Kafka dataclips
SELECT COUNT(*) FROM dataclips WHERE type = 'kafka';

-- Count Kafka dedup tracking records
SELECT COUNT(*) FROM trigger_kafka_message_records;

-- Count work orders from Kafka triggers
SELECT COUNT(*) FROM work_orders wo
JOIN triggers t ON t.id = wo.trigger_id WHERE t.type = 'kafka';

-- Snapshots containing Kafka trigger data
SELECT COUNT(*) FROM snapshots s WHERE s.triggers::text LIKE '%kafka%';

-- Users who own projects with Kafka triggers (for outreach)
SELECT DISTINCT u.email, p.name AS project_name
FROM triggers t
JOIN workflows w ON w.id = t.workflow_id
JOIN projects p ON p.id = w.project_id
JOIN project_users pu ON pu.project_id = p.id
JOIN users u ON u.id = pu.user_id
WHERE t.type = 'kafka';

Based on the results, decide:

1. If no Kafka triggers/dataclips exist: Straightforward removal. Migration just drops the column and table.
2. If Kafka triggers exist but are disabled/unused: Delete them in the migration. Coordinate with affected users if any.
3. If active Kafka triggers exist: Coordinate with affected users before deploying. They need to migrate to Webhook or Cron triggers first.
4. Kafka dataclips: If none exist, no action needed. If they exist, they can be deleted in the migration (they're input records, not run results). If the volume is large, consider a batched async cleanup.
5. Historical snapshots: Snapshots embed trigger data immutably. Old snapshots with type: "kafka" will remain in the database. The code must not crash when encountering them - ensure deserialization gracefully handles unknown trigger types rather than raising on an invalid enum value.

Breaking changes:

• Any workflows using Kafka triggers will stop functioning
• The kafka trigger type will no longer be accepted by the provisioning API (will fail validation naturally)
• The trigger_kafka_message_records table will be dropped
• The kafka_configuration column on triggers will be dropped
• 8 KAFKA_* environment variables will no longer be recognized

Implementation notes

Dependencies to remove from mix.exs:

• {:broadway_kafka, "~> 0.4.2"} (also removes transitive deps: brod, kafka_protocol)

Database migration required:

• Drop the trigger_kafka_message_records table
• Remove the kafka_configuration column from triggers
• Delete any triggers with type = 'kafka' (scope determined by production audit above)
• Delete any dataclips with type = 'kafka' (scope determined by production audit above)
• Remove :kafka from the dataclip type enum in the schema (existing 'kafka' strings in the DB column won't cause issues once the Ecto enum no longer references them, but the migration should clean them up)
• Ensure snapshot deserialization handles unknown trigger types gracefully (old snapshots may contain type: "kafka")

Backend modules to delete entirely:

• lib/lightning/kafka_triggers.ex
• lib/lightning/kafka_triggers/ (entire directory: supervisor, pipeline, event_listener, message_handling, message_recovery, duplicate_tracking_cleanup_worker, trigger_kafka_message_record, pipeline_supervisor)
• lib/lightning/workflows/triggers/kafka_configuration.ex
• lib/lightning_web/live/job_live/kafka_setup_component.ex

Backend files requiring modification:

• lib/lightning/application.ex - Remove KafkaTriggers.Supervisor from supervision tree
• lib/lightning/workflows/trigger.ex - Remove :kafka from type enum, remove kafka_configuration embed and validation
• lib/lightning/workflows/snapshot.ex - Remove :kafka from trigger type enum, remove kafka_configuration field
• lib/lightning/workflows/triggers/events.ex - Remove KafkaTriggerUpdated, KafkaTriggerNotificationSent events and related functions
• lib/lightning/workflows/params_comparator.ex - Remove kafka_configuration from tracked fields
• lib/lightning/workflows.ex - Remove publish_kafka_trigger_events/1, notify_of_affected_kafka_triggers/1
• lib/lightning/invocation/dataclip.ex - Remove :kafka from source types and validation
• lib/lightning/invocation/query.ex - Remove kafka from SQL CASE expressions
• lib/lightning/runs.ex - Remove kafka from SQL CASE expressions
• lib/lightning/projects/provisioner.ex - Remove KafkaConfiguration alias, kafka_config_changeset/2, embed cast
• lib/lightning/projects/sandboxes.ex - Remove kafka_configuration copying
• lib/lightning/projects/merge_projects.ex - Remove kafka_configuration from merge fields (2 locations)
• lib/lightning/config.ex - Remove 8 kafka_* accessor functions
• lib/lightning/config/bootstrap.ex - Remove Kafka env var parsing (lines ~674-715)
• lib/lightning/accounts/user_notifier.ex - Remove deliver_kafka_trigger_failure_notification/3
• lib/lightning/export_utils.ex - Remove Kafka trigger field serialization
• lib/lightning/setup_utils.ex - Remove TriggerKafkaMessageRecord from truncation list
• lib/lightning/collaboration/workflow_serializer.ex - Remove Kafka config serialization/normalization
• lib/lightning_web/live/workflow_live/components.ex - Remove "Kafka Consumer" option, kafka_trigger_title/1, beta badge
• lib/lightning_web/live/workflow_live/edit.ex - Remove kafka trigger title rendering
• lib/lightning_web/live/components/icon.ex - Remove :kafka icon/color mappings
• lib/lightning_web/channels/workflow_channel.ex - Remove kafka_triggers_enabled from config
• lib/lightning_web/controllers/api/provisioning_json.ex - Remove kafka_configuration from JSON output
• lib/lightning_web/controllers/api/workflows_controller.ex - Update module doc
• lib/lightning_web/controllers/dataclip_controller.ex - Remove :kafka from type comment

Frontend files requiring modification:

• assets/js/collaborative-editor/types/trigger.ts - Remove kafkaConfigSchema, kafkaTriggerSchema, Kafka defaults
• assets/js/collaborative-editor/types/sessionContext.ts - Remove kafka_triggers_enabled from AppConfigSchema
• assets/js/collaborative-editor/components/inspector/TriggerForm.tsx - Remove entire Kafka config form section
• assets/js/collaborative-editor/components/inspector/TriggerInspector.tsx - Remove 'Kafka Trigger' title case
• assets/js/collaborative-editor/stores/createWorkflowStore.ts - Remove nested kafka_configuration error handling
• assets/js/collaborative-editor/hooks/useUnsavedChanges.ts - Remove kafka_configuration handling
• assets/js/collaborative-editor/adapters/YAMLStateToYDoc.ts - Remove Kafka trigger transformation
• assets/js/workflow-diagram/nodes/Trigger.tsx - Remove kafka case from getTriggerMeta()
• assets/js/workflow-diagram/components/trigger-icons.tsx - Remove kafkaIcon SVG
• assets/js/workflow-diagram/components/MiniMapNode.tsx - Remove 'kafka' from type union
• assets/js/workflow-diagram/types.ts - Remove Lightning.KafkaTrigger interface
• assets/js/workflow-diagram/util/from-workflow.tsx - Remove kafka has_auth_method handling
• assets/js/yaml/types.ts - Remove StateKafkaTrigger, SpecKafkaTrigger
• assets/js/yaml/util.ts - Remove Kafka-specific position exclusion and TODO
• assets/js/manual-run-panel/types.ts - Remove 'kafka' from DataclipTypes and DataclipTypeNames

Test files to delete entirely:

• test/lightning/kafka_triggers_test.exs
• test/lightning/kafka_triggers/ (entire directory)
• test/lightning/workflows/triggers/kafka_configuration_test.exs

Test files requiring modification (remove Kafka-specific tests/references):

• test/lightning/workflows_test.exs
• test/lightning/workflows/trigger_test.exs
• test/lightning/workflows/triggers/events_test.exs
• test/lightning/runs_test.exs
• test/lightning/work_orders_test.exs
• test/lightning/invocation_test.exs
• test/lightning/invocation/dataclip_test.exs
• test/lightning/invocation/query_test.exs
• test/lightning/projects_test.exs
• test/lightning/projects/provisioner_test.exs
• test/lightning/projects/merge_projects_test.exs
• test/lightning/sandboxes_test.exs
• test/lightning/collaboration/session_test.exs
• test/lightning/collaboration/workflow_serializer_test.exs
• test/lightning/config/bootstrap_test.exs
• test/lightning/accounts/user_notifier_test.exs
• test/lightning/workflow_versions_test.exs
• test/lightning_web/controllers/api/provisioning_controller_test.exs
• test/lightning_web/live/workflow_live/trigger_test.exs
• test/lightning_web/channels/workflow_channel_test.exs
• test/support/factories.ex - Remove triggers_kafka_configuration_factory and trigger_kafka_message_record_factory
• test/support/merge_projects_helpers.ex

Frontend test files requiring modification:

• assets/test/workflow-diagram/nodes/Trigger.test.tsx
• assets/test/workflow-diagram/util/from-workflow.test.tsx
• assets/test/collaborative-editor/adapters/YAMLStateToYDoc.test.ts
• assets/test/collaborative-editor/types/sessionContext.test.ts
• assets/test/collaborative-editor/components/inspector/TriggerInspector.test.tsx
• assets/test/collaborative-editor/components/inspector/EdgeInspector.test.tsx
• assets/test/collaborative-editor/components/inspector/WorkflowSettings.test.tsx
• assets/test/collaborative-editor/components/Header.test.tsx
• assets/test/collaborative-editor/components/ManualRunPanel.keyboard.test.tsx
• assets/test/collaborative-editor/hooks/useJobDeleteValidation.test.tsx
• assets/test/collaborative-editor/hooks/useSessionContext.test.tsx
• assets/test/collaborative-editor/__helpers__/sessionContextFactory.ts

Infrastructure to delete:

• kafka_testing/ (entire directory: 4 docker-compose files + KAFKA_ARCHITECTURE.md)

Documentation to update:

• DEPLOYMENT.md - Remove Kafka deployment section (lines ~238-334)
• .env.example - Remove 8 KAFKA_* environment variables

Environment variables to remove:

• KAFKA_TRIGGERS_ENABLED
• KAFKA_NUMBER_OF_MESSAGES_PER_SECOND
• KAFKA_NUMBER_OF_PROCESSORS
• KAFKA_NUMBER_OF_CONSUMERS
• KAFKA_DUPLICATE_TRACKING_RETENTION_SECONDS
• KAFKA_NOTIFICATION_EMBARGO_SECONDS
• KAFKA_ALTERNATE_STORAGE_ENABLED
• KAFKA_ALTERNATE_STORAGE_FILE_PATH

Release notes

Kafka triggers have been removed from Lightning. Any workflows using Kafka triggers will need to be reconfigured to use an alternative trigger type (Webhook or Cron). The KAFKA_TRIGGERS_ENABLED environment variable and all other KAFKA_* configuration variables are no longer recognized. The kafka trigger type is no longer accepted by the provisioning API.

User acceptance criteria

• Production audit queries have been run and results documented
• Affected users (if any) have been notified and have migrated off Kafka triggers
• No Kafka-related code, types, or references remain in the codebase
• The broadway_kafka dependency (and its transitive deps) is removed from mix.exs and mix.lock
• A database migration drops trigger_kafka_message_records and the kafka_configuration column from triggers
• The migration deletes any existing Kafka triggers and Kafka dataclips
• The :kafka trigger type and :kafka dataclip type are removed from all enums
• Historical snapshots containing Kafka trigger data do not cause errors when viewed
• The provisioning API rejects kafka as a trigger type (fails validation naturally)
• All existing tests pass with Kafka code removed
• mix verify passes cleanly
• cd assets && npm test passes cleanly
• kafka_testing/ directory is removed
• DEPLOYMENT.md and .env.example no longer reference Kafka
• The collaborative editor (Y.Doc serializer) no longer handles kafka_configuration