Integration: Add httpPoll source type to TaskSpawner for declarative polling of any JSON REST API

[kelos-bot]

🤖 Kelos Strategist Agent @gjkim42

Summary

Kelos has six source types today — four pull-based (githubIssues, githubPullRequests, jira, cron) and two push-based (githubWebhook, linearWebhook). Each pull-based source has hardcoded discovery logic for a specific platform. Adding a new polling source requires implementing the Source interface, extending the When struct, updating the spawner binary, regenerating CRDs, and releasing a new version — all before a single work item can be discovered.

Issue #687 proposes a generic push-based webhook source for event-driven integration. This proposal covers the complementary gap: a generic pull-based HTTP polling source that can discover work items from any JSON REST API without writing Go code. Many systems — internal ticketing tools, ServiceNow, Azure DevOps, Shortcut, Notion databases, custom dashboards, legacy ERPs — expose REST APIs for querying work items but do not offer outbound webhooks. An httpPoll source would let teams connect these systems to Kelos declaratively.

Problem

1. Every polling integration requires a code release

The Jira source (internal/source/jira.go) was the first non-GitHub polling source. Adding it required:

• New Jira struct in api/v1alpha1/taskspawner_types.go
• New JiraSource in internal/source/jira.go (200+ lines)
• Jira-specific deployment builder logic in internal/controller/taskspawner_deployment_builder.go:198-237
• Spawner wiring in cmd/kelos-spawner/main.go:611-622
• CRD regeneration, tests, documentation

This is the right approach for first-class integrations. But for the long tail of internal tools and less-common platforms, a declarative polling source avoids this per-integration overhead.

2. Direct Task creation is the only workaround, and it requires custom code

The docs (docs/integration.md:349) acknowledge the gap: "Run agent after deploy, trigger from Slack bot" is shown under "Direct Task creation." Teams must build their own polling scripts, deploy them separately, and manage their own deduplication and concurrency. This negates Kelos's declarative advantage.

3. The Source interface is already platform-agnostic

The Source interface (internal/source/source.go:37-39) returns []WorkItem, and the WorkItem struct (source.go:10-33) is not tied to any specific platform. A generic HTTP source can populate the same struct from arbitrary JSON responses.

Proposal

1. New HTTPPoll field in When

Add to api/v1alpha1/taskspawner_types.go:

type When struct {
    // ... existing fields ...

    // HTTPPoll discovers work items by polling an HTTP JSON API.
    // +optional
    HTTPPoll *HTTPPoll `json:"httpPoll,omitempty"`
}

2. HTTPPoll type definition

// HTTPPoll discovers work items by polling an HTTP endpoint that returns JSON.
// The endpoint must return a JSON array of objects, or a JSON object containing
// a nested array at the path specified by ItemsPath.
type HTTPPoll struct {
    // URL is the HTTP endpoint to poll. Supports Go text/template with
    // environment variables via {{env "VAR_NAME"}}.
    // +kubebuilder:validation:Required
    // +kubebuilder:validation:Pattern="^https?://.+"
    URL string `json:"url"`

    // Method is the HTTP method. Defaults to GET.
    // +kubebuilder:validation:Enum=GET;POST
    // +kubebuilder:default=GET
    // +optional
    Method string `json:"method,omitempty"`

    // Headers are static HTTP headers sent with every request.
    // +optional
    Headers map[string]string `json:"headers,omitempty"`

    // SecretRef references a Secret whose key-value pairs are injected
    // as environment variables into the spawner pod. Use these in URL
    // templates or header values via {{env "KEY"}}.
    // +optional
    SecretRef *SecretReference `json:"secretRef,omitempty"`

    // ItemsPath is a dot-notation path to the JSON array of work items
    // in the response body. When empty, the response body itself must
    // be a JSON array.
    // Examples: "data.items", "results", "issues.nodes"
    // +optional
    ItemsPath string `json:"itemsPath,omitempty"`

    // FieldMapping maps WorkItem fields to JSON object keys in each
    // discovered item. Keys are WorkItem field names (id, title, body,
    // url, labels, kind); values are dot-notation paths into the JSON
    // object. The "id" mapping is required for deduplication.
    // +kubebuilder:validation:Required
    FieldMapping HTTPFieldMapping `json:"fieldMapping"`

    // PollInterval overrides spec.pollInterval for this source.
    // +optional
    PollInterval string `json:"pollInterval,omitempty"`
}

// HTTPFieldMapping maps WorkItem fields to JSON response paths.
type HTTPFieldMapping struct {
    // ID is required. Maps to WorkItem.ID for deduplication and task naming.
    // +kubebuilder:validation:Required
    ID string `json:"id"`

    // Title maps to WorkItem.Title. Used in promptTemplate as {{.Title}}.
    // +optional
    Title string `json:"title,omitempty"`

    // Body maps to WorkItem.Body. Used in promptTemplate as {{.Body}}.
    // +optional
    Body string `json:"body,omitempty"`

    // URL maps to WorkItem.URL.
    // +optional
    URL string `json:"url,omitempty"`

    // Labels maps to a JSON array of strings or a comma-separated string.
    // +optional
    Labels string `json:"labels,omitempty"`

    // Kind maps to WorkItem.Kind. Defaults to "Item" if unmapped.
    // +optional
    Kind string `json:"kind,omitempty"`
}

3. Source implementation

Create internal/source/http_poll.go following the Jira pattern:

type HTTPPollSource struct {
    URL          string
    Method       string
    Headers      map[string]string
    ItemsPath    string
    FieldMapping HTTPFieldMapping
    Client       *http.Client
}

func (s *HTTPPollSource) Discover(ctx context.Context) ([]WorkItem, error) {
    // 1. Build and execute HTTP request
    // 2. Parse JSON response
    // 3. Navigate to ItemsPath (if set) to find the array
    // 4. For each object in the array, extract fields via FieldMapping
    // 5. Return []WorkItem
}

The dot-notation path resolver is a small utility (~30 lines) that walks nested map[string]interface{} trees — no external JSONPath library needed.

4. Example configurations

Internal ticketing system:

apiVersion: kelos.dev/v1alpha1
kind: TaskSpawner
metadata:
  name: internal-tickets
spec:
  when:
    httpPoll:
      url: "https://tickets.internal.example.com/api/v1/issues?status=agent-ready"
      headers:
        Authorization: "Bearer ${API_TOKEN}"
      secretRef:
        name: internal-api-credentials
      itemsPath: "data.issues"
      fieldMapping:
        id: "issue_id"
        title: "summary"
        body: "description"
        url: "web_url"
        labels: "tags"
        kind: "type"
      pollInterval: "5m"
  taskTemplate:
    type: claude-code
    credentials:
      type: api-key
      secretRef:
        name: anthropic-api-key
    workspaceRef:
      name: my-workspace
    promptTemplate: |
      Fix the following issue and open a PR:

      {{.Title}}

      {{.Body}}
    branch: "fix-{{.ID}}"
  maxConcurrency: 2

ServiceNow incidents:

spec:
  when:
    httpPoll:
      url: "https://mycompany.service-now.com/api/now/table/incident?sysparm_query=state=1^assignment_group=DevOps&sysparm_fields=number,short_description,description,sys_id"
      headers:
        Accept: "application/json"
      secretRef:
        name: servicenow-credentials
      itemsPath: "result"
      fieldMapping:
        id: "number"
        title: "short_description"
        body: "description"
        url: "sys_id"
      pollInterval: "10m"

Azure DevOps work items (via WIQL endpoint):

spec:
  when:
    httpPoll:
      url: "https://dev.azure.com/myorg/myproject/_apis/wit/wiql?api-version=7.0"
      method: POST
      headers:
        Content-Type: "application/json"
      secretRef:
        name: azure-devops-pat
      itemsPath: "workItems"
      fieldMapping:
        id: "id"
        title: "fields.System.Title"
        body: "fields.System.Description"
        url: "url"
      pollInterval: "5m"

5. Spawner wiring

In cmd/kelos-spawner/main.go, add after the existing Jira block:

if ts.Spec.When.HTTPPoll != nil {
    hp := ts.Spec.When.HTTPPoll
    return &source.HTTPPollSource{
        URL:          hp.URL,
        Method:       hp.Method,
        Headers:      hp.Headers,
        ItemsPath:    hp.ItemsPath,
        FieldMapping: convertFieldMapping(hp.FieldMapping),
        Client:       httpClient,
    }, nil
}

In internal/controller/taskspawner_deployment_builder.go, add environment variable injection from httpPoll.secretRef and pass --http-poll-url, --http-poll-items-path, etc. as container args (following the Jira pattern of --jira-base-url, --jira-project).

Why this matters strategically

1. Unblocks the long tail of integrations. Every "Add X source type" issue (Integration: Add Slack as TaskSpawner source for ChatOps-driven agent execution #595 Slack, Integration: Add githubSecurityAlerts source type for automated vulnerability remediation #636 Security Alerts, API: Add githubRepositories source type for organization-wide fleet agent operations #664 GitHub Repositories, Integration: Add GitLab issues and merge requests as TaskSpawner source types for multi-platform agent workflows #701 GitLab, Integration: Add Linear issues as TaskSpawner source type for developer-team-driven agent workflows #764 Linear polling) requires a code release. httpPoll lets teams integrate immediately for systems with JSON APIs, while first-class sources can still be built for the most important platforms.

2. Complements Integration: Add generic webhook source type to TaskSpawner for universal event-driven task triggering #687 (generic webhook). Together, httpPoll (pull) and the generic webhook (push) give teams a declarative escape hatch for any system, regardless of whether it supports outbound events.

3. Proven pattern. The Jira source already demonstrates that polling an arbitrary JSON API and mapping responses to WorkItem works well. httpPoll generalizes this pattern without the Jira-specific assumptions.

4. Zero external dependencies. Dot-notation path traversal is ~30 lines of Go. No JSONPath library, no JQ embedding, no external binary. The implementation is simpler than the Jira source (which handles pagination, ADF parsing, and JQL construction).

Implementation scope

Component	Files	Change
API types	api/v1alpha1/taskspawner_types.go	Add HTTPPoll, HTTPFieldMapping structs
Source	New internal/source/http_poll.go	~150 lines following jira.go pattern
Path resolver	New internal/source/json_path.go	~30 lines for dot-notation traversal
Deployment builder	taskspawner_deployment_builder.go	Add secret injection and spawner args
Spawner main	cmd/kelos-spawner/main.go	Wire up HTTPPollSource
CLI printer	internal/cli/printer.go	Add "HTTP Poll" to SOURCE column
CRD regen	make update	Regenerate manifests
Tests	New *_test.go files	Unit tests with httptest servers

Design decisions

Why dot-notation instead of JSONPath? JSONPath has multiple incompatible specifications (RFC 9535, Goessner, Jayway). Dot-notation is unambiguous, covers 95% of real APIs (nested object access), and requires no external dependency. If advanced extraction is needed later, it can be added as a backward-compatible extension.

Why not just use #687 (generic webhook)? Webhooks are push-based — they require the external system to support outbound HTTP events and require configuring the external system to point at Kelos. Many internal tools, legacy systems, and SaaS platforms have read APIs but no webhook support. Polling also provides reliable discovery (no missed events due to network issues) and allows filtering at discovery time.

Why require id in field mapping? The spawner uses the work item ID for task naming and deduplication (checking if a Task already exists for a given ID). Without a stable ID, the same work item would spawn duplicate tasks on every poll cycle.

References

• Source interface: internal/source/source.go:37-39
• Jira source (closest precedent): internal/source/jira.go
• Spawner source wiring: cmd/kelos-spawner/main.go:533-622
• Deployment builder (env/arg injection): internal/controller/taskspawner_deployment_builder.go:198-237
• Generic webhook proposal (push-based complement): Integration: Add generic webhook source type to TaskSpawner for universal event-driven task triggering #687
• Existing platform-specific source proposals: Integration: Add Slack as TaskSpawner source for ChatOps-driven agent execution #595 (Slack), Integration: Add githubSecurityAlerts source type for automated vulnerability remediation #636 (Security Alerts), API: Add githubRepositories source type for organization-wide fleet agent operations #664 (Repositories), Integration: Add GitLab issues and merge requests as TaskSpawner source types for multi-platform agent workflows #701 (GitLab), Integration: Add Linear issues as TaskSpawner source type for developer-team-driven agent workflows #764 (Linear), Integration: Add prometheusAlerts source type to TaskSpawner for alert-driven autonomous remediation #775 (Prometheus)

/kind feature

[kelos-bot]

🤖 Kelos Task Status

Task kelos-triage-894 has succeeded. ✅

[kelos-bot]

🤖 Kelos Agent @gjkim42

Kelos Triage Report

Kind: kind/feature
Priority: priority/important-longterm — Valuable extensibility feature but not blocking current adoption; workaround exists via Direct Task creation
Actor: needs human — Introduces new CRD types (HTTPPoll, HTTPFieldMapping) requiring API design decisions and review of the proposed field mapping approach

Status: STILL VALID

Evidence:

• No merged PRs or commits reference issue Integration: Add httpPoll source type to TaskSpawner for declarative polling of any JSON REST API #894 or httpPoll-related keywords
• The codebase confirms exactly 6 source types today: githubIssues, githubPullRequests, jira, cron (polling) and githubWebhook, linearWebhook (push). No generic HTTP polling source exists
• The When struct in api/v1alpha1/taskspawner_types.go has no HTTPPoll field
• No files matching http_poll or HTTPPoll exist under internal/source/
• The Source interface (internal/source/source.go) and WorkItem struct are platform-agnostic as the issue states, confirming feasibility
• The Jira source (internal/source/jira.go) validates the pattern — it already polls an arbitrary JSON REST API and maps responses to WorkItem
• Issue Integration: Add generic webhook source type to TaskSpawner for universal event-driven task triggering #687 (generic webhook, push-based complement) is open with needs-actor, also awaiting design decisions

Duplicates found: None found. #687 is the push-based complement (explicitly referenced), not a duplicate. Platform-specific source requests (#595 Slack, #636 Security Alerts, #603 Check Runs, #697 Kubernetes Resources, #768 GitHub Discussions) are distinct — httpPoll could subsume some of them for systems with REST APIs but does not replace purpose-built sources.

Recommendation: KEEP OPEN — Well-scoped proposal with a clear implementation plan. Requires maintainer sign-off on the new API surface (CRD types, field mapping design, dot-notation vs JSONPath decision) before implementation can proceed.