Best practice for scheduling multiple workflows (Docker Compose & Kubernetes)?

[HaruspexSan]

Hello Hop community,

I am building a data platform with Apache Hop (v2.17) and looking for the recommended way to schedule multiple .hwf workflows as part of my Master Thesis. Currently, I am using Hop-Web to create the workflows in the UI and I want to use something more lightweight to execute the workflows. My idea was splitting the workflows into topics, having one to parse and store data for each data domain. This includes saving prod data in a dedicated bi database.

I know from previous Q&A (#6105) that Hop Server isn't a time-based scheduler, and the standard approach is triggering the hop-run.sh CLI via a native scheduler (like cron) to run short-lived execution containers.

My environments are:

• Local: Docker Compose (using apache/hop-web as the IDE)

• Production: Kubernetes via Helm

Questions:

• Hop Execution: Instead of triggering 10 different containers for 10 workflows, is the standard Hop practice to create a single "Master Workflow" that calls the other .hwf files, and just schedule that one Master container?

• Local Scheduling: What is the recommended, lightweight approach for handling these schedules locally in Docker Compose without introducing a heavy orchestrator? Especially, if i want to run the workflows scheduled with cron or something similar.

• K8s Scaling (10-50+ workflows): When moving to Kubernetes, what is the best practice that works cleanly with the provided Apache Hop Helm charts? Is it standard to generate dozens of K8s CronJob manifests, or should I be looking at something else?

Finally, thank you for taking your time and helping me out.

Cheers,
Max

[dmainou]

Hi,
you can trigger n number of short lived containers
or

• start a container running hop server.
• execute jobs on the server as required
• kill the server
  or use and metal to run individual jobs using the crontabs

or execute in dataflow or similar

sky is the limit