For running chains of multi-hour jobs with dependencies I've always used Jenkins but always been unsatisfied with it. I've also heard of Luigi but haven't given it a shot yet. Is there anything else similar I should look at? I've had trouble finding a single name for programs like this to search with.
Luigi: https://github.com/spotify/luigi
Example jobs:
- download a bunch of raw data from different sources
- clean and transform raw data into a single format
- push a lot of data into a DB
- routine ML model updates
- generate reports