Complexity of scientific workflow logic and the complexity of the underlying architectures will quickly become a nightmare if those two concerns aren’t properly separated. Scientific workflow DSLs to the rescue! One of the newest and hottest ones out there is Cuneiform, first ever functional scientific workflow DSL. Currently Cuneiform is being rewritten from Java to Erlang, a much better suited language for the job.
Cuneiform supports HTCondor as one of its possible backends. We’ll also talk about HTCondor, oldie but very goodie job scheduling and orchestrating architecture created at UW-Madison and how we use it for LiDAR image processing at HERE.