Skip to content

EOSC-LOFAR/prefactor-cwl

Repository files navigation

prefactor-cwl

CWL version of prefactor

https://github.com/lofar-astron/prefactor/

Build Status

Note that this project is still very much in alpha stage and very much in flux. It is a prototype implementation and is not intended for end user audience. The goal is to demonstrate various technologies and how they could assist simplifying construction and deployment of big scale data reduction pipelines.

requirements

optional:

usage

To run the pipeline with the example dataset just run:

$ make run

To Run using a udocker (user space docker) run:

$ make run-udocker

To try out toil as a schedular run:

$ make toil

HPC cloud

We've made a set of ansbile scripts to automate the deployment of a cluster that can run CWL pipelines: https://github.com/EOSC-LOFAR/ansible

on Cartesius

If you are on cartesius, you first need to activate a python environment:

$ module load python

Since all jobs are quite small, it is better to use the staging partition. You can configure toil to use the staging partition with:

$ export TOIL_SLURM_ARGS="-t 0:30:00 -p staging"
$ make slurm

steps

graph

the pipeline consists of these steps:

step name status file comment
createmap_cal skip we don't use map files
ndppp_prep_cal implemented steps/ndppp_prep_cal.cwl
combine_data_cal_map skip We don't use map files
sky_cal implemented steps/sky_cal.cwl
sky_cal_path skip we don't use map files
calib_cal implemented steps/steps/calib_cal.cwl
h5_imp_cal_map skip we don't use map files
h5imp_cal implemented steps/losoto_importer.cwl
mk_inspect_dir skip we don't need to create dirs
copy_h5imp_cal skip this is managed by CWL
fitclock implemented steps/fitclock.cwl
ampl implemented steps/ampl.cwl disabled for test since not enough subbands
plots implemented steps/plots.cwl disabled for test since not enough subbands
phase implemented steps/phase.cwl
plot_cal_phases implemented steps/plot_cal_phases.cwl
createmap_cal_pngs skip we don't use map files
copy_cal_pngs skip this is managed by CWL
createmap_cal_txts skip we don't use map files
copy_cal_txts skip this is managed by CWL
mk_cal_values_dir skip this is managed by CWL
createmap_cal_npys skip we don't use map files
copy_cal_npys skip this is managed by CWL