Skip to main content
x

Adding new workers and resource management to new dataflow engine written in Python: Pydra

Tags:
Status:
Active, Closed for joining
Contributor/Mentors

Contributor: Aditya Agarwal

About

This project aims to extend Pydra's worker classes to handle a wider range of systems by utilizing PSI/J, which offers a unified API for various schedulers. The objective of this project is to create a Pydra worker class capable of submitting and monitor jobs on different HPC systems, such as Theta, using PSI/J. Furthermore, the contributor intends to use Bokeh to construct an interactive web-based dashboard that will monitor the jobs being submitted and executed by the Pydra worker. The expected outcome of this project is a Pydra worker that can use PSI/J to submit and track jobs on various HPC systems, an updated Dask worker, as well as a jobs monitoring dashboard. The proposed methodology involves designing and implementing the Pydra worker class, evaluating its functionality and performance on various systems, comparing it to existing Pydra workers, and constructing a Pydra dashboard using Bokeh to monitor the jobs. Finally, the contributor will document the usage and installation of the Pydra worker with PSI/J, as well as the Pydra dashboard, and provide users with examples and tutorials.

Completed Deliverables
2023
  • Interactive web-based dashboard for Pydra workers
  • Pydra integration with PSI/J for submission and tracking of jobs
  • Bokeh implementation in Pydra dashboard
  • Examples and tutorials for users
2023