Evolving Workflow Orchestration // Alex Milowski // #291

MLOps.community

Alex Milowski is a researcher, developer, entrepreneur, mathematician, and computer scientist.

Evolving Workflow Orchestration // MLOps Podcast #291 with Alex Milowski, Entrepreneur and Computer Scientist.

// Abstract

There seems to be a shift from workflow languages to code, mostly annotation Python - happening and getting us. It is a symptom of how complex workflow orchestration has gotten. Is it a dominant trend or will we cycle back to “DAG specifications”? At Stitchfix, we had our own DSL that “compiled” into airflow DAGs, and at MicroByre, we used an external workflow language. Both had a batch task executor on K8S, but at MicroByre, we had human and robot in the loop workflows.

// Bio

Dr. Milowski is a serial entrepreneur and computer scientist with experience in a variety of data and machine learning technologies. He holds a PhD in Informatics (Computer Science) from the University of Edinburgh, where he researched large-scale computation over scientific data. Over the years, he's spent many years working on various aspects of workflow orchestration in industry, standardization, and research.

// Related Links

Website: https://www.milowski.com/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

// MLOps Swag/Merch

⁠https://shop.mlops.community/⁠

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Alex on LinkedIn: https://www.linkedin.com/in/alexmilowski/

Timestamps:

[00:00] Alex's preferred coffee

[00:20] Takeaways

[01:27] Workflow Systems Overview

[05:54] Everything as a DAG

[09:25] Workflows, Layering, and Ownership

[12:46] Workflow Awareness for Success

[17:31] Process Engineering Challenges

[21:20] ML Workflow Trends

[27:57] Data Engineering and ML Tools

[33:18] Infrastructure as Code Pros and Cons

[36:20] DSLs in Organizations

[43:04] Agents vs DAGs

[47:48] Agent Workflow Selection

[50:31] RPA Business Automation Tools

[55:00] LLM Integration in BPA

[1:01:14] Diversity in Decision Workflows

[1:03:09] Shadow AI Governance Risks

[1:08:34] Centralized ML Workflow Standardization

[1:12:50] Wrap up