Back to Event Page

Kevin Kho

Community Engineer

Prefect

social media link

Kevin Kho is an Open Source Community Engineer at Prefect, an open-source workflow orchestration management system. Previously, he was a data scientist at Paylocity, where he worked on adding machine learning features to their Human Capital Management (HCM) Suite. Outside of work, he is a contributor for Fugue, an abstraction layer for Pandas, Spark, and Dask. He also organizes the Orlando Machine Learning and Data Science Meetup.

Sessions

May 26, 2022
5:40 PM UTC

Introduction to Workflow Orchestration with Prefect

Modern data applications are as complex as ever, and along with this complexity also comes an ever-growing number of ways these applications can fail. Data sources can go down intermittently. Input data may come in malformed. To guard against this, a non-trivial amount of effort is spent creating code paths to handle these failures gracefully. These include task retrying, execution timeouts, and notifications in the event of failure. Collectively, the effort spent guarding against failure is called negative engineering. Workflow orchestration frameworks are specifically designed to reduce the effort spent on negative engineering by enabling the scheduling and monitoring of workflows. Failure code paths can be added so that the application knows how to respond in the event something unexpected happens. This talk will cover basic workflow orchestration functionality and we'll familiarize ourselves with the features that let us handle these problems gracefully.

Watch On-demand