Accelerate Migration Of Your Data Warehouse with Datafold's AI Powered Migration Agent

Data Engineering Podcast

Summary
Gleb Mezhanskiy, CEO and co-founder of DataFold, joins Tobias Macey to discuss the challenges and innovations in data migrations. Gleb shares his experiences building and scaling data platforms at companies like Autodesk and Lyft, and how these experiences inspired the creation of DataFold to address data quality issues across teams. He outlines the complexities of data migrations, including common pitfalls such as technical debt and the importance of achieving parity between old and new systems. Gleb also discusses DataFold's innovative use of AI and large language models (LLMs) to automate translation and reconciliation processes in data migrations, reducing time and effort required for migrations.
Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Imagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!
• Your host is Tobias Macey and today I'm welcoming back Gleb Mezhanskiy to talk about Datafold's experience bringing AI to bear on the problem of migrating your data stack

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you describe what the Data Migration Agent is and the story behind it?
  • What is the core problem that you are targeting with the agent?
• What are the biggest time sinks in the process of database and tooling migration that teams run into?
• Can you describe the architecture of your agent?
  • What was your selection and evaluation process for the LLM that you are using?
• What were some of the main unknowns that you had to discover going into the project?
  • What are some of the evolutions in the ecosystem that occurred either during the development process or since your initial launch that have caused you to second-guess elements of the design?
• In terms of SQL translation there are libraries such as SQLGlot and the work being done with SDF that aim to address that through AST parsing and subsequent dialect generation. What are the ways that approach is insufficient in the context of a platform migration?
• How does the approach you are taking with the combination of data-diffing and automated translation help build confidence in the migration target?
• What are the most interesting, innovative, or unexpected ways that you have seen the Data Migration Agent used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on building an AI powered migration assistant?
• When is the data migration agent the wrong choice?
• What do you have planned for the future of applications of AI at Datafold?

Contact Info

• LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email [e-mail] with your story.

Links

• Datafold
• Datafold Migration Agent
• Datafold data-diff
• Datafold Reconciliation Podcast Episode
• SQLGlot
• Lark parser
• Claude 3.5 Sonnet
• Looker
  • Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA