Scaling Airflow To 11,000 DAGs Across Three Regions At Intercom With András Gombosi And Paul Vickers The Data Flowcast: Mastering Apache Airflow ® For Data Engineering And AI podcast

内容由The Data Flowcast提供。所有播客内容（包括剧集、图形和播客描述）均由 The Data Flowcast 或其播客平台合作伙伴直接上传和提供。如果您认为有人在未经您许可的情况下使用您的受版权保护的作品，您可以按照此处概述的流程进行操作https://zh.player.fm/legal。

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI »
Scaling Airflow to 11,000 DAGs Across Three Regions at Intercom with András Gombosi and Paul Vickers

3d ago 34:24

MP3•单集首页

The evolution of Intercom’s data infrastructure reveals how a well-built orchestration system can scale to serve global needs. With thousands of DAGs powering analytics, AI and customer operations, the team’s approach combines technical depth with organizational insight.

In this episode, András Gombosi, Senior Engineering Manager of Data Infra and Analytics Engineering, and Paul Vickers, Principal Engineer, both at Intercom, share how they built one of the largest Airflow deployments in production and enabled self-serve data platforms across teams.

Key Takeaways:

00:00 Introduction.

04:24 Community input encourages confident adoption of a common platform.

08:50 Self-serve workflows require consistent guardrails and review.

09:25 Internal infrastructure support accelerates scalable deployments.

13:26 Batch LLM processing benefits from a configuration-driven design.

15:20 Standardized development environments enable effective AI-assisted work.

19:58 Applied AI enhances internal analysis and operational enablement.

27:27 Strong test coverage and staged upgrades protect stability.

30:36 Proactive observability and on-call ownership improve outcomes.

Resources Mentioned:

András Gombosi

https://www.linkedin.com/in/andrasgombosi/

Paul Vickers

https://www.linkedin.com/in/paul-vickers-a22b76a3/

Intercom | LinkedIn

https://www.linkedin.com/company/intercom/

Intercom | Website

https://www.intercom.com

Apache Airflow

https://airflow.apache.org/

dbtLabs

https://www.getdbt.com/

Snowflake Cortex AI

https://www.snowflake.com/en/product/features/cortex/

Datadog

https://www.datadoghq.com/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

81集单集

#IT Management #Tech #Industries #Airflow