使用Player FM应用程序离线!
763: Web Scraping + Reverse Engineering APIs
Manage episode 415684901 series 1469447
Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease.
Show Notes- 00:00 Welcome to Syntax!
- 03:13 Brought to you by Sentry.io.
- 05:00 What is scraping?
- Socialblade.
- 08:01 Examples of past scrapers.
- Canadian Tire.
- 10:06 Cloud app downloader.
- 16:13 Other use cases.
- 16:58 Scraping 101.
- 17:28 Client Side.
- 19:08 Private API.
- Proxyman.
- 22:40 Server rendered.
- 23:27 Initial state.
- 24:57 What format is the data in?
- Google Puppeteer Extension.
- 27:08 Working with the DOM.
- 27:12 Linkedom npm package.
- 29:02 querySelector everything.
- 31:28 How to find the elements without classes.
- 34:08 Use XPath selectors for select by word.
- 34:53 Make them as flexible as you can. Classes change!
- 35:10 AI is good at this!
- 36:26 File downloading.
- 38:20 Working with protected routes.
- Proxyman.
- 40:41 Programatically retrieve authentication keys because they are short-lived.
- Fetch Cookie.
- 43:20 Deal-breakers.
- Mechanical Turk.
- 44:58 What happened with Amazon?
- Uniqlo Self-Checkout
- 46:42 Wes’ portable refrigerator utopia.
- 47:25 Sick Picks & Shameless Plugs.
- Scott: KeyboardCleanTool.
- Wes: Yabai.
- Scott: Syntax on YouTube
Syntax: X Instagram Tiktok LinkedIn Threads
Wes: X Instagram Tiktok LinkedIn Threads
772集单集
Manage episode 415684901 series 1469447
Web scraping 101! Dive into the world of web scraping with Scott and Wes as they explore everything from tooling setup and navigating protected routes to effective data management. In this Tasty Treat episode, you’ll gain invaluable insights and techniques to scrape (almost) any website with ease.
Show Notes- 00:00 Welcome to Syntax!
- 03:13 Brought to you by Sentry.io.
- 05:00 What is scraping?
- Socialblade.
- 08:01 Examples of past scrapers.
- Canadian Tire.
- 10:06 Cloud app downloader.
- 16:13 Other use cases.
- 16:58 Scraping 101.
- 17:28 Client Side.
- 19:08 Private API.
- Proxyman.
- 22:40 Server rendered.
- 23:27 Initial state.
- 24:57 What format is the data in?
- Google Puppeteer Extension.
- 27:08 Working with the DOM.
- 27:12 Linkedom npm package.
- 29:02 querySelector everything.
- 31:28 How to find the elements without classes.
- 34:08 Use XPath selectors for select by word.
- 34:53 Make them as flexible as you can. Classes change!
- 35:10 AI is good at this!
- 36:26 File downloading.
- 38:20 Working with protected routes.
- Proxyman.
- 40:41 Programatically retrieve authentication keys because they are short-lived.
- Fetch Cookie.
- 43:20 Deal-breakers.
- Mechanical Turk.
- 44:58 What happened with Amazon?
- Uniqlo Self-Checkout
- 46:42 Wes’ portable refrigerator utopia.
- 47:25 Sick Picks & Shameless Plugs.
- Scott: KeyboardCleanTool.
- Wes: Yabai.
- Scott: Syntax on YouTube
Syntax: X Instagram Tiktok LinkedIn Threads
Wes: X Instagram Tiktok LinkedIn Threads
772集单集
Todos os episódios
×欢迎使用Player FM
Player FM正在网上搜索高质量的播客,以便您现在享受。它是最好的播客应用程序,适用于安卓、iPhone和网络。注册以跨设备同步订阅。