Modern Web Scraping with Erez Naveh

Data Archives - Software Engineering Daily

Please click here to view this show’s transcript.

43:08

"But it makes a lot of sense especially when you think about how traditional healers and shamans have worked, they haven't felt that separation from nature like Western medics do. And so to rely on the knowledge of other species actually makes a lot of sense. It's probably a lot more than we know at the moment." - Jaap de Roode Jaap de Roode is a biology professor at Emory University, and he is the author of an astonishing new book called Doctors by Nature How Ants, Apes, and Other Animals Heal Themselves . I say astonishing because I had no idea about so much of what he explores in his book. It never occured to me to consider that other species use medicine and have been healing themselves forever. Jaap tells stories of animals across nature, from bumblebees to chimpanzees, how they use plants and natural substances to treat infections, to ward off parasites, to self-medicate. There's so much that we have learned from them, and there's so much more that we still can.…

大约1年之前 57:29

MP3•单集首页

Bright Data has developed some of the most sophisticated consumer tools available to scrape public web data. Erez Naveh is an entrepreneur and former engineer at Meta. He is currently the VP of Product at Bright Data. Erez joins us in this episode to talk about Bright Data’s mission to structure the open web, and the toolkit they’ve developed to make this possible.

Full Disclosure: Bright Data is a sponsor of Software Engineering Daily

Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up and glot.space, an AI-powered language learning app. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter, LinkedIn and his personal website – pawel.io.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern Web Scraping with Erez Naveh appeared first on Software Engineering Daily.

231集单集

#Data Science #Science #Tech News #News #Tech #Data Archives Software Engineering Daily

Data Archives - Software Engineering Daily

Modern Web Scraping with Erez Naveh

Data Archives - Software Engineering Daily

300 subscribers

published 大约1年之前

MP3•单集首页

Full Disclosure: Bright Data is a sponsor of Software Engineering Daily

Please click here to view this show’s transcript.

Sponsorship inquiries: sponsor@softwareengineeringdaily.com

The post Modern Web Scraping with Erez Naveh appeared first on Software Engineering Daily.

231集单集

#Data Science #Science #Tech News #News #Tech #Data Archives Software Engineering Daily

所有剧集

Data Archives - Software Engineering Daily

1
Hyperscaling SQL with Sam Lambert

40 weeks之前

—

Databases underpin almost every user experience on the web, but scaling a database is one of the most fundamental infrastructure challenges in software development. PlanetScale offers a MySQL platform that is managed and highly scaleable. Sam Lambert is the CEO of PlanetScale and he joins the show to talk about why he started the platform, scaling databases, using Vitess for SQL shard orchestration, and more. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business , an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm . Follow Lee at softwarearchitectureinsights.com , and see all his content at leeatchison.com . Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Hyperscaling SQL with Sam Lambert appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Iceberg at Netflix and Beyond with Ryan Blue 47:37

1 year之前47:37

47:37

Apache Iceberg is an open source high-performance format for huge data tables. Iceberg enables the use of SQL tables for big data, while making it possible for engines like Spark and Hive to safely work with the same tables, at the same time. Iceberg was started at Netflix by Ryan Blue and Dan Weeks, and was open-sourced and donated to the Apache Software Foundation in November 2018. It has now been adopted at many other companies including Airbnb, Apple, and Lyft. Ryan Blue joins the podcast to describe the origins of Iceberg, how it works, the problems it solves, collaborating with Apple and others to open-source it, and more. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business , an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm . Follow Lee at softwarearchitectureinsights.com , and see all his content at leeatchison.com . Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Iceberg at Netflix and Beyond with Ryan Blue appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Building a Data Lake with Adam Ferrari 46:19

1 year之前46:19

46:19

Starburst is a data lake analytics platform. It’s designed to help users work with structured data at scale, and is built on the open source platform, Trino . Adam Ferrari is the SVP of Engineering at Starburst. He joins the show to talk about Starburst, data engineering, and what it takes to build a data lake. Full Disclosure: Starburst is a sponsor of Software Engineering Daily Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer . Please click here to see the transcript of this episode. Watch the video episode here Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Building a Data Lake with Adam Ferrari appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Rama with Nathan Marz 45:06

1 year之前45:06

45:06

Building scalable software applications can be complex and typically requires dozens of different tools. The engineering often involves handling many arcane tasks that are distant from actual application logic. In addition, a lack of a cohesive model for building applications can lead to substantial engineering costs. Nathan Marz is the creator of Rama , which is a platform for building end-to-end, scalable backends. Previously Nathan led engineering at BackType which was acquired by Twitter in 2011. In addition, Nathan created the Apache Storm project and is the author of the book Big Data: Principles and best practices of scalable realtime data systems. Nathan joins the podcast today to talk about Rama, and how to build scalable software applications. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business , an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm . Follow Lee at softwarearchitectureinsights.com , and see all his content at leeatchison.com . Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Rama with Nathan Marz appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Bonus Episode: SurrealDB with Tobie Morgan Hitchcock 57:17

1 year之前57:17

57:17

SurrealDB is the result of a long-time collaboration between brothers Tobie and Jaime Morgan Hitchcock. The project has modest origins and started merely to support other projects the brothers were working on. However, over time the project grew and in 2021 they started working on it full-time. Since then the project has gained serious adoption. What’s makes SurrealDB so special? Tobie Morgan Hitchcock is the CEO of SurrealDB and he joins the show to talk about his multimodal database, support for graph and time series data, why they rewrote the entire project in Rust, and more. Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne. Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Bonus Episode: SurrealDB with Tobie Morgan Hitchcock appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Tracking Drug Smugglers and Migrating Databases with Benny Keinan and Lior Resisi 50:40

1 year之前50:40

50:40

Maritime logistics is the process organizing the movement of goods across the ocean. Historically, this has been a challenging problem because of the multinational nature of shipping, as well as piracy, smuggling, and legacy technology. It’s also profoundly important for security reasons, and because 90% of what we buy travels over the oceans. Ocean vessels produce a lot of CO2, which adds climate change and energy dimensions to maritime logistics. Windward AI is a maritime logistics platform that was started 13 years ago by two ex-Israeli naval officers. The idea for the company came from the observation that, at that time, it was hard or impossible to know what’s happening on the deep sea. Benny Keinan is the VP of R&D and Lior Resisi is the Data Platforms Group Lead at Windward AI. They join the podcast today to talk about the technical and practical challenges of maritime logistics, why Rockset was the right database for their unique datasets, the impact of the Ukraine war, and more. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business , an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm . Follow Lee at softwarearchitectureinsights.com , and see all his content at leeatchison.com . Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Tracking Drug Smugglers and Migrating Databases with Benny Keinan and Lior Resisi appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
The Right to Be Forgotten with Gal Ringel 47:45

1 year之前47:45

47:45

Data breaches at major companies are so now common that they hardly make the news. The Wikipedia page on data breaches lists over 350 between 2004 and 2023. The Equifax breach in 2017 was especially notable because over 160 million records were leaked, and much of the data was acquired by Equifax without individuals’ knowledge or consent. Data breaches are increasingly costly to companies and to affected users who must deal with the ensuing identity theft. In 2018 the European Union implemented the General Data Protection Regulation, or GDPR. Despite its mild name, the GDPR had major consequences for individuals’ rights to control their data, and for companies that operate in the EU. Among other things, it gives the right to have personal data removed from a company’s records. This is the so-called “right to be forgotten”. Gal Ringel is the Co-Founder and CEO of Mine , which allows users to identify which companies have their data, and it automates the process of removing data on behalf of its users. In this way, Mine aims to reduce online exposure and minimize risk for anyone using online services. Gal joins the show today to talk about his company, the impact of GDPR, and how his experience in military intelligence, venture capital, and tech led him to co-found the company. Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm Please click here to see the transcript of this episode . Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post The Right to Be Forgotten with Gal Ringel appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Sofascore with Josip Stuhli 49:48

1 year之前49:48

49:48

If you’re a sports fan and like to track sports statistics and results, you’ve probably heard of Sofascore. The website started in 2010 and ran on a modest single server. It now has 25 million monthly active users, covers 20 different sports, 11,000 leagues and tournaments, and is available in over 30 languages. Josip Stuhli has been with Sofascore for 13 years. He started there as an engineer and is currently CTO. Josip joins the show today to talk about the challenges Sofascore encountered over the years, and how the team solved them. He discusses dealing with traffic spikes from game days, structuring and restructuring the codebase, organizing the frontend and backend, and much more. Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer . Please clicke here for the full transcript of this episode Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Sofascore with Josip Stuhli appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Daytona with Ivan Burazin 47:50

1 year之前47:50

47:50

Cloud-based software development platforms such as GitHub Codespaces continue to grow in popularity. These platforms are attractive to enterprise organizations because they can be managed centrally with security controls. However, many, if not most, developers prefer a local IDE. Daytona is aiming to bridge that gap. It’s a layer between a local IDE and a backend server, so developers can work locally while interfacing invisibly with a remote environment. Ivan Burazin is the CEO and Co-Founder at Daytona , and he joins the show today to talk about how Daytona works, Spotify as an inspiration for his product, and more. Jordi Mon Companys is a product manager and marketer that specializes in software delivery, developer experience, cloud native and open source. He has developed his career at companies like GitLab, Weaveworks, Harness and other platform and devtool providers. His interests range from software supply chain security to open source innovation. You can reach out to him on Twitter at @jordimonpmm Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Daytona with Ivan Burazin appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
GraphAware with Luanne Misquitta 57:51

1 year之前57:51

57:51

Knowledge graphs are an intuitive way to define relationships between objects, events, situations, and concepts. Their ability to encode this information makes them an attractive database paradigm. Hume is a graph-based analysis solution developed by GraphAware . It represents data as a network of interconnected entities and provides analysis capabilities to extract insights from the data. Luanne Misquitta is VP of Engineering at GraphAware and she joins the show today to talk about graph databases, and the engineering of Hume. Starting her career as a software developer, Jocelyn Houle is now a Senior Director of Product Management at Securiti.ai, a unified data protection and governance platform. Before that, she was an Operating Partner at Capital One Ventures investing in data and AI startups. Jocelyn has been a founder of two startups and a full life cycle, technical product manager at large companies like Fannie Mae, Microsoft and Capital One. Follow Jocelyn on LinkedIn or Twitter @jocelynbyrne. Please click here to view this show’s transcript. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post GraphAware with Luanne Misquitta appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Chronosphere with Martin Mao 48:08

1 year之前48:08

48:08

Observability software helps teams to actively monitor and debug their systems, and these tools are increasingly vital in DevOps. However, it’s not uncommon for the volume of observability data to exceed the amount of actual business data. This creates two challenges – how to analyze the large stream of observability data, and how to keep down the compute and storage costs for that data. Chronosphere is a popular observability platform that works by identifying the data that’s actually being used to power dashboards and metrics. It then shows the cost for each segment of data, and allows users to decide if a metric is worth that cost. In this way, technical teams can manage costs by dynamically adjusting which data is analyzed and stored. Martin Mao is the Co-founder and CEO of Chronosphere and he joins the podcast today to talk about the growing challenge of managing observability data, and the design of Chronosphere. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business , an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm . Follow Lee at softwarearchitectureinsights.com , and see all his content at leeatchison.com . Please click here to see the transcript for this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Chronosphere with Martin Mao appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Streamlit with Amanda Kelly 47:06

1 year之前47:06

47:06

The importance of data teams is undeniable. Most companies today use data to drive decision-making on anything from software feature development to product strategy, hiring and marketing. In some companies data is the product, which can make data teams even more vital. But there’s a common problem – analyzing data is hard and time consuming. Lots of people have questions they want to answer with data, but data teams often don’t have the resources to move quickly. This can create a pernicious effect where organizations stop asking questions about their own data. Amanda Kelly thinks a lot about data and the dynamics of data teams inside organizations. She’s worked at Google X, and on self-driving cars and cybersecurity. Her experiences on data teams inspired her to co-found Streamlit, which is an open source Python library that gives primitives to assemble a data app for rapid data visualization and interaction. Her goal was to accelerate the iteration loop to go from a question to a data-driven answer. Amanda is currently the COO of Streamlit and a Product Director at Snowflake, and she joins us today to talk all about data and how she’s building Streamlit. Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from information visualization to quantum computing. Currently, Sean is Head of Marketing and Developer Relations at Skyflow and host of the podcast Partially Redacted, a podcast about privacy and security engineering. You can connect with Sean on Twitter @seanfalconer. Please click here to view this show’s transcript. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Streamlit with Amanda Kelly appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Modern Web Scraping with Erez Naveh 57:29

1 year之前57:29

57:29

Today it’s estimated there are over 1 billion websites on the internet. Much of this content is optimized to be viewed by human eyes, not consumed by machines. However, creating systems to automatically parse and structure the web greatly extends its utility, and paves the way for innovative solutions and applications. The industry of web scraping has emerged to do just that. However, many websites erect obstacles to hinder web scraping. This has created a new kind of arms race between developers and anti-scraping software. Bright Data has developed some of the most sophisticated consumer tools available to scrape public web data. Erez Naveh is an entrepreneur and former engineer at Meta. He is currently the VP of Product at Bright Data. Erez joins us in this episode to talk about Bright Data’s mission to structure the open web, and the toolkit they’ve developed to make this possible. Full Disclosure: Bright Data is a sponsor of Software Engineering Daily Paweł is the founder at flat.social the world’s first ‘flatverse’ start-up and glot.space , an AI-powered language learning app. Pawel’s background is as a full-stack software engineer with a lean and experimental approach towards product development. With a strong grounding in computing science, he spent the last decade getting early-stage products off the ground – both in startup and corporate settings. Follow Paweł on Twitter , LinkedIn and his personal website – pawel.io . Please click here to view this show’s transcript . Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Modern Web Scraping with Erez Naveh appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
Observability with Eduardo Silva 44:47

1 year之前44:47

44:47

There are hundreds of observability companies out there, and many ways to think about observability, such as application performance monitoring, server monitoring, and tracing. In a production application, multiple tools are often needed to get proper visibility on the application. This creates some challenges. Applications can produce lots of different observatory observability data, but how should the data be routed to the various downstream tools? In addition, how can data be selectively sent to different storage tiers to minimize costs? Calyptia is a service that helps manage observability data from source to destination. Eduardo Silva is the founder and CEO of Calyptia and he joins us in this episode. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best-selling book, Architecting for Scale (O’Reilly Media), is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business , an engaging and informative podcast produced for people looking to build and grow their digital business with the help of modern applications and processes developed for today’s fast-moving business environment. Listen at mdb.fm . Follow Lee at softwarearchitectureinsights.com , and see all his content at leeatchison.com . Please click here to view this show’s transcript . Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Observability with Eduardo Silva appeared first on Software Engineering Daily .…

Data Archives - Software Engineering Daily

1
AI and Business Analytics with John Adams 30:03

2 years之前30:03