Source review
A look at the target sites before any code: how the data is laid out, how they protect it, what volume is realistic and what it will cost to run.
Web scraping & data extraction
Scrapers that collect data from sites without an API, at scale and past their protections, and deliver it clean.
Prices, listings, contacts, catalogs: a lot of the data a business runs on lives on someone else's website, with no API and no export button. Copying it by hand works for a hundred rows and falls apart at a hundred thousand.
I build scrapers that turn those pages into structured data you can use: fields parsed and normalized, duplicates removed, and results delivered to your database, CRM, a CSV or Google Sheets on a schedule. You get records, not a folder of HTML.
The hard part is keeping a scraper alive. Sites throttle requests, ban IPs, show CAPTCHAs and change their markup. I've run a scraper for a lead generation client for three years: it processed over a million contacts and got past the site's blocks more than 10,000 times, with proxies, a CAPTCHA service and retries built in, and monitoring that shows when something breaks.
What you get
What's included
Scrapers that survive blocks
Proxy pools, CAPTCHA solving services, rate limits, retries and backoff, so one bad response doesn't stop a run of thousands of pages.
Parsing and normalization
Names, addresses, prices and dates turned into consistent fields, with duplicates merged, so the data is ready to use as it arrives.
Headless browsers
Headless Chrome through Selenium WebDriver for pages that need JavaScript, logins or clicks. It is slower and costs more, so it only runs where plain HTTP requests fall short.
Scheduling and monitoring
Runs on a schedule or on demand, with logs, cost and speed per run, and alerts when a site changes and the parser needs attention.
Delivery where you work
Results in your database, CRM, a REST API, CSV files or Google Sheets, in the format your team already uses.
Typical stack
- PHP
- Node.js
- TypeScript
- Python
- cURL
- Selenium WebDriver
- PostgreSQL
- Docker
Case studies
Related work
Lead generation · US lead generation business
1M+ contacts processed, 10,000+ site blocks bypassed
Multithreaded scraping system that turns people-search data into leads
An app that scrapes FamilyTreeNow search results and profiles, parses them into phone numbers and address history, and turns the right matches into leads. Work runs on parallel threads, each behind its own proxy, and a paid service solves the CAPTCHAs.
- PHP
- Yii 2
- MySQL
- pthreads
- cURL
- Selenium WebDriver
- Google Sheets API
- Anti-Captcha API
- Docker
Freelancing / job search · Own product
~3,900 new postings a day, ~500 alerts to Telegram
Upwork job monitor that keeps every posting and sends matches to Telegram
A personal tool on the official Upwork API: a worker copies every new posting into Postgres, my own filters run over that archive, and the matches arrive in Telegram.
- Next.js
- TypeScript
- PostgreSQL
- Drizzle ORM
- Upwork GraphQL API
- Telegram Bot API
- Docker
FAQ
Common questions
Is web scraping legal?
Collecting publicly available data is common practice, but the rules depend on the site's terms, your country and the kind of data, especially personal data. We review this before development starts, and I'll tell you if a project looks risky.
What happens when the website changes?
Sites change their markup from time to time. Monitoring flags a drop in results or a parsing error, and I fix the parser. I can support scrapers on a small monthly retainer.
Should we scrape or use an API?
If the site has an official API that gives you the data you need, use it: it's more stable and cheaper to run. Scraping makes sense when there is no API or the API leaves out what you need.
Let's see if this fits your project.
Send a short brief and get an estimate within 24 hours. Not sure where to start? Begin with a code audit.