Andrew Dorokhov
Estimate

Lead generation · US lead generation business

A multithreaded scraping system that turned millions of contacts into leads

An app that scrapes FamilyTreeNow search results and profiles, parses them into phone numbers and address history, and turns the right matches into leads. Work runs on parallel threads, each behind its own proxy, and a paid service solves the CAPTCHAs.

in production
3 years
site blocks bypassed
10,000+
contacts processed
1M+
Get a project estimate

The challenge

What the client needed

The client runs a lead generation business in the US. The data they collect, people's phone numbers and addresses, sits on FamilyTreeNow: on profile pages behind search result pages, with no API and no export. Collecting it by hand takes a few minutes per person, and the client needed it for thousands of people at a time.

FamilyTreeNow has the phones and addresses, and it defends them. Send requests too fast and you get a 429 or a CAPTCHA page; keep sending from one IP and you get blocked. Proxies drop connections, Cloudflare adds its own layer of protection on top, and halfway through the project the site moved from reCAPTCHA to hCaptcha.

A wrong match costs money as well, because the client ends up calling the wrong person. The client had their own rules for deciding which search result is the right one, and the system had to apply them to every row.

The approach

How I solved it

01

Two passes over FamilyTreeNow

For each row the thread searches the site by name and location, pages through the results and then opens every candidate's profile. A parser turns the profile HTML into structured data: phone numbers, current and past addresses, and the other names a person is listed under. Names get normalized on the way in, from middle names to suffixes written as "JR" or ", Jr.", so names from different jobs compare cleanly. Most pages come over cURL; for pages that need a real browser, a second engine drives headless Chrome through Selenium WebDriver in Docker, and Whitepages plugs in as a second source behind the same interface.

02

The client's rules on parsed data

The client designed their own algorithm for deciding which candidate is the right person. It runs on the parsed profiles as a separate step: only a candidate who passes it and has at least one phone number becomes a lead, and every other row is marked as no match. Each row of each job goes through the same checks.

03

A thread and a proxy per job

A supervisor on thread-safe PHP with pthreads starts a thread for each launched job. The thread takes a free proxy from the pool, keeps its own cookies and holds the proxy until the job is done, so several jobs run side by side without sharing an IP. Pause a job and it later resumes from the last processed row.

04

CAPTCHAs with a price tag

When the site shows a reCAPTCHA or an hCaptcha, the thread sends it to Anti-Captcha, submits the token and carries on. Each solve is counted per job, so the client sees CAPTCHAs per 100 requests, money spent and the cost of 1,000 leads. A cron job checks the Anti-Captcha balance every 10 minutes.

05

Failures turn into retries

Proxy timeouts, SSL errors, Cloudflare pages, 429s and 5xx responses each get their own handling: a pause, a retry or a fresh user agent. A request gets up to 10 attempts and a CAPTCHA up to 15 submissions, so one bad response doesn't stop a 5,000-name run.

06

New leads and fresh old ones

Besides finding new leads, the system refreshes existing records: it searches each one again and updates phones and addresses, keeping the old numbers when nothing newer turns up. Job input comes in three layouts, from a single full-name column to two people per row, and rows with no name or address are skipped.

In motion

Highlights

01

Jobs in parallel

Each launched job gets a thread and a proxy. Progress, leads and match rate update as the threads work through their rows, and the totals at the top add up all of them.

Spreadsheets page with three running sheets: processed counts, progress bars and lead counts grow as the threads work.
02

Leads as they arrive

The run page counts requests, CAPTCHAs and money spent as they happen. New rows drop into the feed with their result and phone numbers.

Run page of one sheet: new records appear at the top of the feed, some as leads with phone numbers, while requests and CAPTCHA counters grow.
03

A rate-limited proxy

When the site answers a proxy with a 429, the thread waits and retries on the same proxy. The pool page shows the retry, then the proxy goes back to work.

Proxy table: request counters grow on three proxies in use, one switches to '429 · retry 2 / 10' and then back to 'In use'.

See it in action

The product

Every job with its progress, leads and the thread working on it.
One run: progress, speed, CAPTCHA spending and the cost of 1,000 leads.
Processed rows with their result, phone numbers and current address.
Adding a job: find new leads or refresh old ones, in one of three layouts.
Source, engine, CAPTCHA service and the request policy in one place.

The outcome

What the client got

  • Millions of contacts went through the system, and it got past the site's blocks more than 10,000 times.
  • Several jobs run at once, each on its own thread and proxy, and pick up where they stopped after a pause.
  • Nobody sits and solves CAPTCHAs: Anti-Captcha handles reCAPTCHA and hCaptcha while the thread keeps going.
  • Each job shows its speed and cost: leads, records and requests per hour, and dollars per 1,000 leads.
  • One export writes the leads to Google Sheets, ready for the client's team.
  • The proxy page shows load and CAPTCHA rate per IP, so a burned proxy is easy to spot and switch off.

Need something similar built?

Send a short brief about your project. Within 24 hours you get an estimate or a clear next step, even if we don't work together.

Get a project estimate