PROJECT

Scraping Websites for Structured Contact Data

Custom software development, Website data scraping, Contact data extraction, Reusable scraper development, Data cleaning and validation, CSV and Excel export, JSON export, CRM-ready data

  • Industry: Market research
  • Croatia Location: Croatia
  • Delivery time: 10 weeks
  • Technologies: Python, JavaScript / TypeScript (Node.js)
3 months

This client spends time collecting contact and business details from company websites, public databases, and online directories. The software is a custom scraping websites solution that takes source URLs and selected fields. It extracts contact records, cleans and validates them, and returns organized files for the market research team. Reusable extraction makes recurring data collection less dependent on manual entry.

Summary

Problem

The client manually spends time collecting and checking contact details.

Solution

The result is a reusable scraper for extracting, cleaning, validating and organizing specified web data.

Result

The software is intended to reduce manual data collection and repeated processing during market research.

Client story

The client is a market research company from Croatia. It collects company names, email addresses and phone numbers from company websites, public databases and online directories.

The data also includes physical addresses and other company details.

The team prepares contact details for market analysis, business development, telesales and CRM imports. Manual collection, repeated checks and duplicate records take time and create additional work.

The requirement was simple: extract selected contact fields from defined sources, repeat the collection as needed and deliver clean, structured files.

The solution was built around screens for source setup, run monitoring, data cleaning and export settings. Records could be reviewed before use.

The client requested

  1. Extract contact fields from websites, databases, directories and online sources
  2. Run a reusable scraper and obtain its source code or notebook
  3. Clean, validate, organize and remove duplicates from extracted records
  4. Export structured data to CSV, Excel, JSON or categorized workbooks

Why ready-made software was not a fit

Ready-made data extraction tools cover only parts of the task, but their descriptions do not collectively define the client's exact fields, cleaning rules and output files. A custom workflow automation tool was needed to combine recurring scraping with validation, deduplication, organized workbooks and reusable code in one process.

Project overview

The solution was designed around the client's actual sources, fields and rules. On the first screen, users define website URLs, contact fields and recurring data collection settings before starting.

An extraction monitor was created to collect data from one or more sources. It displays the run status, source URLs, extracted records and notes about loading or access in one place.

Records then go through cleaning and checking. Duplicate matches, validation status, contact fields and categories remain visible so the data administrator can prepare a consistent contact list.

The final screen was designed for practical data handover. It exports organized records to CSV, Excel or JSON and provides setup instructions or process notes together with the file.

The scraper uses reusable code and can be adapted to the client's rules. Documentation and training materials support independent work with the system.

Key features delivered

  • Website and source list setup
  • Extraction of selected contact fields
  • Reusable and repeatable scraping
  • Extraction run monitoring
  • Email and phone number collection
  • Record validation and cleaning
  • Duplicate detection and removal
  • Contact data categorization
  • Export to CSV, Excel and JSON

Can be added

+ Regular Excel database updates + Email and domain verification + Keyword-based categorization + Date-based categorization + Source code review + Additional scraping methods

Who else can use this solution

These businesses have a similar task, and the same solution can be adapted to their sources, fields, and delivery rules.

A business services company

Task It collects company and contact details from public websites and directories.
How the solution helps The solution extracts selected fields, removes duplicate records, and prepares files for CRM import or business development.

A professional services company

Task It builds contact lists from company websites and online sources.
How the solution helps The solution collects email addresses, phone numbers, and business details in structured files.

Technology stack

Python

Scraping and browser automation

Python supports web crawling, page parsing, and browser-based extraction.

JavaScript / TypeScript (Node.js)

Structured file processing

JavaScript and TypeScript support structured file processing.

Scrapy

Website crawling and scraping

Scrapy (a framework for crawling and scraping websites) collects records from defined sources.

BeautifulSoup

HTML page parsing

BeautifulSoup (a library that reads information from web pages) parses page content and finds specified fields.

selenium

Browser-based extraction

selenium (a tool that controls a web browser) supports extraction where loading behavior requires a browser.

csv-parse

CSV reading and parsing

csv-parse (a Node.js library for reading CSV files) reads structured CSV data.

maatwebsite/excel

Excel and CSV export

maatwebsite/excel (a Laravel package for Excel and CSV files) supports organized spreadsheet delivery.

jsonify

JSON data handling

jsonify (a JavaScript library for working with JSON) supports JSON records without changing global objects.

CSV, Excel, JSON

Structured data delivery

These formats carry organized contact records for analysis, CRM import, and other business uses.

Integrations

CRM systems

The solution exports organized contact records in formats suitable for CRM import.

Excel

The solution exports categorized contact data and organized workbooks as Excel files.

CSV files

The solution reads and writes comma-separated contact records for structured exchange.

JSON files

The solution delivers structured records as JSON for systems that use this format.

Email verification service

An email verification service can check email addresses and domains before delivery.

The result

Set precise sources and fields before collection

The first screen keeps the extraction request specific. Website URLs, source lists, and selected contact fields define what the scraper collects.

How it works

  1. The software stores the specified websites and source lists.
  2. The software keeps selected fields such as email, phone number, and postal address.
  3. The software stores recurring extraction settings for repeat runs.
  • Email addresses and phone numbers are available as separate contact fields.
  • Physical addresses and business details can be included in the selected fields.
Source and field setup

Source and field setup

Functions on this screen
Scrape website data Scrape email addresses Phone Number Extraction Extract physical addresses

See extraction progress and collected records

The extraction monitor brings source URLs, run status, and extracted records together. Access warnings and loading notes make source behavior visible during collection.

The team gets a clear view of the records returned from one or more websites before the cleaning stage.

How it works

  1. Scrapy (a framework for crawling and scraping websites) collects data from the listed sources.
  2. BeautifulSoup (a library that parses web pages) reads website content and extracts specified fields.
  3. The software shows the run state, source URLs, extracted data, and loading or access notes.
  • The extraction supports multiple pages and websites.
  • The scraper can be rerun with reusable rules.
Note: Collection rules must respect website behavior, platform terms, privacy laws, GDPR, and other applicable requirements.
Extraction run monitor

Extraction run monitor

Functions on this screen
Scrape website data Extract Data

Prepare cleaner contact records

The cleaning screen gives the data manager one place to review validation status, duplicate matches, and categories. Contact records stay organized before export.

Duplicate indicators help separate repeated entries from the usable contact list, while categories prepare the data for analysis or outreach.

How it works

  1. The software displays email addresses, phone numbers, and postal addresses in the contact table.
  2. The software compares records and displays duplicate matches and validation status.
  3. The software applies categories to organized contact records.
  • Records can be categorized by dates or keywords.
  • A data deduplication service can support duplicate prevention.
Clean and deduplicate records

Clean and deduplicate records

Functions on this screen
Organize contact data Duplicate Prevention Data deduplication service Clean Large Datasets Data Categorization Extract Contact Details

Deliver data in the format the team uses

The export screen turns reviewed records into a practical handoff for analysis or CRM import. CSV, Excel, and JSON keep the delivery flexible without changing the source fields.

Setup instructions and process notes stay linked to the delivery settings, helping the team understand how the extraction works.

How it works

  1. The software prepares organized contact records for CSV, Excel, or JSON output.
  2. The software creates a file summary for the selected format.
  3. The software provides links to setup instructions or process notes.
  • CSV files support structured exchange.
  • Excel files support categorized workbooks.
  • JSON files support system-to-system data use.
Export and delivery settings

Export and delivery settings

Functions on this screen
Export company contact lists Excel Export Tool Manage CSV data

What the solution gives

With this solution, the market research company can:

  • Collect specified contact and business fields from defined online sources
  • Rerun extraction rules for recurring data collection
  • Review validation status and duplicate matches before delivery
  • Organize records by categories such as dates or keywords
  • Export clean contact data for analysis and CRM import
  • Maintain reusable code, setup instructions, and process notes

Could ready-made systems do the job?

Before the design, the task was compared with ready-made products by their own descriptions.

What the client needs Web scraping APIs Zyte, Scrapfly, ScrapingBee Web scraping tools Octoparse, Oxylabs Data extraction software Data Extraction Custom solution
Extract contact fields from sources partly
Rerun scraper with source code — partly partly
Clean and deduplicate records — partly
Export structured contact files
Fits the client's own rules: stages, roles, reports — — —
All steps in one place — — —
No manual steps between programs partly — partly

By the public descriptions of the products and their category.

Ready-made products cover website access, crawling, extraction, and file delivery, while general data extraction software covers broader data retrieval.

A ready-made product is enough when the task needs only one of these parts. This client needed its own software because the required fields, cleaning rules, categories, reusable code, and exports had to work together in one place.

The products describe automated collection, but the client's review, validation, deduplication, and delivery rules still require a tailored process.

In the end

In the end, the solution was made to take manual work out of the path from public online sources to clean contact files.

FAQ

Thinking about building a similar system? Read the answers below to see how scraping, structured exports, and custom rules fit together.