Job Description

Web Scrape Engineer / Data Harvesting Engineer
About the Company
We are a fast-growing organization specializing in complex supply chain solutions, procurement, program support, and inventory management for government and enterprise clients. We pride ourselves on rapid, innovative, and agile responses to mission-critical needs.
Data is central to how we source, monitor inventories, track suppliers, and optimize procurement. We're seeking a skilled Web Scrape Engineer to help us acquire and maintain high-quality data from external sources in support of our mission.
Position Summary
As a Web Scrape Engineer, you will design, build, maintain, and scale automated systems for extracting structured and semi-structured data from a wide variety of web sources (supplier catalogs, marketplace sites, auction/surplus listings, procurement portals, OEM part databases, etc.). You will work closely with procurement, supply chain, and analytics teams to help feed data pipelines, support decision-making, and maintain data integrity.
You will also implement mechanisms for detecting changes, handling anomalies, and ensuring legal and ethical scraper operation (e.g. respecting robots.txt, rate limiting, captcha bypass where allowed).
Key Responsibilities
Architecture & Design

• Design scalable, maintainable web-scraping systems and frameworks.
• Choose appropriate tools (Python, JavaScript, headless browsers, APIs).
• Ensure systems are resilient to site changes and easy to maintain.

Implementation & Development

• Develop robust crawlers and scrapers to ingest data from multiple sources.
• Handle static and dynamic content with headless browser automation.
• Parse, clean, normalize, and validate scraped data.
• Build incremental/change-detection logic for efficient updates.
• Implement error recovery, retry logic, proxies, and rate-limiting strategies.

Data Pipeline & Integration

• Feed cleaned data into internal systems, databases, or warehouses.
• Collaborate with analytics teams on schemas and reporting needs.
• Provide APIs or access points for internal users.

Maintenance & Monitoring

• Monitor scraping jobs and proactively fix broken scrapers.
• Optimize performance, logging, and monitoring.
• Establish automated testing and regression checks.

Quality, Compliance & Ethics

• Ensure scrapers operate within legal and ethical boundaries.
• Minimize server load and respect site policies.
• Stay current with anti-scraping defenses and mitigation strategies.

Collaboration

• Partner with procurement, supply chain, and parts teams to prioritize sources.
• Provide insights on data trends.
• Mentor junior engineers where applicable.

Required Qualifications
Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).
3+ years of experience building web scraping / data ingestion systems.
Strong proficiency in Python, JavaScript/Node.js, or similar.
Hands-on experience with HTML parsing libraries and selectors.
Familiarity with headless browsers (Selenium, Puppeteer, Playwright).
Experience handling proxies, distributed crawling, and anti-bot measures.
Strong understanding of HTTP, sessions, cookies, and dynamic content.
Database integration experience (SQL/NoSQL).
Version control (Git), CI/CD familiarity, and debugging skills.
Strong problem-solving and communication abilities.
Preferred Skills
Experience in procurement, logistics, or supply chain environments.
Knowledge of government, aerospace, or defense contracting data sources.
Familiarity with anti-scraping defenses and evasion techniques.
Distributed systems, orchestration (Kubernetes), or job queueing tools.
Monitoring and alerting tools (Grafana, Prometheus, ELK, DataDog).
NLP/ML for data classification or entity extraction (optional).
Success Metrics
Coverage and freshness of scraped sources.
Uptime and reliability of scraping jobs.
Speed, scalability, and performance.
Data quality, error rates, and accuracy.
Responsiveness to site changes and maintenance efficiency.
Business impact (supporting better procurement decisions).
Reporting & Structure
This role reports to the Director of Data or equivalent leadership. You'll collaborate with procurement, analytics, and engineering stakeholders, with potential to lead additional engineers as the function scales.

Skills Required