Epicor Kinetic consultant leading master, transaction, and historical data migrations. Supporting a global industrial equipment manufacturer’s post - merger platform standardization.
About the role
- Software Engineer building scalable crawler infrastructure for NationGraph’s public-sector intelligence platform. Automating web data acquisition, extraction, monitoring, and repair across government sources.
Responsibilities
- Own scraping infrastructure end-to-end
- Build systems for creating, deploying, scheduling, monitoring, and maintaining hundreds of thousands of scrapers
- Design abstractions to scale toward millions of sources without linearly scaling engineering effort
- Work with government websites, APIs, procurement systems, PDFs, spreadsheets, meeting records, and legacy systems
- Handle changing websites, undocumented APIs, rate limits, broken sources, and edge cases
- Build orchestration, concurrency, retries, backfills, change detection, observability, cost management, and failure recovery systems
- Ensure detection of broken sources, disappearing data, and silently incorrect extraction
- Work with the ML Research team to use LLMs and agents to discover sources, understand websites, generate scraping logic, detect changes, diagnose and repair failures, and validate data
- Identify common platforms and patterns that can unlock thousands of government agencies
- Make new sources increasingly cheap and automated to onboard
- Help map public-sector information across the U.S. and Canada and build the foundation for worldwide coverage
- Help shape the evolving engineering stack and technical decisions
Requirements
- Production experience building web crawlers, scraping systems, browser automation, or large-scale external data pipelines
- Strong in Python, Go, TypeScript, or another backend/systems language
- Understanding of JavaScript rendering, sessions and cookies, rate limits, proxies, authentication, and changing schemas and websites
- Understanding of distributed systems, including orchestration, queues and concurrency, idempotency, retries, backfills, observability, and failure recovery
- Deep concern for data quality, correctness, and reliability
- Excitement about using LLMs and agents to automate scraping work
- Ability to think in terms of scalable systems rather than individual websites
- Ability to thrive in ambiguity
- Backgrounds in alternative data, quantitative research infrastructure, search and crawling, AI data infrastructure, knowledge graphs, large-scale document processing, or data aggregation are of interest but not requirements
Benefits
- Equity
- Significant ownership over technical decisions and product outcomes
- Opportunity to work closely with the CEO, CTO, and engineering and research team
Job title
Job type
Experience level
Salary
Degree requirement
No Education Requirement
Tech skills
Location requirements
Report this job
Found something wrong with the page? Please let us know by submitting a report below.
Senior data scientists and engineers designing evaluation tasks and training frontier AI models. Mercor partners with leading AI labs and enterprises using human expertise.
Ingénieur Data concevant l’architecture de pipelines AWS et la modélisation dbt pour la plateforme SaaS analytique de BEEM. Collaboration avec les consultants ADS et évolution du produit data/IA.
Senior Cloud Data Architect designing Google Cloud data architectures for Napkyn, a Kepler Group marketing technology company. Integrating marketing data to drive trusted insights and client decisions.
Senior Associate Data Engineer building Azure and Databricks pipelines for Publicis Sapient’s digital transformation clients. Supporting AI - enabled data services, retrieval systems, and scalable cloud platforms.
Principal Data Engineer architecting Snowflake, dbt, and streaming infrastructure for JobGet’s AI - powered hiring platform. Enabling real - time matching, analytics, and machine - learning data products.
SAP Data Migration Lead guiding S/4HANA Greenfield migration for a global glass reinforcement manufacturer. Leading governance, data readiness, migration cycles, cutover, and go - live support.
Senior Data Migration Specialist leading secure EMR migrations for WELLSTAR’s Canadian healthcare platforms. Improving data tools, validation, governance, and go - live readiness.
Senior data scientists and engineers designing evaluation tasks and criteria for frontier AI models. Mercor partners with leading AI labs to advance real - world AI capabilities.
Senior data science and engineering experts designing evaluations and workflows to train frontier AI models. Contract work with Mercor and leading AI research labs.