Web Crawler Engineer

Job Summary

A company is looking for a Senior Web Crawler Engineer.

Key Responsibilities

• Design, implement, and maintain scalable web crawlers for efficient data collection and indexing
• Develop solutions to extract structured and unstructured data from diverse web sources
• Optimize crawling strategies for coverage, freshness, and efficiency while ensuring compliance with web standards

Required Qualifications

• Strong programming skills in Python, Rust, C++, or similar languages
• Familiarity with distributed systems, cloud infrastructure, and big data technologies
• Experience with HTML parsing and handling dynamically generated sites
• Knowledge of anti-crawling techniques and mitigation strategies
• Practical experience equivalent to a BS/MS degree in Computer Science, Engineering, or a related field