Java Developer with Web Crawler Experience
Role: Java Developer with Web Crawler Experience
Location: Austin TX(Hybrid)
Responsibilities:
1. Web Crawler Development: Design and implement efficient and scalable web crawlers in Java to collect data from various online sources.
2. Data Extraction: Develop and maintain systems for structured data extraction, handling various data formats (HTML, JSON, XML, etc.).
3. Data Storage and Processing: Design data storage and processing pipelines, ensuring extracted data is clean, structured, and easily accessible.
4. Performance Optimization: Optimize web crawling processes for speed, efficiency, and accuracy, while ensuring minimal impact on source websites.
5. Error Handling and Logging: Implement error-handling mechanisms and logging systems to detect and resolve issues during crawling operations.
6. Data Integrity and Compliance: Ensure data collection practices are ethical, legal, and compliant with relevant regulations (e.g., robots.txt, copyright laws).
Requirements:
Proficiency in Java and experience with Java-based web scraping libraries (e.g., Jsoup, Apache
Knowledge of web crawling frameworks and tools, such as Scrapy, Selenium, or Puppeteer.
Strong understanding of HTML, CSS, JavaScript, and web data structures.
Familiarity with data parsing and handling techniques for JSON, XML, and other common formats.
Experience with database technologies (SQL, NoSQL) to store and manage scraped data.
Knowledge of protocols, headers, proxies, and load handling.
Recommended Jobs
Sr Fire Protection Engineer
Sr Fire Protection Engineer Department: Sr Fire Protection Engineer Location: San Antonio, TX NIKA is hiring for a Sr Fire Protection Engineer to support the Defense Health Agency’s Engineering…
Sr. Staff Test Engineer
Company Description As a Sr Test to Sr Staff Test Engineer, you will be responsible to apply a broad level of expertise and knowledge in test discipline, leading and contributing to the developm…
Sales Executive
Job Description Job Description Job Title : Sales Executive Location : Waco, Bryan, Temple and Austin, Texas Integ is a commercial printing company delivering successful results from star…
Sterile Processing Tech
Overview Join our team as a PRN, Sterile Processing Tech in Jackonville, TX. Why Join Us? Thrive in a People-First Environment and Make Healthcare Better Thrive: We empower our team…
Weekend CNC Operator
Job Description Job Description Description: THE COMPANY Built for Heroes and based in Florence, Texas, Staccato is on a mission to proudly serve those who protect and embody American freedom…
Senior Producer
Archetype Entertainment is a new game studio in Austin, Texas led by industry veterans from BioWare and other AAA studios. Archetype is a division of Wizards of the Coast (Hasbro). We are building a…
PRN Physical Therapist Kruse Village
Job Description Job Description PRN Physical Therapist Kruse Village
CNA/ HHA Hiring Immediately - 78228
Urgently Hiring! It’s more than just a job! We are looking for YOU to make a difference in someone’s life! Come join one of the fastest-growing home care companies in the nation. As a leading …
Full Time Gastroenterology Job Katy, TX
Enterprise Medical Recruiting is assisting a private group in Katy, Texas, to add a new Gastroenterologist. The team is comprised of 1 GI and 7 APPs. They are looking for someone to do bread-and-butt…