Data Engineer - Web Scraper
- Worldwide
HENI is looking to increase its data-gathering capability and we are looking for Python Developers (Junior through to Senior) to help us do this. As a Python Developer/Web Scraper the role is split into three main areas:
- Maintenance of existing crawlers
- Setting up new crawlers
- Scraping the NFT/Blockchain market
Tooling
From a tooling perspective, we use Python (Scrapy) to set up the spiders. We have an AWS based infrastructure and data is stored in S3 buckets as well as SQL based databases (MySQL and PostgreSQL).
Requirements
- Writing and maintaining software for digital data collection for hundreds of websites
- Developing software to allow for scaling of data gathering across thousands of sources and across multiple team members
- Working with and deploying data pipelines (e.g. Scrapy, written in Python), process and cleaning the data, and storing accurately into our database
- Setting up scrapers, fixing any bugs or issues when websites changes
Benefits
- Competitive salary + bonus
- Work in a dynamic and fast-paced environment with new challenges
- Work with a modern tech stack and the latest frameworks
- Have your say – a real chance to influence the tech stack
- Working with a company that uses the most up to date blockchain technology
- Get involved in a variety of projects, see how they develop into polished products and services
- Flexible & fully remote working options – you choose where you work
- Competitive holiday allowance
- Full private healthcare
- Learning budget for personal development
Need to know information
- This role can be fully remote, anywhere across the globe.
- We are looking for people who can do this full time (40 hours per week).
- We can offer a competitive salary or hourly rate. It's almost impossible to put a figure on this as different countries have different living costs but we are very competitive.
- If you are based in the UK, we can offer a permanent contract. If you are based overseas, we can offer a freelance contract.
