We are hiring a Data Engineer to build the pipelines behind Upfynd's catalogue. We list products using what they publish about themselves on their own websites and in public software directories. You will build the systems that collect, clean and refresh that information responsibly.
Why this role matters
People make decisions based on what they read on Upfynd, so pricing, descriptions and links need to be right. Clean, fresh data is what lets a small team look after a very large directory without it going stale.
What your week looks like
- Build pipelines that gather public product information such as names, pricing, logos and links.
- Clean, normalise and de-duplicate records so each product appears exactly once.
- Detect when product information changes and flag listings that need a refresh.
- Respect robots.txt, rate limits and takedown requests in every crawler you build.
- Build the data warehouse and models that power analytics and reporting.
- Monitor pipeline health and fix failures quickly.
You will do well here if you have
- 2-4 years building data pipelines in production.
- Strong Python and SQL.
- Experience with workflow tools such as Airflow, Prefect or Dagster.
- Experience with web data collection using tools like Scrapy or Playwright.
- Care for data quality, documentation and responsible collection.
Nice to have
- Experience with dbt and a warehouse such as BigQuery or PostgreSQL.
- Experience with entity resolution and fuzzy matching.
- Familiarity with Docker and cloud storage.
Your first 30, 90 and 180 days
- 30 days: You understand how catalogue data flows today and where it breaks.
- 90 days: You have shipped a pipeline improvement that reduces duplicates or stale listings.
- 180 days: You own the catalogue data platform, with reliable freshness checks and clear data quality metrics.
Where and how you will work
This is a full-time, onsite role at our office on the 11th floor of O-Hub, Infocity, Bhubaneswar. You will sit with the people you build with, which keeps decisions quick and conversations easy. Your everyday tools will include GitHub, Python, Airflow, PostgreSQL, MongoDB and Docker.
About Upfynd
Upfynd is a directory of new software, ranked by the people who actually use it. Every person gets one upvote per product, reviews come from real users, a person on our team checks every listing before it goes live, and nobody can pay to move up. Makers list and claim their products for free, and anything sponsored is clearly labelled and never touches rankings.
How to apply
Send us your CV along with links to pipelines or data projects you have built. Just like every listing on Upfynd, every application is read by a real person, and we reply to everyone who applies.
Skills and tools
- Python
- SQL
- Apache Airflow
- Scrapy
- ETL
- Data Modeling
- PostgreSQL
- MongoDB
- dbt