Responsibilities
Design and develop scalable, reliable ETL processes using Python and DataBricks
Build and maintain data pipelines for extracting, transforming, and loading data from various sources
Take ownership of the full data engineering lifecycle, from extraction to transformation and loading
Optimize data workflows, ensuring robust error handling, monitoring, and performance tuning
Work within an agile environment, actively participating in sprint planning, stand-ups, and retrospectives
Conduct code reviews and maintain high coding standards
Develop tooling and automation scripts to improve operational efficiency
Implement comprehensive testing for data pipelines, including unit and integration tests
Integrate data sources via REST APIs and other techniques
Maintain up-to-date technical documentation and data flow diagrams
Requirements
Proficient in Python, with a focus on clean, efficient, and maintainable code
Hands-on experience with DataBricks and cloud-based data engineering tools
Skilled in Snowflake or other cloud data warehousing platforms
Solid understanding of ETL principles, data modeling, and integration best practices
Comfortable working in agile, fast-paced, collaborative environments
Experienced with Git and version control systems
Detail-oriented with a strong problem-solving mindset
Familiar with Linux systems and REST API integrations
Advantages
Experience with data visualization tools (such as Power BI)
Knowledge of data orchestration tools like Apache Airflow
Exposure to big data frameworks including Hadoop or Spark
Background in database administration or performance optimization