Principal Data Engineer

• Lead, mentor, and manage a team of data engineers specializing in streaming technologies. 
• Design and maintain scalable data storage solutions with ClickHouse for fast analytics on streaming data. 
• Design and implement high-throughput, low-latency streaming data pipelines using Apache Kafka. 
• Oversee the development of stream processing applications using Apache Spark or Apache Flink. 
• Implement real-time data transformations and analytics using KSQL. 
• Ensure integration between on-premises streaming solutions and cloud services (e.g., BigQuery, Looker). 
• Lead the design and implementation of ETL processes, extracting, transforming, and loading data into a data warehouse. 
• Ensure data integrity, consistency, and accuracy through robust data quality assurance. 
• Optimize performance and implement best practices in data engineering, covering data quality, security, and efficiency. 
• Collaborate with stakeholders to gather requirements, aligning data strategies with business objectives. 
• Stay current with emerging technologies in both streaming and cloud environments, evaluating their potential application. 

Required Skills: 

• 5+ years of hands-on data engineering experience, including expertise in Python or Scala or Java. 
• 3+ years of experience with cloud vendors (AWS, Azure, GCP), data warehouse services (e.g., Redshift, Databricks), and cloud storage (Azure Storage, S3). 
• Production experience with KSQL for real-time data analytics and ClickHouse for fast analytics on streaming data. 
• Experience with stream processing tools like Apache Kafka, Apache Flink, or Apache Spark Structured Streaming. 
• Strong experience with ETL/ELT tools (e.g., Airflow, ADF, Glue, NiFi) and orchestration. 
• Proficient in code versioning (Git) and CI/CD pipelines for data projects. 
• Experience with data modeling, optimization techniques, and stream processing patterns like windowing and exactly-once processing. 
• Experience in managing data warehouses, streaming environments, and scalability in high-volume data settings.
• Excellent leadership and mentorship skills, with at least 1 year in a leadership role.

Nice to Have: 

• Experience in iGaming industry
• Experience with NoSQL databases and stream processing tools such as Kafka Connect and Kafka Streams. 
• Knowledge of Superset for data visualization. 
• Experience integrating real-time machine learning models into streaming environments. 
• Expertise with monitoring and observability tools for streaming systems across on-premises and cloud environments.