Data Engineer

<p>This role is for one of Weekday’s clients<br>Min Experience: 5 years<br>Location: Remote (India)<br>JobType: full-time</p><p><strong>Requirements</strong></p><p><strong>REQUIREMENTS</strong></p><ul> <li>Proficient in&nbsp;</li> <ul> <li> <strong>Programming language</strong>: Python, PySpark , Scala&nbsp;</li> <li> <strong>Azure Environment:&nbsp;</strong>Azure Data Factory, Databricks, Key Vault, DevOps CI CD</li> <li> <strong>Storage/ Databases:&nbsp;</strong>ADLS Gen 2, Azure SQL DB, Delta Lake</li> <li> <strong>Data Engineering:&nbsp;</strong>Apache Spark, Hadoop, optimization, performance tuning, Data modelling</li> <li><strong>Experience working with data sources such as Kafka and MongoDB is preferred.</strong></li> </ul> <li>Experience with Automation of Test Cases of Big Data &amp; ETL</li> <li>Pipelines and Agile Methodology</li> <li>Basic Understanding of ETL Pipelines</li> <li>A strong understanding of AI, machine learning, and data science concepts is highly beneficial.</li> <li>Strong analytical and problem-solving skills with attention to detail.</li> <li>Ability to work independently and as part of a team in a fast-paced environment.</li> <li>Excellent communication skills, able to collaborate with both technical and non-technical stakeholders.</li> <li>Experience designing and implementing scalable and optimized data architectures followed by all best practices.</li> <li>Strong understanding of data warehousing concepts, data lakes, and data modeling.</li> <li>Familiarity with data governance, data quality, and privacy regulations.</li> </ul><p>&nbsp;</p><p><strong>Key Responsibilities:</strong></p><ul> <li> <strong>Data Pipeline Development:</strong> Design, develop, and maintain scalable and efficient data pipelines to collect, process, and store data from various sources (e.g., databases, APIs, third-party services).</li> <li> <strong>Data Integration:</strong> Integrate and transform raw data into clean, usable formats for analytics and reporting, ensuring consistency, quality, and integrity.</li> <li> <strong>Data Warehousing:</strong> Build and optimize data warehouses to store structured and unstructured data, ensuring data is organized, reliable, and accessible.</li> <li> <strong>ETL Processes:</strong> Develop and manage ETL (Extract, Transform, Load) processes for data ingestion, cleaning, transformation, and loading into databases or data lakes.</li> <li> <strong>Performance Optimization:</strong> Monitor and optimize data pipeline performance to handle large volumes of data with low latency, ensuring reliability and scalability.</li> <li> <strong>Collaboration:</strong> Work closely with other product teams , TSO and business stakeholders to understand data requirements and ensure that data infrastructure supports analytical needs.</li> <li> <strong>Data Quality &amp; Security:</strong> Ensure that data systems meet security and privacy standards, and implement best practices for data governance, monitoring, and error handling.</li> <li> <strong>Automation &amp; Monitoring:</strong> Automate data workflows and establish monitoring systems to detect and resolve data issues proactively.</li> <li>Understand the broad architecture of the GEP's entire system as well as Analytics.</li> <li>Take full accountability for role, own development and results</li> </ul>