Data Engineer

<p>Job Title: Data Engineer – Databricks</p>
<p>Company: V4C.ai</p>
<p>Type: Full-time</p>
<p>Experience Level: Mid to Senior</p>
<p>Key Responsibilities</p>
<p>Design, build, and maintain scalable data pipelines using Databricks and Apache Spark.</p>
<p>Integrate data from various sources into data lakes or data warehouses.</p>
<p>Implement and manage Delta Lake architecture for reliable, versioned data storage.</p>
<p>Ensure data quality, performance, and reliability through testing and monitoring.</p>
<p>Collaborate with data analysts, scientists, and stakeholders to meet data needs.</p>
<p>Automate workflows and manage job scheduling within Databricks.</p>
<p>Maintain clear and thorough documentation of data workflows and architecture.</p>
<p>Requirements</p>
<p>Experience: 3+ years in data engineering with strong exposure to Databricks and big data tools.</p>
<p>Technical Skills:</p>
<p>Proficient in Python or Scala for ETL development.</p>
<p>Strong understanding of Spark, Delta Lake, and Databricks SQL.</p>
<p>Familiar with REST APIs, including Databricks REST API usage.</p>
<p>Cloud Platform: Experience with AWS, Azure, or GCP.</p>
<p>Data Modeling: Familiarity with data lakehouse concepts and dimensional modeling.</p>
<p>Version Control &amp; CI/CD: Comfortable using Git and pipeline automation tools.</p>
<p>Soft Skills: Strong problem-solving abilities, attention to detail, and teamwork.</p>
<p>Nice to Have</p>
<p>Certifications: Databricks Certified Data Engineer Associate/Professional.</p>
<p>Workflow Tools: Experience with Airflow or Databricks Workflows.</p>
<p>Monitoring: Familiarity with Datadog, Prometheus, or similar tools.</p>
<p>ML Pipelines: Exposure to MLflow or model integration in pipelines.</p>