Key responsibilities
- Design and develop scalable batch and incremental data pipelines using Fabric Data Factory or Azure Data Factory.
- Build Fabric Lakehouse and Warehouse solutions using OneLake, Delta tables, SQL, and Spark.
- Develop reusable Python/PySpark notebooks for transformation and data-quality processing.
- Implement full-load, incremental-load, CDC, and parameter-driven pipeline patterns.
- Create curated data layers to support reporting, analytics, and downstream consumption.
- Implement logging, exception handling, alerts, retry mechanisms, and audit controls.
- Optimize SQL queries, Spark jobs, pipelines, data models, and storage structures.
- Support deployment across development, test, and production environments.
- Troubleshoot production incidents and conduct root-cause analysis.
- Work with business analysts, architects, report developers, and source-system teams.
- Perform peer reviews and mentor Level 1 engineers.
Mandatory technical skills
- Strong SQL skills, including joins, window functions, CTEs, stored procedures, and performance tuning.
- Hands-on experience with Python and/or PySpark.
- Experience with Azure Data Factory or Fabric Data Factory.
- Experience with Lakehouse, data warehouse, or enterprise data-platform solutions.
- Knowledge of Delta Lake, Parquet, partitioning, and medallion architecture.
- Understanding of data modeling, including star and snowflake schemas.
- Experience using Git and CI/CD tools such as Azure DevOps.
- Working knowledge of Power BI semantic models and reporting requirements.
- Understanding of identity, role-based access, secrets, and cloud security.
Preferred skills
- Experience with Fabric OneLake, shortcuts, Warehouse, Data Engineering, or Real-Time Intelligence.
- Experience migrating data workloads from on-premises or legacy platforms to Azure/Fabric.
- Knowledge of Azure SQL, ADLS Gen2, Synapse Analytics, Azure Functions, or Databricks.
- Experience with REST APIs, JSON, event-based ingestion, or streaming data.
- Microsoft Fabric Analytics Engineer or Azure Data Engineer certification.
Education:
BTECH/BCA