Data Engineer(ETL, SQL, SSIS, Python), AWS

Data Engineer role to develop and maintain data pipelines in AWS Data Lakehouse using PySpark, SQL, SSIS, and Python.

Requirements

• Develop and debug data pipelines in Databricks using PySpark and SparkSQL.
• Implement data movement from SQL Server to Amazon S3 using efficient batch and incremental loading techniques.
• Query and manage SQL Server environments to validate data consistency during the migration phase.
• Refactor existing SQL-based transformations into modular Python scripts or Scala functions.

Tagged as: Python, SQL