Data Engineer II

Job Summary:

We are seeking an experienced and highly skilled Data Engineer II to join our dynamic team. The ideal candidate will possess extensive expertise in data warehousing, ETL processes, cloud-based data platforms, SQL, Python programming, data visualization, and orchestration/scheduling tools. You will play a critical role in designing, developing, and maintaining our data infrastructure, ensuring the seamless integration and transformation of data from various sources to support robust analytics and business intelligence.

Key Responsibilities:

• Data Architecture Design: Lead the design, implementation, and maintenance of scalable data warehouse solutions to meet the organization's data storage and retrieval needs.
• ETL Process Management: Oversee the development and management of ETL processes using Informatica Intelligent Cloud Services (IICS)/FiveTran to ensure efficient and accurate data transfer between systems.
• Cloud Data Platforms: Architect and manage cloud data warehouses using Snowflake/BigQuery, ensuring optimal performance, security, and scalability.
• Data Transformation: Utilize dbt (data build tool) to model, transform, and document data pipelines, ensuring high data quality and consistency.
• SQL Development: Develop, optimize, and maintain complex SQL queries and scripts to support data extraction, transformation, and loading processes.
• Python Development: Develop, optimize, and maintain data processing scripts and applications using Python to support various data engineering tasks.
• Data Visualization: Create and manage data visualizations using tools such as DOMO, Tableau, Power BI, or Looker to help stakeholders interpret complex data insights.
• Orchestration and Scheduling: Implement and manage orchestration/scheduling tools such as Apache Airflow, Prefect, or similar to automate and monitor data workflows.
• Team Leadership: Mentor and guide junior data engineers, fostering a culture of continuous learning and improvement.
• Cross-functional Collaboration: Work closely with data analysts, data scientists, and other stakeholders to understand data requirements and deliver robust data solutions.
• Performance Tuning: Monitor and enhance the performance of data systems to ensure high availability and fast query responses.
• Documentation and Standards: Maintain comprehensive documentation for data models, ETL processes, and data infrastructure while establishing best practices and standards.
• Security and Compliance: Implement and manage data security protocols and ensure compliance with relevant regulations and best practices.

Required Skills and Qualifications:

• Educational Background: Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• Experience: Minimum of 5-7 years of experience in data engineering, data warehousing, and ETL development, with at least 2 years.
• Technical Proficiency:
• Expertise in designing and managing data warehouses.
• Advanced proficiency in ETL tools like Informatica Intelligent Cloud Services (IICS), Fivetran, Azure Data Factory, Google Cloud Dataflow etc.
• Extensive hands-on experience with Snowflake for data warehousing and management.
• Proficient in dbt for data modeling and transformation.
• Strong knowledge of Google BigQuery and its ecosystem.
• Proficiency in SQL and database management.
• Advanced skills in developing, optimizing, and maintaining complex SQL queries and scripts.
• Advanced skills in Python for data processing and automation.
• Experience with data visualization tools such as DOMO, Tableau, Power BI, or Looker.
• Experience with orchestration/scheduling tools such as Apache Airflow, Prefect, or similar.
• Experience with data observability tools to monitor data pipeline health and ensure data quality.
• Knowledge of data governance principles and experience in implementing governance policies and procedures.
• Experience with CI/CD practices for data engineering workflows.

• Leadership Skills: Proven ability to lead and mentor a team of engineers, manage projects, and deliver results.
• Analytical Skills: Strong problem-solving skills with the ability to analyze complex data requirements and develop efficient solutions.
• Communication Skills: Excellent written and verbal communication skills to effectively collaborate with team members and stakeholders.
• Preferred Qualifications:
• Experience with other cloud platforms (e.g., GCP, AWS, Azure) and their data services.
• Knowledge of data governance and data quality frameworks.
• Experience with version control systems like Git.