A Data Engineer in the realm of Corporate Services typically plays a critical role in managing and optimizing data-related tasks and processes within a corporate environment.
Dubai – United Arab Emirates
Data Engineer | Corporate Services | Data AI Automation Overview of the role We are currently looking for competent & performance driven Data Engineer to contribute build Enterprise Data Lake for Al Futtaim by ingesting, modelling various Data sources using Data platform. What you will do
• Responsible for implementing robust data pipelines using Microsoft and Databricks technical Stack
• Responsible for building Azure Devops to deploy the ADF pipelines.
• Ability to talk to Business clients, understand the requirements and convert them to technical specification documents.
• Provide technical design, pyspark coding to accomplish the project deliverables as planned/scoped
• Provide technical design, pyspark coding to accomplish the project deliverables as planned/scoped
• Explore the latest Azure technologies and passion to learn new products from Databricks Skills Required skills to be successful
• Bachelor’s Degree preferably Information Technology
• 5 years’ experience in Data Engineering
• Advanced working knowledge and experience with relational and non-relational databases
• Experience building and optimizing Big Data data pipelines, architectures and data sets
• Strong analytic skills related to working with unstructured datasets
• Preference will be given to UAE Nationals with family book What equips you for the role
• The ideal candidate for this role should possess a strong experience in Azure Datafactory (ADF), Databricks, Eventhub, Python, PySpark ,Azure Synapse and SQL
• Azure Devops experience to deploy the ADF pipelines.
Role Overview:
1. Data Pipeline Development: Data Engineers design, develop, and maintain data pipelines that extract, transform, and load (ETL) data from various sources into data storage or analytical platforms. These pipelines ensure that corporate data is readily available for analysis and reporting.
2. Data Integration: They integrate data from different systems and departments within the corporate infrastructure, ensuring data consistency and reliability.
3. Database Management: Data Engineers manage databases, including data warehousing solutions, to store and organize corporate data efficiently. This involves tasks such as schema design, indexing, and optimizing database performance.
4. Data Quality: Ensuring data accuracy and quality is a key responsibility. Data Engineers implement processes and checks to cleanse and validate data, preventing inaccuracies in reporting and analytics.
5. Data Security: They are responsible for implementing data security measures to protect sensitive corporate data from unauthorized access and breaches.
6. Data Architecture: Data Engineers work on designing and optimizing the data architecture, making decisions about data storage technologies, data modeling, and scalability.
7. Data Analytics Support: They support data scientists, analysts, and business users by providing access to high-quality data and assisting in data-related inquiries.
8. Automation: Data Engineers automate data processes wherever possible to reduce manual efforts and improve efficiency.
9. Performance Optimization: Continuously monitoring and optimizing data pipelines and databases for performance and scalability is crucial to ensure that data is available when needed.
10. Documentation: They document data-related processes, workflows, and systems to ensure transparency and maintainability.
Skills and Tools:
- Programming languages like Python, Java, Scala, or SQL
- Data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake)
- ETL tools (e.g., Apache NiFi, Talend, Apache Spark)
- Database management systems (e.g., PostgreSQL, MySQL, NoSQL databases)
- Data modeling and schema design
- Data security and encryption techniques
- Big data technologies (e.g., Hadoop, Hive, HBase)
- Cloud platforms (e.g., AWS, Azure, Google Cloud)
- Version control systems (e.g., Git)
- Automation and orchestration tools (e.g., Apache Airflow)
In summary, a Data Engineer in Corporate Services plays a pivotal role in managing data infrastructure, ensuring data quality and security, and enabling data-driven decision-making within the corporate environment. Their work contributes to the effective use of data for various business purposes, including reporting, analytics, and strategic planning.